Create payloads (fylr backup)

This page describes the process of creating a local backup of JSON payloads from a source instance

The fylr backup command performs GET requests to the API of the source instance, until all relevant base types and records are retrieved and stored in local files.

A basic backup command looks like this:

fylr backup \
  --server '<source-url>/api/v1/' \
  --login 'root' \
  --password '<cleartext>' \
  --dir '<instance folder>/' \
  --compression 9 \
  --purge

--server, --login and --password refer to the source server.

Parameters

This is a complete overview of the command line parameters (run fylr backup -h):

Usage: fylr backup [flags]

Backup ez5 or FYLR via API

Flags:

-h, --help                      Show context-sensitive help.
-v, --verbose                   Set to true, to show additional info.
-n, --log-network               Set to true, to log all network traffic.
    --server=STRING             Source url (overwrites URL of source instance from config)
-l, --login=STRING              If --server is set, use as login. Make sure to use the system root user to connect if used together with --purge.
-p, --password=STRING           If --server is set, use as password
    --client-id=""              If --server is set, use as OAUTH2 client ID
    --client-secret=""          If --server is set, use as OAUTH2 client secret
    --client-token-url=""       If --server is set, use as OAUTH2 token url
    --insecure                  Set to true, to not verify the server's certificate chain and host name
    --log=STRING                Set output to logfile
    --purge                     For backup: set to true, to purge the target directory. For restore: set to true, to purge the target and copy the datamodel. The current password of the user used for the login will be set for the system root user.
    --continue                  Set to true, to continue.
    --verify                    If set, verify payloads of an existing backup (same what --continue does at the beginning).
    --chunk-size=100            chunk size for fetching/pushing data.
    --include-events=""         Comma separated list of event types. Use "-" to skip backup/restoring of events. Empty string will backup/restore all known event types.
    --max-parallel=1            Maximum numbers of parallel workers. 0 uses the number of available CPUs. This creates more load on the source / target system. Defaults to "1" (only one parallel process).
-d, --dir=STRING                Target directory.
    --compression=0             0: no compression, 1: speed, 9: best.
    --all-versions              Set to true, to request all versions of an object.
    --include=""                Filter regexp to include objecttypes.
    --maximum-count=0           Limit records. Set to 0 for unlimited.
    --retry-max-count=10        Number of retries for failed requests with network problems.
    --retry-sleep-between=30    Wait time in seconds between retries for failed requests.
    --pretty                    Output pretty JSON.

--dir

Target folder for the backup files. If it does not exist, it will be created. If it exists, and --purge is used, an existing folder is deleted and created new.

  • this parameter is mandatory!

  • type: string

--server

API Url of the source instance. The Url must include the API base endpoint, for fylr this is <source url>/api/v1.

  • this parameter is mandatory!

  • type: string

--login

Username of the user in the source instance. It should be a user with root rights or sufficient read rights.

  • this parameter is mandatory!

  • type: string

--password

Password of the user in the source instance.

  • this parameter is mandatory!

  • type: string

--purge

Defines the mode of the backup (purge or continue).

If this is true, the complete backup starts from the beginning, and an existing backup folder with the same name is purged.

  • type: bool

  • default: false

--continue

Defines the mode of the backup (purge or continue).

If this is true, the backup continues from the last point in the manifest.json file, if a previous backup run was interrupted.

  • type: bool

  • default: false

--size

The target size for objects in payloads. The effective size of the generated payloads is limited by --chunk.

  • type: int

  • minimum: 1

  • maximum: 1000

  • default: 100

--chunk

The requested size for objects from the source instance. Can be used to control the size of the responses. It can be lowered if the requests cause timeouts or network problems.

  • type: int

  • minimum: 1

  • maximum: 1000

  • default: 100

--chunk-size

This parameter is available in fylr from version v6.18.0.

The requested size for objects from the source instance. Can be used to control the size of the responses. It can be lowered if the requests cause timeouts or network problems.

  • type: int

  • minimum: 1

  • maximum: 1000

  • default: 100

--limit

Set this to a number bigger than 0 to limit the number of objects of each objecttype. This can be used to test or debug, and can be combined with --include to only backup a small sample of the source instance.

  • type: int

  • minimum: 0

  • default: 0

--max-count

This parameter is available in fylr from version v6.18.0.

Set this to a number bigger than 0 to limit the number of objects of each objecttype. This can be used to test or debug, and can be combined with --include to only backup a small sample of the source instance.

  • type: int

  • minimum: 0

  • default: 0

--max-parallel

This parameter is available in fylr from version v6.18.0.

Maximum numbers of parallel workers.

0 uses the number of available CPUs.

Defaults to 1 (only one parallel process).

  • type: int

  • minimum: 0

  • default: 1

--compression

GZIP compression level.

If 0 is selected, there is no compression and the payloads are stored as .json files. If a value bigger than 0 is selected, the payloads are stored as compressed .json.gz files.

  • type: int

  • minimum: 0

  • maximum: 9

  • default: 0

--all-versions

Set to true so that all history versions of the records are requested.

  • type: bool

  • default: false

--include

If this is a valid non empty regex string, only objecttypes are backupped where the internal objecttype name matches the regex.

  • type: string

--include-events

Comma separated list of event types.

Use --include-events=- to skip backup of events.

If this parameter is unset, all events are backupped

  • type: string

  • default: ""

--retry-max-count

If a request fails with one of the following HTTP status codes that indicate network problems, the request is repeated for a maximum number of times: 502: Bad Gateway, 503: Service Unavailable, 504: Gateway Timeout

  • type: int

  • default: 10

--retry-sleep-between

Defines the waiting time in seconds between repeated failed requests.

  • type: int

  • default: 30

--client-id

If the source instance uses OAuth2 for user authentication, this is the configured Client ID.

  • type: string

--client-secret

If the source instance uses OAuth2 for user authentication, this is the configured Client Secret. Can be kept empty if the instance is public.

  • type: string

--client-token-url

If the source instance uses OAuth2 for user authentication, this is the OAuth2 callback endpoint of the instance. For fylr this is <source url>/api/oauth2/token.

  • type: string

--insecure

Set to true to skip the certificate check for the connection to the source instance.

  • type: bool

  • default: false

--verbose

Set to true to log debugging info.

  • type: bool

  • default: false

--log-network

Set to true to log the requests and responses.

  • type: bool

  • default: false

--log

If this is a valid file path, the log output is written to this file. If this is empty (default), the log output is written to stdout instead.

  • type: string

--pretty

This parameter is available in fylr from version v6.17.0.

Set to true to save the data in a prettified JSON.

  • type: bool

  • default: false

Migration of events: differences between easydb5 and fylr

Events of the following types are either not migrated at all, or they will automatically be changed to be imported into fylr.

All event types which are not mentioned here, will be migrated without any type changes.

The event info block will be migrated without any changes. IDs which relate to objects, files, etc., and which are not part of the event info block, are automatically replaced during the restore process.

Event Type API_CALL

Event Type API_PROGRESS

Event Type COLLECTION_OWNER_RIGHTS_ERROR

Event Type CUSTOM_TYPE_UPDATER_START

The event type will be changed from CUSTOM_TYPE_UPDATER_START to CUSTOM_DATA_TYPE_UPDATER in fylr.

Event Type CUSTOM_TYPE_UPDATER_RESPONSE

The event type will be changed from CUSTOM_TYPE_UPDATER_RESPONSE to CUSTOM_DATA_TYPE_UPDATER in fylr.

Event Type CUSTOM_TYPE_UPDATER_RESPONSE_ERROR

The event type will be changed from CUSTOM_TYPE_UPDATER_RESPONSE_ERROR to CUSTOM_DATA_TYPE_UPDATER_ERROR in fylr.

Event Type CUSTOM_TYPE_UPDATER_RESPONSE_FAILED

The event type will be changed from CUSTOM_TYPE_UPDATER_RESPONSE_FAILED to CUSTOM_DATA_TYPE_UPDATER_ERROR in fylr.

Event Type SCHEMA_COMMIT

The event type will be changed from SCHEMA_COMMIT to DATAMODEL_COMMIT in fylr.

Event Type EXPORT_ASSET

Event Type EXPORT_OBJECT

Event Type ASSET_EXPORT_TRANSPORT_DOWNLOAD

The event type will be changed from ASSET_EXPORT_TRANSPORT_DOWNLOAD to EXPORT_TRANSPORT in fylr.

Event Type EXPORT_STOPPED

Event Type DOWNLOAD_EXPORT

Event Type ASSET_DOWNLOAD

The event type will be changed from ASSET_DOWNLOAD to FILE_DOWNLOAD in fylr.

The basetype of this event will be changed from asset to file in fylr.

Event Type ASSET_EXPORT_DOWNLOAD

The event type will be changed from ASSET_EXPORT_DOWNLOAD to FILE_DOWNLOAD in fylr.

The basetype of this event will be changed from asset to file in fylr.

Event Type SUGGEST_INDEX_DONE

Event Type SUGGEST_INDEX_FAILED

Event Type JANITOR_SESSION_PURGE

The event type will be changed from JANITOR_SESSION_PURGE to JANITOR in fylr.

Event Type MAPPING_DELETE

The basetype of this event will be set to mapping in fylr. In easydb5, there was no basetype for this event type.

Event Type MAPPING_INSERT

The basetype of this event will be set to mapping in fylr. In easydb5, there was no basetype for this event type.

Event Type MAPPING_UPDATE

The basetype of this event will be set to mapping in fylr. In easydb5, there was no basetype for this event type.

Event Type RESOURCE_NOT_AVAILABLE

Event Type SERVER_START

Event Type SERVER_SHUTDOWN

Event Type SESSION_INVALID

Event Type SUGGEST_INDEX_DONE

Event Type SUGGEST_INDEX_FAILED

Event Type USER_CREATED

The event type will be changed from USER_CREATED to USER_INSERT in fylr.

The basetype of this event will be set to user in fylr. In easydb5, there was no basetype for this event type.

Event Type USER_LOGIN

The basetype of this event will be set to user in fylr. In easydb5, there was no basetype for this event type.

Event Type USER_LOGOUT

The basetype of this event will be set to user in fylr. In easydb5, there was no basetype for this event type.

Event Type LOGIN_FAILED

Event Type USER_ACCEPTED_MESSAGE

Event Type WEBHOOK_OK

The event type will be changed from WEBHOOK_OK to WEBHOOK in fylr.

Last updated