Create payloads (fylr backup)

This page describes the process of creating a local backup of JSON payloads from a source instance

The fylr backup command performs GET requests to the API of the source instance, until all relevant base types and records are retrieved and stored in local files.

A basic backup command looks like this:

fylr backup \
  --server '<source-url>/api/v1/' \
  --login 'root' \
  --password '<cleartext>' \
  --dir '<instance folder>/' \
  --compression 9 \
  --purge

--server, --login and --password refer to the source server.

--purge deletes a backup in --dir in case there is already one. All existing files in this directory are deleted!

By default the backup over the API only stores the data in local files; the files themselves are only referenced by URL and are loaded from the source instance during the restore, so until the restore is finished the source instance still needs to be reachable.

Use --include-files to pack the file bytes into the backup instead, making it self-contained.

Not all events will be migrated from easydb5 to fylr, because they have no purpose in fylr. Also, some events need to be changed.

For a complete overview, see below

Parameters

This is a complete overview of the command line parameters (run fylr backup -h):

Usage: fylr backup [flags]

Backup ez5 or FYLR via API

Flags:

-h, --help                      Show context-sensitive help.
-v, --verbose                   Set to true, to show additional info.
-n, --log-network               Set to true, to log all network traffic.
    --server=STRING             Source url (overwrites URL of source instance from config)
-l, --login=STRING              If --server is set, use as login. Make sure to use the system root user to connect if used together with --purge.
-p, --password=STRING           If --server is set, use as password
    --client-id=""              If --server is set, use as OAUTH2 client ID
    --client-secret=""          If --server is set, use as OAUTH2 client secret
    --client-token-url=""       If --server is set, use as OAUTH2 token url. If not set, this is derived from --server
    --insecure                  Set to true, to not verify the server's certificate chain and host name
    --log=STRING                Set output to logfile
    --purge                     For backup: set to true, to purge the target directory. For restore: set to true, to purge the target and copy the datamodel. The current password of the user used for the login will be set for the system root user.
    --continue                  Set to true, to continue.
    --verify                    If set, verify payloads of an existing backup (same what --continue does at the beginning).
    --chunk-size=100            chunk size for fetching/pushing data.
    --include-events=""         Comma separated list of event types. Use "-" to skip backup/restoring of events. Empty string will backup/restore all known event types.
    --max-parallel=1            Maximum numbers of parallel workers. 0 uses the number of available CPUs. This creates more load on the source / target system. Defaults to "1" (only one parallel process).
-d, --dir=STRING                Target directory.
    --compression=0             0: no compression, 1: speed, 9: best.
    --all-versions              Set to true, to request all versions of an object.
    --include-deleted           By default soft-deleted objects are skipped. Set this to include them in the backup, so that links pointing at a deleted object can be restored faithfully (paired with restore's --include-deleted-linked).
    --include=""                Filter regexp to include objecttypes.
    --maximum-count=0           Limit records. Set to 0 for unlimited.
    --retry-max-count=10        Number of retries for failed requests with network problems.
    --retry-sleep-between=30    Wait time in seconds between retries for failed requests.
    --pretty                    Output pretty JSON.
    --include-files=""          How to store file bytes in the backup. Empty (default): store only the file URLs, the restore fetches the bytes from the source (which must stay reachable). "original": pack each non-leave file's original bytes into the backup directory (files/<id-floored-to-1e6>/<id-floored-to-1e3>/<id>-<filename>), so the backup is self-contained and byte-for-byte and the source need not stay reachable; renditions (preview versions) are not packed, the restore regenerates them, so such a backup cannot be restored with --upload-versions. "with-versions": also pack the renditions, so that a restore with --upload-versions uploads them from the backup (byte-identical) instead of regenerating them. Files kept on the remote (leave) are never packed; their upstream URL is preserved.

--dir

Target folder for the backup files. If it does not exist, it will be created. If it exists, and --purge is used, an existing folder is deleted and created new.

• this parameter is mandatory!

• type: string

--server

API Url of the source instance. The Url must include the API base endpoint, for fylr this is <source url>/api/v1.

• this parameter is mandatory!

• type: string

--login

Username of the user in the source instance. It should be a user with root rights or sufficient read rights.

• this parameter is mandatory!

• type: string

--password

Password of the user in the source instance.

• this parameter is mandatory!

• type: string

--purge

Defines the mode of the backup (purge or continue).

If this is true, the complete backup starts from the beginning, and an existing backup folder with the same name is purged.

The parameters --purge and --continue are mutually exclusive. Not both can be true.

If the target folder already exists, this parameter (or --continue) must be set, otherwise the backup will fail.

• type: bool

• default: false

--continue

Defines the mode of the backup (purge or continue).

If this is true, the backup continues from the last point in the manifest.json file, if a previous backup run was interrupted.

The parameters --purge and --continue are mutually exclusive. Not both can be true.

If the target folder already exists, this parameter (or --purge) must be set, otherwise the backup will fail.

• type: bool

• default: false

--chunk-size

The requested size for objects from the source instance. Can be used to control the size of the responses. It can be lowered if the requests cause timeouts or network problems.

• type: int

• minimum: 1

• maximum: 1000

• default: 100

--max-count

Set this to a number bigger than 0 to limit the number of objects of each objecttype. This can be used to test or debug, and can be combined with --include to only backup a small sample of the source instance.

• type: int

• minimum: 0

• default: 0

--max-parallel

Maximum numbers of parallel workers.

0 uses the number of available CPUs.

Defaults to 1 (only one parallel process).

This creates more load on the source system.

• type: int

• minimum: 0

• default: 1

--compression

GZIP compression level.

If 0 is selected, there is no compression and the payloads are stored as .json files. If a value bigger than 0 is selected, the payloads are stored as compressed .json.gz files.

• type: int

• minimum: 0

• maximum: 9

• default: 0

--all-versions

Set to true so that all history versions of the records are requested.

Not to be confused with asset versions (see parameters for fylr restore)!

• type: bool

• default: false

--include

If this is a valid non empty regex string, only objecttypes are backupped where the internal objecttype name matches the regex.

• type: string

--include-deleted

This parameter is available in fylr from version 6.33.0.

By default, linked objects that are soft-deleted (their latest version carries _latest_version_deleted_at) are skipped from the backup. Set this parameter to true to include them as top-level entries in the payload, in their soft-deleted state.

Links pointing at a soft-deleted target are always written to the backup as a lookup:_id wrapper carrying _latest_version_deleted_at next to _system_object_id and _allow_defer.

For a faithful round-trip (soft-deleted targets restored as they are, with their incoming links intact) use this together with fylr restore --include-deleted-linked. If the backup was done without --include-deleted and restored with --include-deleted-linked, the wrappers are kept but their lookups defer, so the links are stored as _purged_or_deferred.

Use this paired with the fylr restore parameter --include-deleted-linked

• type: bool

• default: false

--include-files

This parameter is available in fylr from version 6.34.0.

By default a backup only stores the file URLs, and the restore fetches the bytes from the source instance (which therefore has to stay reachable until the restore finishes). Set --include-files to pack the file bytes into the backup itself, under a files/ directory, so the restore uploads them from the local backup and no longer depends on the source instance. It takes a mode:

• empty (default): store only the file URLs (the behaviour described above).

• original: pack each file's original bytes, but not its renditions (preview versions). On restore the target regenerates the renditions itself, so an --include-files=original backup cannot be restored with --upload-versions — that option uploads the source's renditions, but they are not in the backup and the source may be gone, so the restore rejects the combination.

• with-versions: pack the originals and their renditions, so that a restore with --upload-versions uploads the renditions from the backup byte-identical instead of regenerating them. Without --upload-versions the target still regenerates them (exactly as for a URL-only backup) — the flag stays explicit. This is the basis for a fully self-contained, byte-for-byte backup.

Files stored leave on remote are never packed: the backup keeps their real upstream URL (it requests files_real_url=1 on the /db list, which needs the root system right), and the restore re-creates them as references to that upstream. Their renditions are ordinary local files, so with with-versions a leave original's renditions are packed and uploaded from the backup too.

The backup form under /inspect/migration offers this as an Include Files selector, and its backup viewer browses the nested files/ directories.

• type: string (one of "", original, with-versions)

• default: ""

--include-events

Comma separated list of event types.

Use --include-events=- to skip backup of events.

If this parameter is unset, all events are backupped

Specific event types, which are irrelevant for the migration, are never in the backup, even if they are specifically requested.

For now, this includes OBJECT_INDEX, SESSION_INVALID, FRONTEND_ERROR and COLLECTION_OWNER_RIGHTS_ERROR, but this might be extended in the future.

• type: string

• default: ""

--retry-max-count

If a request fails with one of the following HTTP status codes that indicate network problems, the request is repeated for a maximum number of times: 502: Bad Gateway, 503: Service Unavailable, 504: Gateway Timeout

• type: int

• default: 10

--retry-sleep-between

Defines the waiting time in seconds between repeated failed requests.

• type: int

• default: 30

--client-id

If the source instance uses OAuth2 for user authentication, this is the configured Client ID.

• type: string

--client-secret

If the source instance uses OAuth2 for user authentication, this is the configured Client Secret. Can be kept empty if the instance is public.

• type: string

--client-token-url

If the source instance uses OAuth2 for user authentication, this is the OAuth2 callback endpoint of the instance. For fylr this is <source url>/api/oauth2/token.

• type: string

--insecure

Set to true to skip the certificate check for the connection to the source instance.

Only use this option if you can trust the remote server!

• type: bool

• default: false

--verbose

Set to true to log debugging info.

• type: bool

• default: false

--log-network

Set to true to log the requests and responses.

• type: bool

• default: false

--log

If this is a valid file path, the log output is written to this file. If this is empty (default), the log output is written to stdout instead.

• type: string

--pretty

Set to true to save the data in a prettified JSON.

• type: bool

• default: false