Insert payloads (fylr restore)

This page describes the process of inserting a local backup of JSON payloads to a target fylr instance

The fylr restore command performs POST requests to the API of the target fylr instance. It iterates over the list of payloads from the manifest.json file.

A basic restore command looks like this:

fylr restore \
  --server '<fylr url>/api/v1' \
  --login root
  --password '<cleartext>' \
  --manifest '<instance folder>/manifest.json' \
  --client-id web-client \
  --client-secret foo \
  --client-token-url '<fylr url>/api/oauth2/token' \
  --chunk 1000 \
  --timeout-min=1 \
  --file-api rput \
  --file-version original \
  --purge # --purge OR --continue

• --server, --login and --password refer to the target server

• for the --server parameter, include the HTTP Basic Auth: http://<login>:<password>@<fylr url>/api/v1

• --chunk defines the batch size of objects in POST requests to api/v1/db

  • if the objects are too big or complex, the requests might take too long and cause a timeout

  • in this case, lower this value and continue restoring with --continue

Parameters

This is a complete overview of the command line parameters (run fylr restore -h):

Usage: fylr restore [flags]

Restore API backup into FYLR

Flags:

-h, --help                      Show context-sensitive help.
-v, --verbose                   Set to true, to show additional info.
-n, --log-network               Set to true, to log all network traffic.
    --server=STRING             Source url (overwrites URL of source instance from config)
-l, --login=STRING              If --server is set, use as login. Make sure to use the system root user to connect if used together with --purge.
-p, --password=STRING           If --server is set, use as password
    --client-id=""              If --server is set, use as OAUTH2 client ID
    --client-secret=""          If --server is set, use as OAUTH2 client secret
    --client-token-url=""       If --server is set, use as OAUTH2 token url. If not set, this is derived from --server
    --insecure                  Set to true, to not verify the server's certificate chain and host name
    --log=STRING                Set output to logfile
    --purge                     For backup: set to true, to purge the target directory. For restore: set to true, to purge the target and copy the datamodel. The current password of the user used for the login will be set for the system root user.
    --continue                  Set to true, to continue.
    --verify                    If set, verify payloads of an existing backup (same what --continue does at the beginning).
    --chunk-size=100            chunk size for fetching/pushing data.
    --include-events=""         Comma separated list of event types. Use "-" to skip backup/restoring of events. Empty string will backup/restore all known event types.
    --max-parallel=1            Maximum numbers of parallel workers. 0 uses the number of available CPUs. This creates more load on the source / target system. Defaults to "1" (only one parallel process).
    --manifest=STRING           Path to manifest.json.
    --base-config=STRING        Base config to upload to the target. Not supported with --continue. By default the base config from the backup is uploaded. Use "-" to not upload a base config.
    --datamodel=STRING          Datamodel to upload to the target. Not supported with --continue. By default the datamodel from the backup is uploaded. Use "-" to not upload a datamodel.
    --upload-ignore-files-with-errors Set to true, to ignore file upload errors and strip objects from them. Not available in rput_bulk modes.
    --max-parallel-upload-files=4 Max number of parallel original file + its versions uploads. Defaults to 4 (0 for bulk), max is 100.
    --timeout-min=10            Timeout for connections to target (minutes).
    --include-password          Include password in user restore.
    --skip-constraints          Skip constraints during restore.
    --file-api=""               API used to upload files. Leave empty to not upload files. "put": restore tool uploads files synchronous. "rput": target server loads files from remote URLs. "rput_leave": target server stores remote URLs, no data is copied to storage. "rput" and "rput_leave" are faster, "put" might take long.
    --file-api-access-token="" Use this to pass an access token to fylr backends. This is needed to load files from fylr source instances. It appends the "access_token" query parameter to the remote url of files, and removes the "x-fylr-signature" query parameter.
    --file-version=""           Set to version to use for upload. "original" might take long for "put". Use "preview" for test runs.
    --upload-versions           Set to true, to upload the preview versions (renditions) instead of regenerating them on the target, so it keeps byte-identical renditions. For an --include-files=with-versions backup the renditions are uploaded from the backup; otherwise they are fetched from the source, which must stay reachable during the restore. Cannot be combined with an --include-files=original backup (its renditions are not packed).
    --include-deleted-linked    By default, linked fields whose target carries _latest_version_deleted_at are dropped during restore — the target won't be re-imported, so the link would become a deferred 'Purged / Deferred object'. Set this to keep them, e.g. for a partial restore where the deleted targets are imported in a separate run.
    --skip-reindex              If set, skip reindex at the end of the restore.
    --rename-versions=,...      Rename versions before uploading. This affects uploaded rights as well as file versions. The versions need to be given in the notation "<cls>.<version>:<new version>", e.g. "image.preview:640px" would replace the "preview" version of image to "640px". If the "<new version>" is omitted, the version is removed.

--manifest

Manifest file that has been created by the backup command.

• this parameter is mandatory!

• type: string

--server

API Url of the target instance. The Url must include the API base endpoint, for fylr this is <target url>/api/v1.

• this parameter is mandatory!

• type: string

--login

Username of the user in the target instance. It should be a user with root rights or sufficient read rights.

• this parameter is mandatory!

• type: string

--password

Password of the user in the target instance.

• this parameter is mandatory!

• type: string

--purge

Defines the mode of the restoring (purge or continue).

If this is true, the complete restore starts from the beginning, and the target instance is purged.

The parameters --purge and --continue are mutually exclusive. Not both can be true.

• type: bool

• default: false

--continue

Defines the mode of the restoring (purge or continue).

If this is true, the restore continues from the last point in the progress.json file, if a previous restore run was interrupted.

The parameters --purge and --continue are mutually exclusive.

• type: bool

• default: false

--base-config

Path to a specific base config file. Defaults to <instance folder>/base_config.json from the backup.

To not upload a base configuration to the target instance, specify --base-config=-

This parameter is not allowed in combination with --continue.

• type: string

• default: base_config.json

--datamodel

Path to a specific datamodel file. Defaults to <instance folder>/datamodel.json from the backup.

To not upload a datamodel to the target instance, specify --datamodel=-

This parameter is not allowed in combination with --continue.

Trying to upload a datamodel without --purge will not work, if there is already a datamodel on the target instance!

Only use --datamodel=- if the target instance already has a matching datamodel!

• type: string

• default: datamodel.json

--skip-constraints

Skip constraints during restore.

• type: bool

• default: false

--skip-reindex

This parameter is available in fylr from version 6.29.0.

If set, skip reindex at the end of the restore.

• type: bool

• default: false

--include-deleted-linked

This parameter is available in fylr from version 6.33.0.

By default the restore ignores linked objects whose target was soft-deleted in the source. The restored object then simply has no value for that field, so the frontend does not render a "(Purged / Deferred object)" placeholder.

Set to true to keep those wrappers. Two cases to be aware of:

• Backup made with --include-deleted:

  • the soft-deleted target is in the same payload and the lookup resolves to it on restore

  • the link is preserved and the target stays soft-deleted in the restored instance

• Backup made without --include-deleted

  • the target is not in the payload

  • the lookup uses _allow_defer and shows the link as _purged_or_deferred over the API

  • this mirrors the state of a real soft-delete-then-purge of the target in the source

A round-trip that preserves both the link and the target requires both flags: fylr backup --include-deleted and fylr restore --include-deleted-linked.

• type: bool

• default: false

--include-password

Include user password hashes. If this option is true, the restore tool checks if there is at least one user where a password hash is present. This is done by checking the has_passwords flag in the manifest.json. If this value is false, the restore will stop with an error.

If the source is an easydb5: make sure that the source instance is configured to output the user passwords, and repeat the backup. See https://docs.easydb.de/en/sysadmin/configuration/easydb-server.yml/available-variables/ under include_passwords for the necessary settings in the source instance.

For a fylr source instance this is not necessary, since the password hashes are always returned over the API.

• type: bool

• default: false

--include-events

Comma separated list of event types.

Use --include-events=- to skip restoring of events.

If this parameter is unset, all events are restored which have a type that is known in fylr. All other events from the backup are ignored.

• type: string

• default: ""

--chunk-size

The upload batch size for objects to the target instance. Can be used to control the size of the requests. It can be lowered if the requests cause timeouts or network problems.

• type: int

• minimum: 1

• maximum: 1000

• default: 100

--file-api

Method used to upload files.

• "" (empty or unset): do not upload any files at all (default)

• put: upload files directly to the target server

• rput: only upload file URLs, target server loads files from remote URLs

• rput_leave: target server only stores remote URLs, no data is copied to storage

• rput_bulk: like rput, but URLs are uploaded in batches, not in a single request per URL (faster)

• rput_bulk_leave: like rput_leave, but URLs are uploaded in batches (faster)

rput, rput_leave, rput_bulk and rput_leave_bulk are faster, put might take long.

A file that the source instance itself only references at a remote URL (left there via leave_on_remote) is always restored as a reference, whatever method is chosen — the source flag is additive to the run method. All other files follow the chosen method (since fylr 6.34.2; 6.34.0 and 6.34.1 routed every file by its source flag alone).

By default no files or URLs are uploaded. You have to specify one of the file upload methods to transfer any files.

• type: string

• default: ""

--file-api-access-token

Use this to pass an access token to fylr backends. This is needed to load files from fylr source instances. It appends the access_token query parameter to the remote url of files, and removes the x-fylr-signature query parameter.

• type: string

--file-version

Specify which version of the source asset is used. Default is "original" which might take long for --file-api=put.

Use "preview" for test runs with smaller assets.

• type: string

• default: "original"

--upload-versions

Set to true, to not regenerate the preview versions (renditions) on the target, but upload/link the source's versions instead, so the target keeps byte-identical renditions.

The upload method --file-api is used for versions the same way as for the original file. The renditions are fetched from the source, so the source instance must stay reachable during the restore.

• If put, rput or rput_bulk is used, the versions are copied from the source instance, but no versions are produced

• If rput_leave or rput_bulk_leave is used, the versions are linked by URL, and no files are copied to the target instance and no versions are produced

For a backup made with --include-files=with-versions the renditions are packed on disk, so --upload-versions uploads them from the backup (no source required). For a URL or --include-files=original backup they are fetched from the source instead — and for --include-files=original this is rejected, because the renditions are not packed. In every case --upload-versions is explicit: without it the target regenerates the renditions.

• type: bool

• default: false

--rename-versions

Rename versions before uploading. This affects uploaded rights as well as file versions.

The versions need to be given in the notation "<cls>.<version>:<new version>", e.g. "image.preview:640px" would replace the "preview" version of class image to 640px.

If the <new version> part is omitted, the version is removed.

See also: https://docs.fylr.io/for-system-administrators/migration/renaming-renditions-during-migration

• type: string

--max-parallel

Maximum numbers of parallel workers.

0 uses the number of available CPUs.

Defaults to 1 (only one parallel process).

This creates more load on the target system.

• type: int

• minimum: 0

• default: 1

--max-parallel-upload-files

Maximum number of parallel uploads of original files and their versions. Defaults to 4 (0 for bulk). The maximum is 10 (unlimited for bulk).

• type: int

• minimum: 0

• maximum: 10

• default: 4

--timeout-min

Timeout for connections to target in minutes.

Increase this value if you notice timeouts or related network problems when uploading payloads or files to the target system.

• type: int

• minimum: 1

• default: 10

--upload-ignore-files-with-errors

Set to true to ignore file upload errors and strip objects from them.

• type: bool

• default: false

--client-id

If the target instance uses OAuth2 for user authentication, this is the configured Client ID.

• type: string

--client-secret

If the target instance uses OAuth2 for user authentication, this is the configured Client Secret. Can be kept empty if the instance is public.

• type: string

--client-token-url

If the target instance uses OAuth2 for user authentication, this is the OAuth2 callback endpoint of the instance.

For fylr this is <source url>/api/oauth2/token.

• type: string

--verify

Set to true to verify payloads of an existing backup.

This is the same what is done at the beginning if --continue is used.

• type: bool

• default: false

--insecure

Set to true to skip the certificate check for the connection to the target instance.

Only use this option if you can trust the remote server!

• type: bool

• default: false

--verbose

Set to true to log debugging info.

• type: bool

• default: false

--log-network

Set to true to log the requests and responses.

• type: bool

• default: false

--log

If this is a valid file path, the log output is written to this file. If this is empty (default), the log output is written to stdout instead.

• type: string