Migrate with map to local storage and 2 servers

How to migrate from an easydb5 server to a fylr server, using assets files on a common NFS share, no need to duplicate assets.

This method can save time and storage. easydb5 is only needed to extract metadata, but not to extract asset files, which are used from storage directly as is.

The following example migrates all from https://easydb.example.com to https://fylr.example.com.

Checks and Requirements

The requirements of the default recommended installation apply here as well, described here...

Except:

Storage for original assets

As the common NFS share is used, exisiting original asset files are already accounted for in storage. So you can ignore that part of the requirements in the default recommended installation.

You may want to have free space for future asset uploads, but that is just the same consideration as with only the easydb5.

Storage for asset versions

The asset versions of easydb5 are typically used by fylr for a while and then are replaced by fylr versions, which are slightly different. How long to use them can be decided on demand, later.

For the requiremeents, you have two options:

A. If you use easydb5 versions read-only (recommended): This is easier and safer, but needs more storage for assets which have both easydb5 versions and fylr versions. So get the storage amount that your easydb5 asset versions take and make sure you have the same amount of storage free, for fylr versions. To be on the safe side, 50% more than the same amount, in case you have lots of assets which get bigger fylr versions compared to easydb5 versions. Example: 1000 GB for easydb5 versions -> 1500 GB free for fylr versions recommended.

B. If you choose to delete easydb5 versions on the fly whenever fylr creates fylr versions: Then minimal storage is needed, but it is harder to configure and has less safety net (easydb5 cannot use the previews any more). Example: 1000 GB for easydb5 versions -> 500 GB free for fylr versions recommended.

Allow passwords to be transferred

In standard configuration, easydb does not output account passwords for migration, as a security feature. To make a full migration, you have to change that setting temporarily, at least for the step "extract metadata" ("backup"). The setting in easydb5 is:

server:
  api:
    user:
      include_password: true

... e.g. in easydb-server.yml . For more information see http://docs.easydb.de/en/technical/api/user/#returning-password-hashes

1. fylr installation

A standard installation with changed storage configuration.

1.a Use docker compose for fylr installation

Use the default recommended installation as described here. But before starting fylr, change docker-compose.yml and fylr.yml as described below.

1.b Edit /srv/fylr/docker-compose.yml

• Add the below shown volume paths, without duplicating the hierarchy (so only one services:, only one fylr:, only one volumes:

• Make shure you adjust the volume paths left of the :, so .e.g. /srv/easydb/eas/lib/assets/orig might by instead /nfs/asset/orig on your server.

• The example below has two volumes for two easydb partitions. Your easydb may have more partitions. Create one fylr volume per easydb partition.

Example (irrelevant lines not shown):

services:
  fylr:
    volumes:
      - "/srv/easydb/eas/lib/assets/orig:/mnt/orig_early:ro"
      - "/srv/easydb/eas/lib/assets/dest:/mnt/dest_early:ro"

1.c Adjust /srv/fylr/config/fylr/fylr.yml

fylr+:
  allowpurge: true

1.d Allow purging fylr in the frontend

see the screenshot:

2. Check that fylr is at least Version 6.26.0

3. Extract metadata from easydb5

3.a. Surf to https://fylr.example.com/inspect/migration

3.b. expand the Paragraph Create backup (by clicking the triangle)

3.c. Fill in at least the following values:

• URL of server : Fill in your equivalent of https://easydb.example.com

• Login: root

• Password: password of easydb's root account

• OAuth2: uncheck this box, it is only needed to extract from fylr

• Max Parallel: Choose less than the number of CPU cores on your easydb server.

• Purge: you can leave this on, it does not affect easydb or fylr. (It was added to overwrite backup files, but currently it creates a new backup anyway)

Click Backup . This can take from one minute to several hours depending on your data.

See here if you want more information about this process.

4. Inject metadata into fylr

4.a. Surf to https://fylr.example.com/inspect/migration (log in as root)

4.b. expand the Paragraph Restore backup (by clicking the triangle)

4.c. Fill in at least the following values:

• Backup : choose the backup that you created above

• URL : Fill in your equivalent of https://fylr.example.com

• Login: root

• Password: password of fylr's root account

• File Mode: choose Use files from source - rput_leave (bulk)

• File Version: use the default original

• Copy file preview versions: Enable this box.

• Include Password: Can be turned off for test runs. When turned on, passwords are included. But for that, the above backup has to be made with a less secure easydb configuration active. See http://docs.easydb.de/en/technical/api/user/#returning-password-hashes

• Include Events: Turn on if you want to transfer the events that were recorded in easydb. Considered not needed unless you know you want it.

• OAuth2: This box has to be enabled.

• OAuth2 Client Id: leave the default fylr-web-frontend

• Max Parallel: Choose a number that is less of the available CPU cores of the fylr server.

• Purge or Continue: Purge : This will overwrite fylr's contents with easydb, which is the whole point. Continue : Is useful if your previous attempt aborted with a timeout or network error and should be continued.

Click Restore . This can take from a few minutes to many hours depending on your data.

• It will continue if you close your browser.

• You can come back to it via https://fylr.example.com/inspect/migration

• And also directly via https://fylr.example.com/inspect/migration/mymigrationname

See here if you want more information about this process.

When the restore/inject is done, a reindex will be done for fylr, so it will take a while until you can see the data in the webfrontend https://fylr.example.com.

5. Teach fylr where to find easydb assets on disk

5.a Collect the "Remote URL Prefixes" of all easydb partitions

Look into https://fylr.example.com/inspect/files/ (log in as root)

• Click on an ID number of a version file (the Version column has small or full or others but not ORIGINAL).

  • note the content of field Remote URL, it might contain e.g. https://easydb.example.com/eas/partitions-inline/2/0/1270/1270/4839d32e5c8ecca1

• Click on an ID number of an original file (Version column has ORIGINAL).

  • Also for this asset note the content of field Remote URL. It might contain e.g. https://easydb.example.com/eas/partitions-inline/1/0/1270/1270/acda0f0f5982bb64

• Next you need to cut off the last parts of the Remote URLs in your notes, so that only the URL prefix remains, which is what you need. The prefix ends before the single zero. So in our example the prefixes are:

https://easydb.example.com/eas/partitions-inline/2/ (for Versions and)
https://easydb.example.com/eas/partitions-inline/1/ (for Originals)

• Count the number of partitions in https://easydb.example.com/servermanager. For that you have to choose easydb Asset Server (EAS) at the top:

  
• Continue searching through different files as above until you have the URL prefix for each partition (but often there are just two).

5.b Configure the locations

Surf to https://fylr.example.com/locationmanager (log in as root)

Create the following two:

• Fylr location easydb originals

  • Read Only: enable this

  • Directory (in container) /mnt/orig_early

  • Remote URL Prefix example: https://easydb.example.com/eas/partitions-inline/1/ Use here one of the prefixes you collected in 5.a. (Do not confuse this with the Prefix field. Use Remote URL Prefix.)

• Fylr location easydb versions

  • Directory (in container) /mnt/dest_early

  • Remote URL Prefix: As above, use one of the collected Remote URL Prefixes. Example: https://easydb.example.com/eas/partitions-inline/2/

  • If you have enough free storage space to double all asset versions (see explanation in requirements, above), then set this location to Read Only. Then none of them will be deleted. Otherwise set it as Default Location for versions . Then, easydb previews will be deleted as they are replaced with fylr previews. Read Only is safer, (especially if you still want to use easydb!) and thus recommended.

• In case your easydb has more than two partitions, you may have to add more fylr locations.

5.c Let fylr use the asset files of easydb5 directly from disk, without asking easydb

• Go to https://fylr.example.com/inspect/files/

• search with Location=remote

• Choose Action Map to local storage and Search result, not Selected. Click the button Action at the right.

• Now the easydb is not used by fylr any more.

This was the central step. fylr is now independent from easydb and easydb can be turned off. (after testing fylr of course.)

6. Adjust previews to fylr standards

As previews from easydb are different from fylr previews, it is recommended to replace easydb previews with fylr previews. This will be done in the background while you can already work with fylr. Although this whole step is optional, you should do it if you encounter problems. When seeking support from the developer you will likely be asked to do this step to come to a clean coherent state.

6.a Produce fylr previews

• Surf to https://fylr.example.com/inspect/files/ (login as root)

• In the drop down menu Version choose original and click the Search button.

• In the drop down menu Action choose produce versions .

• To the right select the round button below Search result (not below Selected).

• Click the Action button.

6.b Check whether all previews have been replaced

• Is the File queue empty at https://fylr.example.com/inspect/system/queues/?queue=file ? At the top it would show something like: There are 18 parallel and 10 parallel high priority only file workers active. The queue`` contains 0 total entries.

• Surf to https://fylr.example.com/inspect/files/ (login as root)

  • In the drop down menu location choose easydb versions and click the Search button.

  • The Search shows zero results when all have been replaced.

6.c Remove easydb preview versions to regain storage (optional)

• At first remove the location easydb versions in the location manager. But do not remove originals!

• Next, remove fylr's access to them (e.g. remove it from /srv/fylr/docker-compose.yml and recreate the container). Do not remove originals.

• Check that the fylr webfrontend still shows previews, at your https://fylr.example.com.

• Then, when all is still working, you can delete the easydb previews to free storage capacity. Warning: Removing the preview versions of easydb should only be done if the easydb is not needed any more. Warning 2: But the easydb originals are used by fylr, so do never remove them!

7. Final steps

Check fylr config

Make sure that you have uploaded a fylr license and tested/configured optional features that you need, e.g. Single Sign On, email sending, schema-sync via objectstore, hotfolder, etc..

Stopping easydb

We recommend stopping all easydb services and Apache and preventing certificate renewal etc..

You may want to stop regular maintenance tasks for easydb, typically in /etc/cron.d/easydb*.

Change fylr domain

In case you want to change fylr's domain (e.g. to the former domain of easydb), see here.