DOCUMENTATION
  • Welcome
  • Releases
    • 2025
      • Release v6.20.2 (2025-05-07)
      • Release v6.20.1 (2025-05-06)
      • Release v6.20.0 (2025-04-30)
      • Release v6.19.2 (2025-04-11)
      • Release v6.19.1 (2025-03-31)
      • Release v6.19.0 (2025-03-27)
      • Release v6.18.2 (2025-03-11)
      • Release v6.18.1 (2025-03-07)
      • Release v6.18.0 (2025-02-26)
      • Release v6.17.3 (2025-02-14)
      • Release v6.17.2 (2025-02-07)
      • Release v6.17.1 (2025-02-06)
      • Release v6.17.0 (2025-01-30)
    • 2024
      • Release v6.16.0 (2024-12-12)
      • Release v6.15.0 (2024-11-27)
      • Release v6.14.2 (2024-11-05)
      • Release v6.14.1 (2024-11-01)
      • Release v6.14.0 (2024-10-31)
      • Release v6.13.3 (2024-10-16)
      • Release v6.13.2 (2024-10-10)
      • Release v6.13.1 (2024-10-02)
      • Release v6.13.0 (2024-09-25)
      • Release v6.12.2 (2024-09-18)
      • Release v6.12.1 (2024-08-01)
      • Release v6.12.0 (2024-07-25)
      • Release v6.11.5 (2024-07-09)
      • Release v6.11.4 (2024-07-05)
      • Release v6.11.3 (2024-07-03)
      • Release v6.11.2 (2024-06-21)
      • Release v6.11.1 (2024-06-14)
      • Release v6.11.0 (2024-06-05)
      • Release v6.10.2 (2024-05-15)
      • Release v6.10.1 (2024-05-08)
      • Release v6.10.0 (2024-04-30)
      • Release v6.9.3 (2024-03-19)
      • Release v6.9.2 (2024-03-15)
      • Release v6.9.1 (2024-03-06)
      • Release v6.9.0 (2024-02-28)
      • Release v6.8.5 (2024-02-02)
      • Release v6.8.4 (2024-02-01)
      • Release v6.8.3 (2024-01-12)
      • Release v6.8.2 (2024-01-05)
    • 2023
      • Release v6.8.1 (2023-12-22)
      • Release v6.8.0 (2023-12-14)
      • Release v6.7.4 (2023-11-15)
      • Release v6.7.3 (2023-11-14)
      • Release v6.7.2 (2023-11-03)
      • Release v6.7.1 (2023-10-17)
      • Release v6.7.0 (2023-10-13)
      • Release v6.6.4 (2023-09-29)
      • Release v6.6.3 (2023-09-28)
      • Release 6.6.2 (2023-09-14)
      • Release v6.6.1 (2023-08-10)
      • Release v6.6.0 (2023-08-03)
      • Release v6.5.1 (2023-06-23)
      • Release v6.5.0 (2023-06-22)
      • Release v6.4.0 (2023-05-31)
      • Release v6.3.1 (2023-04-28)
      • Release v6.3.0 (2023-04-05)
      • Release v6.2.5 (2023-03-16)
      • Release v6.2.4 (2023-02-01)
      • Release v6.2.3 (2023-01-12)
      • Release v6.2.2 (2023-01-12)
      • Release v6.2.1 (2023-01-05)
    • 2022
      • fylr first Production Ready Release 🎉 (2022-12-22)
  • License
  • Help
    • FAQs
    • Tutorials
      • For Users
      • For Administrators
        • Exporting & Importing Hierarchical Lists
        • Regenerating preview images
        • Search Text in images or office files
      • For System Administrators
        • How to setup and use IIIF
        • External access: Sharing collections with anonymous users
    • Glossary
  • FOR USERS
    • Getting Started
    • Asset / Records Management
      • Creating Records
      • Editing Records
        • Input Fields
        • Group Editor
      • Deleting Records
    • Quick Access
      • Collections (& Presentations)
      • Saved Searches (& Lists)
    • Lists
    • Plugins
      • Plugin Overview
  • FOR ADMINISTRATORS
    • Permissions
      • User
      • Groups
      • Object Types
      • Pools
      • Tags & Workflows
      • Presets
    • Tools
      • CSV Importer
        • General Information
        • Options
        • Examples
          • All Data Types
          • Lists
          • Hierarchies
          • Files
      • JSON Importer
        • Step-by-Step Tutorial
          • Write Import Manifest
          • Create Basetype Payloads
          • Create Object Payloads
          • Collection Payloads
          • Optional: Update links between Objects
          • Start Import
      • Permissions Download & Upload
    • Base Configuration
      • General
      • Access
      • User Management
      • Languages
      • Email
      • Export & Deep Links
      • Workflow Webhooks
      • Publications
      • File Worker
        • Preview Configuration
        • Location Defaults
        • Custom .icc Color Profiles
      • Objectstore
      • Services
      • License Management
      • Development
      • Plugins
    • Plugin Manager
    • Location Manager
    • Messages
    • Events
    • Backup Manager
    • Additional Features
      • IIIF
      • Connector
      • Wordpress
      • Zooniverse
      • Protocols
        • OAI/PMH
  • Files and version production
  • FOR SYSTEM ADMINISTRATORS
    • Installation
      • Linux
        • multiple fylrs in one Linux
        • proxy and fylr
      • Windows
      • Kubernetes
    • Configuration
      • fylr.example.yml
      • fylr.default.yml
      • performance tuning
      • pre-load frontend config
      • Load Custom Plugins
      • HTTP and HTTPS
      • DNS Domains
    • Backups & Restore
    • Migration Tool
      • Create payloads (fylr backup)
      • Insert payloads (fylr restore)
      • Best Practice
      • Using the fylr inspect page
    • Integration
      • Authentication
      • Hotfolder
    • Symptom & Solution
      • Log messages that can be ignored
      • too many clients are connected
      • too many nested clauses
      • context canceled
      • ContainerConfig error
      • Purge objects
    • PostgreSQL versions
  • Tutorials
    • Project Workflow
    • Hotfolder & File System Connect
      • Preparations Before Usage
      • Setting Up An Upload Collection
      • Importing Files
    • PDF Creator
    • Extracting File Metadata Later On
    • Overlay Resource
    • Authentication
      • LDAP
      • SAML
    • Data Model Sync
    • Purge a fylr instance
    • typo3 plugin
    • Use fylr in Google docs via CI HUB
  • FOR DEVELOPERS
    • API
      • OAuth2
      • Endpoints
        • /api/collection
        • /api/config
        • /api/db_info
        • /api/db
        • /api/eas
        • /api/event
        • /api/export
        • /api/group
        • /api/l10n
        • /api/mask
        • /api/message
        • /api/oaipmh
        • /api/objects
        • /api/objecttype
        • /api/plugin
        • /api/pool
        • /api/publish
        • /api/right
        • /api/schema
        • /api/search
        • /api/settings
        • /api/suggest
        • /api/system
        • /api/tags
        • /api/transitions
        • /api/user
        • /api/webdav
        • /api/xmlmapping
        • /api/task
    • System Data Types
      • pool
      • file
      • user
      • group
      • pool
      • collection
      • message
      • publish
      • event
    • User Data Types
      • text, text_oneline
      • string
      • text_l10n, text_l10n_oneline
      • boolean
      • number
      • integer.2
      • double
      • date, datetime
      • daterange
      • geojson
    • Custom Data
    • Emails
    • Export
    • Exec server
    • File versions
    • WebDAV
    • Plugin
    • Collection Pin Code
    • easydb 5
    • Localization
    • Access private Repositories
Powered by GitBook
On this page
  • Names used in this tutorial
  • The general steps
  • Important things to consider
  • Using lookups instead of IDs
  • Basic payload structure
  1. FOR ADMINISTRATORS
  2. Tools
  3. JSON Importer

Step-by-Step Tutorial

Step by step tutorial on how to generate JSON Payloads

PreviousJSON ImporterNextWrite Import Manifest

Last updated 1 year ago

This is a step by step tutorial on how to generate JSON Payloads, that can be imported into fylr using the in the frontend.

The importer reads a that defines the import and loads a list of payloads. Each payload is stored in a file. Each payload contains tge definition of the API endpoint, and a list if JSON objects that define basetypes or user objects. Mixed types in payloads are not allowed. All payloads are then posted to the API in the order of the payload list.

This tutorial uses an that covers most (but not necessarily all) aspects of importing payloads.

The payload is not part of the import process but needs to be uploaded and comitted before importing any objects.

Here, we will focus on the structure of the payloads. However, the payload files can be generated using any sources and tools that allow the output of valid JSON files. Each source material needs specialized tools to convert the data into payloads.

Names used in this tutorial

  • Basetypes:

    • Basetype objects like tags, users, groups or pools

    • These always have the same structure and are not defined in the datamodel

  • Objects, User objects, Record:

    • Objects defined in the datamodel (by the user), that contain the actual data

    • Here, objects means user objects, in contrast to basetype objects

The general steps

1. Write the

  • The import manifest is read by the importer and contains information about the payload URI and batch size, and most important, a list of all payload files

  • The filenames of all payloads that are generated have to be added to the payload list in the order in which they have to be imported

These are actual fylr objects that contain data

4. Collections

5. Update imported objects (optional)

6. Start import process

Important things to consider

  • The order of importing is important!

    • Every object that is referenced by another object, must have already imported in a different payload, otherwise a database error can be caused

    • Every parent of a hierarchical basetype or object must have been imported before

      • Hierarchical objects must be imported level by level

      • This means the payloads have to be sorted to represent a breadth first search of the hierarchical tree

    • Linked objects that are referenced must have been imported before

  • Masks should not be used for importing, since values might be ignored if they are not enabled in the mask

    • Always use "_mask": "_all_fields" so that all fields are imported without any limitations

The easiest way to see how a payload is formatted, is to capture the POST/PUT request that is sent by the frontend when a basetype or object is saved. In a browser, open the developer console and find the network tab to see the requests.

Save an object in the editor in the frontend, and find the request that is performed. The body of the post request contains an object JSON represantation, which has exactly the same structure as the objects in the payloads that are used here.

Using lookups instead of IDs

Every object is identified by the _id. If you reference a basetype or object in another basetype or object, you need to know its ID, so that they can be linked. These IDs are assigned in the server to any new inserted object.

Since during migration, the IDs are not known yet, you need to use lookups. All basetypes have a unique field reference, that is used to find the object using a lookup, instead of referencing it by ID.

For user objects, you can use any unique (and not empty) text/string field for referencing the object, or you add an extra field in the datamodel, that can be used as a reference.

Basic payload structure

Each payload file must contain a JSON object with the following structure:

{
  "import_type": "db",
  "objecttype": "bilder", // only needed for import_type "db"
  "objects": []
}

All Payloads must define an import_type and an array of objects. Each object in the array defines a single basetype or user object. The import type specifies the API endpoint. For db, which is the endpoint for user objects, you also must specify the objecttype.

  • Tags:

    • import type: tags

    • array name: tags

  • Groups:

    • import type: group

    • array name: groups

  • Users:

    • import type: user

    • array name: users

  • Pools:

    • import type: pool

    • array name: pools

  • Collections:

    • import type: collection

    • array name: collections

  • User Objects:

    • import type: db

    • array name: objects

    • objecttype must be set

2. Create payloads for

3. Create payloads for

that are linked in main objects

that link to simple objects

Import and collection objects

If necessary, to link to objects that could not be referenced in the first import round.

Load the manifest in the JSON Importer and

JSON importer
manifest
example datamodel
Import Manifest
basetypes
user objects
collections
update imported objects
start the migration
Simple objects
Main objects
Tags
Groups
Users
Pools