easydb 5

Changes from easydb 5 to fylr

Architectural changes

General

The general architecture of fylr is different from easydb5. The main server is written in Go based on a microservices architecture. This should allow better scaling and use less hardware resources than easydb5.

Indexer

fylr uses OpenSearch as its primary indexing tool, but also supports ElasticSearch (same as in easydb5).

Database

fylr supports PostgreSQL and (for development/testing) Sqlite as database backend. Other backends can be added more easily as we no longer use database stored procedure or triggers any more.

All objects are stored in a key-value based system, which makes API commit much faster and lighter. It also enables us to have pool based datamodel specifics and a later point.

All history objects are stored in the same key-value based system which makes it much easier and faster to access history version.

File storage

fylr uses a database for metadata and can use S3 storage or traditional file storage for assets and previews.

Export

The export is not using disk space anymore to export objects. When starting an export, only a list of files is computed. When the file is requested, its being generated on the fly.

• export/stop is not supported any more

• assert export attributes does nothing with "JSON" and "EASParentId" attributes

• _log loads only events from the last run

• csv subfield separator has changed from "#" to "."

• csv field syntax has changed, there is no longer output into a nested collection cell using []

• csv option "empty_columns" is no longer supported

• FILE_DOWNLOAD & OBJECT_DOWNLOAD are new events. "event.info" can be passed via url for additional log entries in the event.info field

Mapping + XSLT

• Mapping can be used with xml_one_object_per_file=false

• Mapping can be used together with XSLT processing

File worker chain

The file worker chain (formerly easydb-asset-server) is new and rethought. The chain can now be configured using base config. The worker system can employ any program to generate any versions for any file. So besides the standard setup for images, videos, and office files, you can easily extend this system to configure preview generations of your choice, such as 3D, CAD, or other files.

Upgrade

Upgrade from easydb5 to fylr requires a complete backup & restore. The fylr binary provides a command line facility to export an easydb 5 (or fylr) and restore into a fylr using the API.

API facing changes

Most of the API will stay unchanged in fylr. In most cases all API accessing programs including the easydb Webfrontend should work as before.

Rightsmanagement

The extensions setting for allow_upload is gone. Only classes can now be configured. All extensions which are configured for the specific class are then allowed to be uploaded.

collection.BAG_CREATE[inherit_owner] is not supported by fylr. inherit_owner is always assumed.

Remove UNLINK from _generated_rights of objects (api/db & api/db_info). The UNLINK right exists for pools as well as collections, so it is ambigious in the context of an object.

_standard-Rendering

If _standard.1 has no data, the fallback to #<system-object-id> is set even if there is _standard.2 or _standard.3. In easydb 5 we would not set _standard.1 if either one was set.

When rendering linked objects the _standard.1-3 is merged with the top level's on each standard level.

fylr only supports 1 hop for _standard-Rendering.

    OK object.lk_artist_id -> artist.name
    OK object.lk_artist_id -> artist.nested_alternatives.aname
NOT OK object.lk_artist_id -> artist.nested_synonym.lk_synonym_id -> synonym.name

The reason for this change is that fylr does not keep _standard around, it computes this for every request.

Also, the _standard render rule has changed, the separator is now a rule for "before" and not for "after".

xml format easydb_flat

This format is no longer supported in fylr. It has been a format which used the internal tables in easydb 5 and dumped them. We could mimic this in fylr, but since that format is not in widespread use, we removed support for it.

Custom Data Type Updater

• "plugin_info" is gone, use "%info.json%" command line replacement instead.

• updater scripts can return event log information

• new action "end_event"

• state is map[string]interface{}

• log can be delivered, this will be added to event.info.Log

OAI/PMH

The XSLT is applied to the full output of the verb, not only to one record.

The easydb XML elements are not prefixed using easydb: anymore. Instead we set xmlns on the top level easydb XML element.

We support sets for collections now. Also, the set names might have changed. The collection sets reported for each record are not filtered by the rightsmanagement of the user.

Non spec "limit" url parameter.

New endpoint /oai

/api/v1/db

Format full:

• includes empty fields (easydb 5 did not include empty fields in format full.)

• _changelog contains different fields than easydb 5, for versions changelog is also complete (with all versions)

• date_range type aggreates in the middle (from+to/2) and not at "from"

• no "base_types_only" support in group mode

Additional support:

• _version:auto_increment: true

• _tags._id:lookup.reference (check: is this really new?)

• _create_user: system.root can set the creating user (used for migrations), _create_user is also in the output

• skipConstraints: allows to skip unique checks & no pool storage (for migrations)

/api/v1/db_info

• /update: "_available_tags" are not including "is_default" any more. This is not very useful with multiple pools as the pools can define their own defaults for the tags.

• output of _generated_rights includes more rights than in easydb 5

/api/v1/eas

• class_version_extensions is no longer being indexed

• metadata groups have a new "value" next to "print"

• _not_allowed is not used, instead version is skipped

• format: short does not return versions, new format standard does

• POST /eas/put on longer supports mapping, use GET /eas instead

• GET /eas?mapping always requires objecttype

• rput has a "delete_url" now to pass in an url which is called upon delete of the file

/api/v1/export

• new url parameter format can be used to filter output for _log

• _log is now sorted ASC

/api/v1/group

• _ipv4_subnet_filter renamed to _ip_subnet_filter, added support for ipv6

/api/v1/mask

The format=xml is no longer supported.

• System fields _acl, _tags, _collection, _owner, _published are hidden from API if configured not to be shown.

• _all_fields mask sets standard for 1st file field and all text fields.

/api/v1/right/preset

• Getting a single preset via ID is new

/api/v1/pool

• _compiled_tags got a new attribute _origin which shows the origin of the tag.

/api/v1/search

• fields.with_path is new, fields collecting for path items has changed. One entry is presented as individual array containing one element per path element, whereas easydb 5 would return individual path elements in one big array.

• sort.with_path defaults to false

• include_deleted is a new switch, which allows to also search deleted objects

• changelog_range understands query to set the type of the comment search

• type: event allows searching in events, "info" is searchable

• removed class_version_extension + class_version_filesize

• moved technical_metadata.date_time_original, technical_metadata.create_date

• colorprofile includes other, like sRGB

• _linked._asset supports the same fields as regular eas fields

• some _linked._asset are available in regular aggregations (like date fields)

• removed aggregation type asset. Use type term instead with _linked._asset. as prefix for the fieldname

• removed search.highlight, this is not supported in FYLR

• new sub search feature for type in. Take field to fetch and subsearch to perform a subsearch to fill the in parameter

• type range has new from_equals and not_equals

• new output formats: standard_extended, long_inheritance, full_inheritance

• new point_in_time feature

• new search_after feature

• language was changed to languages

• linked_object facet has now _linked_object

• removed format from date_range aggregation

• new sort by "_standard_parent", "_standard_parents"

/api/v1/schema

The format=xml is no longer supported.

• exclude/include always implies format "long" (TBD)

/api/v1/settings

This endpoint loses all subpaths, as they have been moved to /api/system.

/api/v1/system

New endpoint which offers:

• /purgeall: purge all data and start over, includes "set_password"

• /reindex: initiate reindex

• /backup: perform a backgrounded backup

• /errortest: store a test error

• /status: status info about the server

/api/v1/objects

The column can also target nested fields, but not reverse.

/api/v1/eas

Renamed skip_extension_check to produce_versions=false.

Format for "short" changed. Use "standard" for compability.

/api/v1/objecttype

Added _filename_replacements as new top level attribute. This lists the available replacements for files.

Added _compiled_tags as new top level attribute.

api/v1/eas/produce

The transform body parameter is no longer an array but a single transform entry

/api/v1/session

• This endpoint was removed. Use OAUTH2 and /api/user/session instead

/api/v1/event

• url parameter base_type has been renamed to basetype

• POST /api/event is now symetric. It receives the event as an object inside the top level object's event property.

• DELETE /api/event/list is new, removed DELETE /api/event (url + body post)

• DELETE /api/event/id is new

• url parameter "background=1" can be used to background event writing (use from within plugins to not dead lock sqlite installations)

/api/v1/user

• default delete policy in base config is "ask" (not "delete")

• new endpoint /api/user/change_password (moved from /api/user/session)

• include_password=1 returns _password_hash, which includes method:hash (this is writable). This also throws an error if the user is not root. ez5 would not output the _password_hash in that case.

• GET /api/user/session returns the current session for the token

• user.created_timestamp -> _created_at

• user.last_updated_timestamp -> _updated_at

• new: _last_seen_at

/api/v1/collection

• Deep-Link changed for type collection and email users: login and password swapped for collection and for email the user logs in with email & uuid instead of email & acl uuid

• New API create_object.linked_object_pools

/api/v1/config

• Separate system and plugin config. New object layer config.

• Can be filtered accessed via path

• Supports partial updates now

Misc. (German)

• fylr ist in Go programmiert, nicht mehr in C++ wie die easydb 5. Dadurch läuft das theoretisch auf allen Betriebssystemen nativ

• Neues Frontend-Design, ansonsten dasselbe Frontend wie in der easydb 5

• Einige Plugins sind in den Core gewandert: OAI/PMH, Webdav Support

• Webdav kann jetzte auch jetzt Read / Write, nicht nur Write und wird direkt von fylr unterstützt

• Es braucht keinen Apache + Easydb Asset Server (EAS) mehr. Execserver ist Teil vom fylr Binary

• Asset-Links sind nicht mehr Apache -> Platte sondern auch über fylr , d.h. sind auch rechtegemanagt

• fylr verbraucht weniger CPU/RAM als easydb 5 und ist horizontal skalierbar

• Alles was externe Programme betrifft ist in den fylr-Execserver ausgelagert, ein skalierbarer Mikroservice zum Ausführen von “Jobs”, zB Bilder kleinrechnen, Metadaten ziehen

• Sämtliche Plugin-Callbacks laufen auch über den Execserver. Plugins können dadurch in beliebigen Programmiersprachen entwickelt werden

• Exporter läuft auch über den Execserver so dass zb XSLT 3.0 benutzt werden kann

• fylr läuft unter Docker / Kubernetes, docker-compose / Helm files verfügbar

• Filestorage kann jetzt auch das S3 Protokoll.

• Alles was mit Dateien passiert und passieren kann (also Formatsumwandlungen, Metadata-Extraction, egal was), ist über .yml Recipes scriptbar. Sogar die Metadata Extraction läuft über den Execserver (allerdings in unserem eigenen Binary welches wiederum exiftool aufruft, das kann man aber wrappen und erweitern)

• Metadataextraktion kann vom Typ abhängig gemacht werden

• Replacement von Original während Inject möglich

• der neue /inspect view ist ein Bare Metal DB View auf die Strukturen und Daten in fylr

• Aktuelle DBs sind Postgres + Sqlite. Wir wollen auch noch CockroachDB supporten (das gestaltet sich allerdings nicht so einfach, da die mit einem anderen TRANSACTION mode als Postgres unterwegs sind)

• Schema Updates in fylr werden NICHT mehr in der SQL DB reflektiert, dort ist alles in einem Key-Value Verfahren gespeichert. D.h. die Anzahl der DB Tabellen in fylr ist fix und ändert sich nicht mit dem User Schema

• Plugins können über die Base Config aktiviert / deaktiviert werden. Auch können Plugins direkt im Frontend nachinstalliert werden

• Integrierter IIIF Support (Presentation API 3.0, Image API 2.0), Tiler konfigurierbar über den Execserver (d.h. ihr könnt zB alles auf JP2 umstellen, oder layered TIFFs)

• Integrieter Bilder-Zoom wird auf den IIIF Endpoint gemappt

• Front end plugins sind kompatibel mit easydb 5, die Server Plugin müssen angepasst werden. Hybrid Server Plugins mit shared Python Code sind möglich (so unser FTP Transport Plugin)

• Exporte werden in Echtzeit generiert und verbrauchen keine extra Plattenplatz mehr

• Die API hat einige für Migrationen sinnvolle Erweiterungen bekommen: Extern setzbare UUID, System Object ID, Object ID, verlinkung von "deferred" Objekts, d.h. ein Promise dass ein verlinktes Objekt später migriert wird

• Support von Polyhierarchien

• Support von automatischen Numerierungen zur Realisierung von Tektoniken

• Event sind jetzt auch in der Elastic und maximal schneller suchbar als in easydb 5

• Visuelles Backuptool im Inspect Backend (da sind wir noch nicht am Ende, schauen wir mal wo das Tool wirklich landet)

• Speicherplatzarmes ablegen der Custom Data inkl externer UUID. Ein Datum wird nur einmal abgelegt und dann verlinkt. In easydb5 wurde das in jeder Zelle als Kopie gehalten

• Datenmodell + Lokalisierung + Mappings sind jetzt auch in der Datenbank, d.h. es gibt "auf Platte" keine Dateien mehr die durch die Applikation angelegt werden

• "suggest" wird durch einen Term splitter in der DB unterstützt und braucht deshalb keine periodischen Index-Rebuilts mehr

• Schnelleres Rechtemanagement, Abfragen mit "generate_rights: true" sind genauso schnell wir ohne

• Authentifizierung ist komplett neu und auf OAUTH2 umgestellt. Erweiterung auf OpenID ist vorgesehen, aber noch nicht umgesetzt. OpenID Client Unterstützung ist möglich, aber nicht umgesetzt.

• Geplant: Feldvererbungen in Hierarchien: Unbelegte Felder erben auf Wunsch automtisch die Einträge vom Vater bzw Großvater