Release v6.8.0 (2023-12-14)

Published 2023-12-14 14:44:53Z

• fylr_checksums.txt

• fylr_v6.8.0_darwin_amd64.tar.gz

• fylr_v6.8.0_darwin_arm64.tar.gz

• fylr_v6.8.0_linux_amd64.tar.gz

• fylr_v6.8.0_windows_amd64.zip

Release 6.8.0

Important

• If you are using a custom file worker configuration with the recipe imageconverter:reformat, you need to switch that to imageconverter:browserthumbs (see below).

• We stopped shipping a copy of the embedded resource files in our download packages. Use the new fylr resources --copy command to access resources embedded in the fylr binary instead. For more info see fylr resources --help.

• This release deletes all OAUTH2 tokens. So a re-login will be required for all users after update.

• Re-index is advised. We changed some base mappings, so it is advised to run re-index. Use /inspect/system for that.

New

• /api/search: Support user._last_seen_at in index.

• /api/db: Support column types number and integer.2 in lookup:_id.

• All file urls are now signed with an HMAC signature. Using that, we can save time during the file request by skipping the rights management. Url signature have an expiration which can be adjusted in the base config. /api/search and /api/db support a new query parameter file_url_expire=DAYS. Setting this to 0 causes the Urls to not be signed (that is the old behaviour). For extensible Urls like iiif_url and zoom_url the signature is part of the path. In addition to passing the signature in the url parameter, clients can choose to pass the signature (x-fylr-signature removed from the query parameter to make use of a browser local cache) in the http header x-fylr-signature of the request.

• /api/db: Support modifier :append to add additional text to text fields. Values are concatenated to existing values, if the records don't contain a value yet, it is created.

• PUT /api/collection/list: New endpoint to support fast collection injection. Before, this was a single item API and only for GET we supported multiple collections. This is used in fylr restore to speed up restores of collections.

• POST /eas/put, POST /eas/rput and POST /eas/ID support a new parameter custom_version_config. With this parameter JSON can be sent to overlay configuration used for the file production recipes. Standard produce configuration is set in the base config (or per default settings). Now, this parameters allows a per-file overwrite and extension of the produce parameters. Using this for videos, the recipe parameter video-thumb-select can be set to select a certain timestamp to produce the video preview thumbnail.

• POST /api/eas: New url parameter permission_check_only allows to check if the current user is allowed to use that endpoint. Is essentially running the same permission check as it does if a payload body is received.

• DELETE search/point_in_time: New endpoint to remove point in time snapshots of the search. This endpoint uses the same payload as for OpenSearch.

• Resource overloading: A new overlay feature allows to merge mount external resources for fylr.resources and fylr.services.webapp.path. Files present in the external resources take precedence over fylr embedded files. With this it is possible to inject and overwrite selected files only without needing to copy and maintain a full resource folder outside fylr. The command fylr resources was added to manage embedded files and the overlayfs. You can use it to dump embedded files from fylr to disk as a starting point to work with local resources or webfrontend. For more info see fylr resources --help. With docker, this might look like: docker exec fylr /fylr/bin/fylr resources --help

• Load all cookbooks from recipe folder: With this it is possible to inject external recipes using an overlayed resource.

• New config fylr.logger.wrongPasswordLevel: This sets the log level for the wrong password message. It also includes the remote address of the requester.

• ICC color profiles: Support color profile in image thumbnail recipe "browerthumbs". With this it is possible to configure color profiles in the the file worker configuration. Also, added a place to upload own color profiles in base config. colorprofile is also supported for custom rendering using GET /api/eas/download.

• /api/export: New file_metadata parameter to include all metadata of file data in XML and JSON exports.

• New option for metadata profiles: deep_link_url allows to embed a deep link url to the exporter file version where the metadata is written to.

• POST /api/db?dry_run=1: New query parameter added. This new parameter allows to execute the json unmarshal & plugin steps without actually saving the object. No data is written.

• Allow --clip to be switch on/off in the image:browserthumbs recipe. The reformat recipe was removed. If you are using a custom file worker configuration which uses that recipe, you need to switch to browserthumbs after update and set size to 0.

• /api/settings: Endpoint was extended to show all capabilities available with the current license.

Improved

• For files, always output "eas": { "1": [] }. Before in some cases we would output "eas": { "1": null }.

• /api/pool: Support setting a new _id_parent. This has been supported before, but only half-baked. Now it is fully supported, with re-indexing of all affected objects.

• eas/download: Add trace log with storage url requested for io.Copy. During a bug hunt for an interrupted file delivery we noticed that there is not output of the actual url requested from the storage backend. Added this in trace level, so in situations like this, debugging is easier. The bug mentioned turned out to be a network problem outside fylr's scope.

• _path[]._standard.1.eas: In such renderings, only render first file. This helps reducing the size of the JSON for heavy _standard.eas objects. Multiple eas objects in _path should not really be needed for frontends, so we do not need to output them.

• Collection Sharing: The sharing email now mentions the sharing user name.

• A new environment variable FYLR_CONVERT_VIDEO_MP4_THREADS can be used to limit the number of cores which ffmpeg uses during encoding.

• fylr convert got a new --video-ffmpeg-params which can be used to fine tune the video encoding by ffmpeg. This is available in the recipe video:resize.

• fylr.services.webapp.reverseProxy.custom: Also match the scheme for the custom reverse proxy config. Before the source scheme was ignored.

• Improvements to fylr backup and fylr restore.

• Fixes & improvements for emails, login and register pages.

• /inspect/files got a new form field for filename extension. This page also got a lot faster by an improved loading technique.

• /api/export/list: Optimise file loading so that bigger export lists load a lot quicker.

• File loading got faster: Using a cache column technical_metadata, file loading is much faster in places. Especially if data bases contain files with lots of metadata, this new cache column improves loading speeds for objects containing such files. This new cache column also lowers the memory pressure caused by fylr significantly. In our tests, the data read from the database was reduced to a tenth of the original size (8 MB -> 800 KB). The initial update for that column can take a while during fylr startup.

• Plugins can now use full file data during db_pre_save callbacks. With this it is possible that code in a formula plugin accesses available file information.

• Janitor for session tokens: A new janitor for session tokens cleans up the oauth2_tokens table. This table can grow huge over time and fylr had not cleaned that up in prior versions. The migration step for this release deletes all OAUTH2 tokens. So a re-login will be required for all users after update.

• /api/user: Support for more columns to search for and sort on: state, _last_seen_at. Requires a re-index.

• Changes the recipe replacements %_source.metadata._technical_metadata.KEY% to %_source.technical_metadata.KEY%. Before we supported replacements of all metadata, but that was unused by our recipes.

Fixed

• Rightsmanagement: Checking the standard mask for pool rights was not implemented correctly. fylr would use the standard mask for the pool in which the right was defined, and not the pool in which the right was used.

• /api/search: Fixed wildcard search on analysed fields. If a string "02-abc-def" was used on an analysed field, the splitting, normalising and putting the search together would fail and end up with an empty search, matching all objects. This case was implemented only for strings which yield only one token and otherwise the wrong empty search token was produced.

• Fixed a cache race problem which prevented Sqlite backends from purging properly.

• Improved callback impact from plugins to the API when using Sqlite. Sqlite only supports one writer, so any write transactions during a plugin call are impossible. This patch does not write the users browser language if unnecessary.

• /api/objects: Fix IIIF & HTML formats for models with reverse. When reverse was included in objects for IIIF or HTML rendering, fylr would panic with "langs not set".

• /api/db: Fixed lookup:_id for changed column types. In certain situations with column types changed (e.g. from loca text to normal text), a consecutive lookup could cause a panic and fail.

• Video files: If ffmpegthumbnailer isn't available (like on Windows), videos which are not seek-able failed to produce a thumbnail. Fixed this by using ffmpeg with a fallback to a fixed early frame selection if an error occurs during the "significant change in scene" selection.

• Formatting of MM/YYYY dates in some locales has been fixed.

• Use the fylr system console for Indexer logging. If fylr.elastic.logger was set, the log was directly written to Stderr instead of using our wrapper to pass it to the /inspect/system/console.

• Export: Fixed transports which contain deleted user's emails.

• Startup: If a language und is set without explicitly setting region, date and time format, we would error at startup during base mapping index creation. This is now fixed by setting default languages in such a case. The error is still there and can be found in the logs.

• Remote Addr: Use x-forwarded-for http header to find the address the request is originating from. In most setups, this fixes the IP based group filter. This should fix cases where the group IP filter doesn't work because of a reverse proxy sitting in front of the fylr api service. Even fylr's own reverse proxy didn't get this right and changed the IP address for the group filter to the local callback address.

• /api/search: Fixed sorting of standard info. This was broken in some cases and errored out with a "painless script error" in OpenSearch backends.

• Callback db_pre_save: Improved read back data from plugins. This new code fixes a case where we did not catch an aborted connection from the execserver, cause the returned JSON looked perfectly fine. For some reason the error which is send after the JSON as plain text never makes it to the client. Using an io.ReadAll AFTER the json encoding fixes the problem as we receive an err from that function: "unexpected EOF".

• Improved & fixed cases in Import-XML-Mapping.

• Windows: Fixed a bug when logging in via ldap with "log steps" enabled in the base config. If the ldap entry contained a NUL character, fylr would panic and login would be denied.

• /api/pool: Do not update is_system_pool. The previous versions would set is_system_pool to false even if a system pool was stored. This will be fixed by a migration step. System pool system:root, system:standard, system:none will be set back to is_system_pool: true in case they had been changed.

Frontend

• A new feature has been added to change the thumbnail of videos. This new feature can be found in the video player when viewed from an editor. It allows users to change, view, or reset the thumbnail of the current video.

• Two new buttons have been added for videos in editors, allowing users to navigate the video frame by frame.

• Support has been added in the pool manager to change the parent pool.

• A new option has been added in the group editor for editing nested elements. This new functionality allows finding nested elements that exactly match all the specified values (including empty ones) or only those that match the indicated values.

• Support for the new color profile functionality in the export manager has been added. This new option allows exporting assets by choosing the color profile.

• A new option in the group editor for text fields, append, has been added. This new functionality allows adding text following the text already contained in the fields affected by the edition.

• A new GIF player has been added to the asset browser; now, if the asset being viewed is a GIF or webp, this player can be activated to view the animation in the case of animated files.

• Image loading in "Standard View" mode has been enhanced; the server will now be requested to provide the version that best fits the size of the objects in the search.

• Image loading has been improved using the new function of signatures on them, allowing the frontend to improve the performance of image loading.

• A new option has been added in the export manager to allow exporting all metadata in JSON and XML exports.

• A new fixed field has been added in the metadata export profiles. Deep link URL – if this field is used, the deep link of the object will be exported.

• Improved behavior of nested fields configured as Append only. Now, rows that already existed in the table will be shown as readonly fields, and only adding new fields will be allowed.

• Numerous corrections and adjustments have been made in the CSS.

• A bug in the permission settings for shared collections has been corrected.

• A problem has been corrected where some tags were not displayed correctly in the metadata mapping editor.

• Fixed an issue where the server parameter was sometimes erroneously included in URLs for sharing objects.

• The user type selector in the user manager has been corrected, now correctly displaying SSO and LDAP options.

• Users will no longer be asked to reload the application if a BASE_CONFIG_UPDATE event from a plugin is detected.

• Improvements have been made in the CSS of the pdf-creator to reduce design mismatches between the editor and the final PDF.

Plugins

fylr-plugin-formula-column

• Fixed writing events if the code contained an UTF-8 character like "€".