Release v6.7.0 (2023-10-13)

Published 2023-10-13 10:47:13Z

• fylr_checksums.txt

• fylr_v6.7.0_darwin_amd64.tar.gz

• fylr_v6.7.0_darwin_arm64.tar.gz

• fylr_v6.7.0_linux_amd64.tar.gz

• fylr_v6.7.0_windows_amd64.zip

Important

• License management is now finally activated and there is no opt-out anymore. Please contact our support if you have not yet obtained a license.

• This release brings the new data type geo_json. Elasticsearch supports the new features only with a paid license. As an alternative fylr now supports Opensearch, too.

• A re-index is recommend due to many changes to the index. Go to /inspect/system to trigger that.

• The fylr binary now embeds the webfrontend code and the contents of the resources folder. It is no longer necessary nor advised to reference webapp: path: or resources: in your fylr.yml. These configuration lines should be removed, unless you are a developer and working with fast changing code for the webfrontend.

• fylr does not implicitly default to -c fylr.yml any more. If you relied on this you need to start with fylr server -c fylr.yml.

• The fylr docker image contains an explicit list of easydb plugins now, which are included in our distribution. Before, we used a wildcard to include all easydb plugin. This change assures that only plugins are included which are known to work with fylr. Nevertheless some of the plugins might be subject to additional licensing.

• A Favicon can be uploaded in the base config now.

New

• New data type geo_json. This new data type is compliant with the Geo JSON Standard. This data type is supported in a new _standard.geo.1 collector which can be enabled in the mask management. The /api/search supports new aggregation types geo_bounds, geotile_grid and search filter geo_bounding_box.

• Standard info has a new option stop_if_set which allows the standard rendering to stop at the first set value of a list of fields.

• Mask management has a new option to sort nested by filename of the first asset in the nested.

• Standard info can now also render the file name of the type file in the standard info text.

• /api/event + /api/search: Support search by objecttype and system_object_id.

• Group management allows to auto-assign users based on system groups. Together with the IP filter of that group, groups for intranet and internet users can be created.

• For groups, the IP filter gained an exclude option which reverses the logic of matching.

• Support Opensearch as indexer. Opensearch is a fork of Elasticsearch 7.17.0 after they changed their license away from true Open Source. fylr automatically recognizes Opensearch backends and changes mapping and other queries automatically. In our tests Opensearch performs like Elasticsearch with no noticable differences. We will continue to support Elasticsearch. Opensearch is our new standard backend for our automated test suite, as it has Geo support built-in.

• /api/transition: Added setting who_not to revert the match for users and groups. This allows to define a workflow which is executed if the given users & groups don't match the currently logged in user.

• File worker: We now support importing TIFF with clipping path. If such a clipping path exists, the image is cut out and the background is left transparent. For existing TIFF this requires a re-produce triggered via /inspect/files.

• /api/db can now match nested records partially (if keys are unsent in the payload) in group mode. With this, a nested can be modified while ignoring during columns for the equality check.

• Support sorting by original filename in nested records.

• Support color profiles for thumbnail generation. You need to re-sync your thumbnails for this. The current support automatically uses sRGB_IEC61966-2-1_black_scaled with no user visible option to change this. If you need this in the base config let our support know. This improved colors of thumbnails if the original uses color profiles which are different from sRGB. The underlying fylr convert command gained a --colorprofile option with a bunch of built-in profiles to use.

• /api/config: Support for browser favicon.

• /api/search: Support new subfield .metadata_fulltext for type file. This can be used from inside the query language too.

• Lots of bug fixing for fylr backup and fylr restore will allow more instances to be automatically and easily migrated from easydb 5.

Webfrontend

• Added a checkbox to reverse affected users or groups in workflows.

• Custom icons can now be configured for objects that have no assets; this icon will be shown in the standard view of the object in the search.

Improved

• WMF and other vector formats now support transparency. Affected files need to be re-produced.

• /inspect/users show the mail schedule of a user.

• fylr backup + fylr restore: Fixed and improved many cases where the previous version would not perform well (backup & restore of e.g. publish, xmlmapping, plugins, intra- & internet groups). Also, token renewal is now automatically supported via the Go stdlib. Also, the base config restore will not overwrite default locations, logging & purge settings. Also, reference duplicates for base types are avoided. Restored eas files will contain a reference URI pointing back to the source of the file (url + eas-id + version).

• Prefer attached preview images over embedded base64 encoding. This fixes issues with Outlook and other mail clients which were unable to display embedded images in mail notifications.

• Better language & locale support. New settings in the base config allow for a finer configuration of locale settings. We now separate date, time and number formatting. New options include a number format 1'234'567.89.

• /api/db: Support partial matches in nested records. By not sending a nested column it will be omitted. Before we required all values to match against the current record. Now it is possible to remove e.g. a keywords by ignoring the accompanying description.

• New fylr.debug.inspectEnableElasticIndices enables a page in /inspect/elastic where Elastic indexes can be listed and deleted.

• Setup default logger in fylr.default.yml. Before, the zerolog-Default was used which only outputs the time of an event with no date.

• Export: Relax data export if files are in error state. With this patch, files in error state are no longer skipped, if only their data is exported. Before the export would fail in such cases, leading to unexportable data.

• File production: Do not produce videos bigger than the original size. This was only supported for images before.

• /api/search: Filtering by the metadata of a file (like the PDF text) is now possible using the new subfield .metadata_fulltext. Works in query language too.

• /api/mask: Add custom_settings. This can be used by front ends to store custom information, we have the same for columns and other data.

• The fylr binary now includes the web frontend of fylr. It is no longer necessary to reference this in your fylr.yml.

• fylr does not automatically load a fylr.yml from the same directory the binary is in. It only uses the built-in fylr.default.yml at startup, unless a fylr.yml is explicitly mentioned on the command line (using the -c or --config) option.

• Fixed creating PDF with Microsoft fonts like Arial. They are now included in the fylr docker image.

• Improved error reporting if an S3 volume cannot be connected.

• Windows: Calling soffice for previews of Office documents is now more reliable. We use --safe-mode to not get any hidden UI which could block the file preview production without notice.

• Use pdftoppm from the Poppler utils to render page previews of PDF. This is much faster than the previously used magick convert (which delegated this to ghostscript) and has better quality. Existing PDF need a re-produce if you want to update them.

• The /inspect utility to backup and restore fylr and easydb 5 instances is now called Migration. The URL for that tool was changed to /inspect/migration. This was done for a clearer separation from /inspect/system/backup which is the database dump to storage solution to create backups of the entire database (currently mostly used for support).

• Enhanced documentation in fylr.example.yml.

• /api/db: Fixed removing nested entries with zero values in group mode.

• Login: Don't allow users with unconfirmed emails to login. However, these users can still start the forgot password process. Also, system users are unaffected by this check.

• fylr restore no accepts --base-config - to skip restoring the base config. Furthermore, settings like default_locations, license and others are not restored.

• Improved design & layout of the mails fylr sends out.

• Changed default video sizes to: 360p, 720p, 1080p.

• Allow size configuration up to 10000px for images and videos (before the limit was 2000px).

• /api/plugin/manage: Add date fields created_at, updated_at and last_checked_at.

• Improved error message if storage cannot be connected.

• Introduced new LookupNotFound api error for object, file, tag, collection, group, user lookups. Before this was handled by the generic error handler.

• Support for ARW files (Sony RAW format).

• Metadata mapping supports the date format "2017:05:22 16:07:28Z" which can fix some imports.

• /inspect/files: Re-sync file queuing got a lot faster by using in database only queuing.

• Windows: Make office conversion more reliable. LibreOffice under Windows would sometimes start a UI (despite --headless) and also return immediately to the cmd line without producing the interim PDF which is needed when converting PDF to images. With --norestoreand --safe-mode command line options, we try to work around these problems.

• Auto-detect TIFF clipping path & use when converting to PNG. This cuts out a transparent background for such images.

• /api/config: The default location names now only use name of the location (and no longer the id) for better compatibility in backup & restores as the IDs are unpredictable.

• /api/eas/rput: improved automatic filename detection from the remote HTTP headers. This includes an ISO Latin compatible reading as well as an improved Docker image which now includes mimetypes for improved recognition.

• /api/plugin/manage: Ensure top level dir in an uploaded plugin ZIP is named like plugin. This prerequisite was not checked in previous versions of fylr. The top level directory of a plugin must match it's name. If not, the file delivery fails.

• /api/config: Extend license:info to include a validation block. The new license:info.validation shows information about the compiled license validation. This patch also refuses uploading a license with an unverified signature. Also, uploading the JSON as string is no longer possible.

• /api/schema: Allow to delete object types even if they are used in transitions. This unlinks the object type from the affected transition.

• Index: include facet only fields in index data. Before the index was only getting the field if expert or fulltext was set.

• /api/config: Added company, department, last_name, first_name and phone to the mappable LDAP & SAML fields for user creation and updates.

• /api/search: After adding database language in the baseconfig, a re-index is needed. If none has completed yet, the Elastic throws a typical error about sort problems which confuse users. This patch catches that specific error and returns a specialised error with the name IndexErrorWaitForReindex which can now be localised to inform the user about the problem (re-index not completed).

Webfrontend

• Improved buttons for moving fields in the mask editor; now, if a modifier key (Alt or Cmd) is used + click on "Move to bottom" or "Move to the top," it will not automatically scroll, thus maintaining the mask editor's view.

• Improved shortcuts to object types in the main menu; now, all users who can access at least one object will see the shortcut in the main menu if it is configured.

• Enhanced the pool selector field to display the complete path of the selected pool.

• Improved the JSON importer to create linked objects when the metadata mapping is configured to add metadata to linked objects.

• Improved image loading from URLs that do not contain the file name.

• Improved the Number Input to allow more configurations for separation and decimal characters, enhancing compatibility with some languages.

• Improved how Fylr manages date and number formats; it is now possible to override this configuration from the base config in the language menu.

• Added standard text configuration for EAS fields in the mask editor.

• Made improvements to the license manager.

• Improved the plugin panel to display useful information about plugins.

Fixed

• Video production was fixed to actually down size preview video renderings to the desired height. In 6.6.x the height parameter was not passed through to the converter which resulted in full size videos produced for all sizes. To re-produce videos with the correct size, re-sync via /inspect/files is needed.

• /api/transition: Fixed problems which occurred if workflows contained deleted users in email notifications. These notifications are now ignored.

• /api/transition: Fixed problems with workflows which were saved again before their notifications were sent.

• /api/tags: Allow saving of tags without _acl even if the stored _acl (which are copied to the new tags) contain errors.

• Removal of a mail schedule for a user is now supported.

• Fixed cache expiration for fylr start page and plugins.

• Address some cache race problems.

• Support column type date_range in nested with option nested index set in mask. In such a scenario the built-in data type of Elasticsearch does not support multiple values (as we have in nested records). Fixed this by switching the underlying data type in the index.

• Mapping of dates fixed for files which contained multiple dates in their metadata.

• Support videos with odd heights. The height is adjusted to an even number, in order for the MP4-codec to be able to encode the video.

• Fixed sorting of loca fields: work around a problem with the ICU plugin of Elasticsearch >= 8. For some reason it is not possible to index a single Q or W character. This bug is not present in Opensearch or Elasticsearch 7.x. We filed a bug report for this.

• Backup manager SQL dumps didn't restore the sequence of object_rel correctly. Also use smaller chunks for certain tables to avoid a psql "out of memory" message during restore.

• /api/schema: Fixed upload with mask splitters in mask with nested sort.

• Metadata mapping for date columns was fixed for some cases.

• /api/eas/download: Fixed HTTP header response for range requests. If "range: bytes=" were requested, the response was wrong and off by 1.

• /api/db: Sorting by linked objects in nested which have a date inside the standard output did not work properly. The date inside the linked object was rendered using the localization for the language resulting in date value which sort wrong if sorted by string. Fixed this by rendering a special version of the linked object standard info in case we render it for the nested sorting information.

• /eas/produce: If the produce recipe doesn't support the source format, we output an error message.

• Fixed loading of rights which contain deleted masks. This fixes cases where user were unable to login.

• /api/search: For formats like long_inheritance and standard_extended, include_fields now also used inheritance to compile the values.

• Fixed reading of width and height for HEVC formats.

• /api/plugin/manage: In case the class unknown is not configured as valid upload class, we would reject to install a new plugin. This fix ignores the baseconfig setting for this and allows to import a plugin nevertheless.

• Workflows: fixed tag filter changed check for more than one tag. If a tagfilter contained more than one tag to check the changed status, it would not work correctly.

• /api/db: Fixed reverse nested when set to read only in mask: If a mask defines reverse nested as read only, the db load & merge would fail for reverse nested object versions greater than 1.

• Fixed sequence setting for cases where a payload contained only predefined ids.

• Fixed indexing of multiple date values in masks where the nested records use a nested index type.

• Correctly set transport state in exports to done.

• /api/search: Fixed cases where sorting by localised values would return wrong results. This was particularly visible if the sorting involved numbers (like a year) as the only value in text columns.

• Collections: Don't output _hotfolder_upload_urls, if a user does not have the create_in_collection and link permissions on the collection.

• IIIF tile production was fixed for TIFF and other source formats which would produce a 32 bpp interim bitmap instead of the for tiler supported 24 bpp format. We now force the 24 bpp.

• Export: Custom downloads which set colorspace were fixed.

• /inspect/files with parents active was fixed for Sqlite backends.

• Permissions: If files were in reverse nested, we did not check the permission correctly. The permissions for reverse nested come from the top level object the reverse nested is in. This patch acknowledges that and passes in the correct obj_uuid to the download url. There, the file is matches inside the reverse nested structure and the rights are checked properly.

• Hotfolder: Fixed detection of series in hotfolder. Also, for nested records, improve merging of metadata. Now it's possible to link a file into a nested record while enriching that record with metadata mapped from the file.

Webfrontend

• Fixed errors in the version list of the asset browser in the info menu.

• Fixed an issue when trying to save workflows that contain deleted users.

• Fixed an error in the Presentation Manager when adding new elements.

• Fixed an error in Pool Selectors that prevented pool searches.

• Fixed errors when loading a base config from JSON.

• Fixed an error related to inherited data in the text format of objects.

• Fixed the display of inherited fields in the quick view panel.

• Fixed an error where new workflows could not be saved in Fylr.