Release v6.6.0 (2023-08-03)

Published 2023-08-03 13:48:50Z

• fylr_checksums.txt

• fylr_v6.6.0_darwin_amd64.tar.gz

• fylr_v6.6.0_darwin_arm64.tar.gz

• fylr_v6.6.0_linux_amd64.tar.gz

• fylr_v6.6.0_windows_amd64.zip

Important

The big changes in this release are:

• Custom produce configs in the base configuration are no longer valid, this update resets them to the defaults. If you have configured your own versions, you must re-configure them after the update.

• All existing file versions get a new produce hash, so fylr considers all versions outdated and wants them re-computed. This is not done automatically and will only happen if a re-sync is triggered (via /inspect/files).

• Removed fylr_* utils and incorporated them into fylr as subcommands. Use fylr -h for a full overview of the new capabilities. This change requires a non fylr+ YML to be changed!. The fylr.default.yml has been changed accordingly.

• A new subcommand fylr convert is used to simplify our recipes and be able to make file preview renderings more powerful and better in a future release. In order for this to work, you must add a block to you fylr.yml. If you use the fylr.default.yml as basis (and that we highly recommend), you don't need to change anything. You must add a command fylr (see below).

Blocks like this, should be removed and merged into a new top level setting in fylr.services.execserver:

Before:

fylr:
  services:
    execserver:
      services:
        "NAME":
          commands:
            fylr_metadata:
              env:
                - FYLR_METADATA_BLURHASH=10M
              prog: "fylr_metadata"
            fylr_pdf2pages:
              prog: "fylr_pdf2pages"
            fylr_iiif:
              prog: "fylr_iiif"

After:

fylr:
  services:
    execserver:
      # this is new, a global env section
      env:
        - FYLR_METADATA_BLURHASH=10M
      # this is new, a global commands section
      commands:
        # it is essential to add "fylr" here, so the "fylr convert" and other tools can be called
        fylr:
          prog: fylr
        exiftool:
          prog: exiftool
        ffmpeg:
          prog: ffmpeg
        inkscape:
          prog: inkscape
        soffice:
          prog: soffice
        pdftotext:
          prog: pdftotext
        magick:
          prog: magick
        ffprobe:
          prog: ffprobe
        ffmpegthumbnailer:
          prog: ffmpegthumbnailer
        node:
          prog: node
        python3:
          prog: python3
        xsltproc:
          prog: xsltproc
        # used by fylr-plugin-example (Tesseract)
        java:
          prog: java
      services:
        "NAME":
            # this still works
           commands:
              ...         

• A re-index is recommended for all uses, Go to /inspect/system to start this after the update.

• inkscape needs to be installed for WMF support.

• pdftotext needs to be installed for PDF text metadata support.

• java is pre-defined in fylr.default.yml but is currently not needed by fylr, as long as you don't play with the fylr-plugin-example. However, in a future release we will require this to support OCR features.

If you are struggling with getting your fylr.yml right, you can use fylr server --dump-config to output the current config as fylr compiles it. If fylr is running, you can use /inspect/config for a deep dive into your settings.

New

• File previews: Added support for WMF file format. The conversion of this format requires inkscape as a helper program.

• File previews: Support for RAW file formats: CR2, NEF, DNG and RAF. The preview generated is using the metadata embedded thumbnail of the RAW file.

• PDF support: We added text search support for PDF. This uses embedded text from the PDF, this is not an OCR feature. However, most PDFs include the text they show. The text is added to the fulltext index information of the objects which link the PDF. Existing PDF need to re-sync their metadata for the text to be indexed.

• Synchronise fylr backends. fylr now keeps a list of running instances in a database table. With that, cleanups for index and file queues can be performed at startup.

• /api/plugin: Renamed /plugin/base to /plugin/extension. This new name of the path reflects much better what this endpoint does: It serves the api extensions provided by the plugin. The plugin manifest.yml also uses extension: to configure these.

• Removed user search languages: This setting was only confusing most users, so now the database languages of a user are also used for searching.

• DELETE /api/db: This endpoint gained the capability of setting references to NULL (if allowed) where the deleted objects would be in use. This is communicated to the user using a 202 response.

• Execserver: Add /healthz endpoint.

• /api/objecttype: A new setting empty_placeholder_icon can be stored. This icon can be used to display objects of that type if no asset is available or wanted.

• fylr convert replaces all helper programs in standard recipes. Also, fylr_metadata, fylr_pdf2pages and fylr_iiif have been incorporated into the fylr binary. This will help us to support file formats even better in the future.

Improved

• backup/restore: Improvements and bug fixes Include publications, mappings, plugins (fylr only) and some baseconfig .

• Improvements to pages and emails. The emails in this release look much better than before and are localised.

• Collections: Support for from and to dates for ACL items. Now it is possible to set expiration and or future start dates for sharing.

• Usermanagement: For sso and ldap users no passwords can be stored. For other system users, such as oai_pmh and deep_link we allow this. Also, system rights can be stored for these users, so it is possible to actually login as oai_pmh user to configure and check their permissions more easily.

• /api/eas: We no allow access to the id if at least one version is accessible. That is that same amount of data one sees if the file is embedded in an object.

• fylr.yml: fylr.api.oauth2Server.<CLIENT>.redirectURIs for client fylr-web-frontend are not set per default anymore, but auto-set (if empty) to fylr.externalURL. This simplifies configuration of OAUTH2.

• WebDAV: Removed Windows-WebDAV-URLs from collection settings. This URLs do not work anymore on more recent Windows versions, and Windows does support the http:// based URLs to connect to WebDAV.

• /api/schema/commit: Improved re-index check. The improved checks include changes to standard, re-ordered fields and type changeswhen suggesting a re-index to the user. We also suggest re-index if mask where deleted or added.

• Renamed login of system users, to not start with system:. Unlike root the other system users oai_pmh, deep_link, deleted_user started their login with system:, which looks weird in search results from the logins of these users. The reference is kept to start with system:. This affects updates as well as new setups.

• /eas/mapping: Support merging of nested. If a nested already exist with a value other than the to be merged value, the same record is used. This fixes mapping of nested records with more than one column.

• /api: Timestamps gained better support when scanning during inject. We use the ISO8601 parser to parse regular timestamps, output for timestamps is RFC3339.

• /api/eas/rput: Change filename detection. Now, if a filename is given and has an extension we do not check the remote to improve the filename. Before, if the file class was unknown to our recipes we would add the remotely delivered extension to the provided filename. This is a surprising behaviour and caused problem with restoring .xslt files from easydb 5. Those files are delivered as .html.

• Embed all /resources into the fylr binary. It is now optional to configure fylr.resources. This setting is only useful during development now. However, the web frontend is still not part of the fylr binary and need to exist on disk and configured in fylr.yml.

• Use the stdlib token refresh capabilities in go-client library. This helps refreshing tokens during long running backups using fylr backup.

• /api/search: Support searching of user.email.

• Base config: When email configuration has set No TLS we now really do not attempt to use STARTTLS during a connection to the email server.

• Improved error handling during exec server plugin callback read backs. If a plugin reported an error, we would fail to deliver this properly to the caller.

• Base config: Improved user visible error messages.

• /api/search: Improved type text wildcard searches. If a text is analysed and generated multiple tokens and then search with multiple tokens and wildcard, we throw an error. This combination is unsearchable using elastic. Also, throw an error is mode "wildcard" is used with query "match" or "match_phrase", both combinations are not supported.

• File previews: An aac file version for more audio formats. This can be used to "preview" the audio file in a browser.

• HTML pages like login and register gained localised error support.

• Metadata mapping: Enable list mode for XMP-iptcCore:Scene tag.

• /api/db: Faster dependent object calculation for re-index. This patch bring a faster and more accurate algorithm for calculating which objects need to be re-indexed if objects are updated. The way to determine related objects for indexing has been refactored. We now use a cache table based system (table "object_rel") to find related objects which needs re-indexing when objects are saved. The old algorithm would not catch all cases, and would already fail in easy reverse cases with an top level obj connection OT A and OT B (both reverse). B would like to C. C.file was changed without A being re-indexed.

• Startup: re-queue stale jobs: The index and file queues can now be cleared from stale jobs. These jobs can be left-overs from a shutdown while the job was still processing. All jobs which timestamp is older than the oldest fylr running, will be set back to state NEW and can be picked up from a worker again.

• /inspect/objects: Output api column names in title of label in the HTML rendering.

Fixed

• Bidirectional records: Fixed cases where an incomplete bidirectional record would not save properly.

• /inspect/files: FIxed re-sync originals.

• fylr.yml on Windows: Fixed fylr.stdErrFile when installed as a service and using a relative path.

• backup: Fixed some SQL commands to support a proper restore.

• Execserver: The log level was not changed when changed via baseconfig, it was set only at startup.

• Rights management: Fix for permissions on versions. Before the class +version was compiled using the class of the version and not of the original. So, for a version "office.preview" fylr would check the right for "image.preview" which was wrong.

• PDF: PDF production was fixed so that the frontend viewer works better now. It might be necessary to re-sync PDF originals.

• Index: Fixed re-index of objects with files with class unknown. Before such objects would never get re-indexed (on changes) because of a fast path of file sync with class unknown.

• Janitor: Fixed and improvements for deletion of events. The previous code would never really delete huge amounts of events (like 1M+), so expect a busy cleanup with this update. We now a clean up job until no more todos are encountered.

• Sorting by localised values for user types and base types got confused if not all languages were set. Now the sorting acts smarter to sort by the best value in the order of the user's configured database languages.

• /api/objecttype: Added missing support for tag filters in column filters.

• /api/tags: Store order of tags when stored. This fixed various issues we had with our frontend when storing ACL for tags. For most parts this was not working correctly.

• Removed obsolete /docs endpoint. Our documentation has moved to docs.fylr.io

• Don't crash with empty fylr.externalURL. This setting can be omitted for exec server only configurations.

• /api/event: Add objecttype to CSV exports.

• Rights management: More checks when _owner is changed.

• Backend: Improve mime type support for disk stored assets This patch makes better use of the storage layer by delivering the detected mime type to the layer at put time. Before this was left to auto-detection and written correctly from file only for http storage without redirect. So, disk storage and S3 remote delivery where not setting our detected mime type correctly. This is especially important for SVG, which is detected as text/xml by the Go stdlib, but that is not sufficient for browsers to display this correctly when referenced inside an IMG src.

• Concurrent map write fix: If accessing /api/config very frequently such a panic could occur and crash fylr.

• Janitor: Fixed deleting of users who are _owner of another another user.

• /inspect/system/stats: The statistics were only compiled for the first 10 object types.

• /api/db: Omit _path from output if mask has all path output configured to off. The system field which steers the output of path(s) was not reflected over the API. Now, if all output if turned off no _path or _paths is output in the api response.

• /api/search: _path,_system_object_is is not allowed in fields. Fixed this to report an error, also changed that error to a featured error.

• /api/db: Accept (and ignore) _mapped_metadata_created during inject. This key is used by fylr to indicate that metadata originates from a on-the-fly mapping of objects. The data may end up in an inject and can be safely ignored. Before the API would throw an error about an unknown key. Fixes a case reported for the JSON importer.

• Collection WebDAV hotfolder: Fixed metadata mapping, if configured.

• Workflows: Send user configured emails if a transition is rejected. Previously these emails were forgotten and not sent.

• /api/plugin: Fixed bundling code for plugins by excluding plugins which depend on others and were the other plugin wasn't installed. This fixes console errors in the web frontend during startup.

• /api/search: Fixed sorting by user.generated_display_name. This needs a new Elastic base mapping and thus an index rebuild.