# ai metadata plugin

The ai metadata plugin sends images in a fylr instance to a LLM like ChatGPT to write the generated texts into user defined fields in fylr objects.

### Plugin description

The **ai-metadata plugin** enables the integration of LLMs compliant with the [Chat Completions API](https://developers.openai.com/api/reference/chat-completions/overview) by OpenAI into fylr. Custom prompts can be configured and mapped to fylr Object Types through metadata mappings. The generated output is written into objects during their creation or during the execution of a (scheduled) background task.

### Datamodel requirements

To successfully use the ai-metadata plugin to describe your instances content, your Object Type needs to have

* **a file field**: the source image for the LLM to describe
* **text fields or lists**: (multilingual) text fields to map the LLM output to
* also supported: dates, boolean fields

### Installation

{% hint style="info" %}
**This plugin is a paid feature.** The plugin is only available after a valid license has been purchased.

Please contact us to obtain the license.
{% endhint %}

Move to your fylr instances Plugin Manager to install the ai-metadata plugin by finding the <img src="/files/kwLgzERZBeiUdaOoiht0" alt="" data-size="line">-Icons in the bottom of the plugin list and clicking the plus symbol.

While installing enabling the plugin is the default behavior. You can enable or disable the plugin anytime, active configuration will be preserved until enabled again.

<figure><img src="/files/flbUHRV1V69FX4F12RWv" alt="" width="343"><figcaption><p>An ongoing ZIP installation of the ai-metadata plugin</p></figcaption></figure>

Install the plugin by release URL to always get the latest updates, after contacting support you will receive an installation URL instead of a ZIP.

{% hint style="info" %}
For the setup of this plugin **root** permissions are suggested.
{% endhint %}

### Configuration

After installing and activating the plugin, select it in the plugin list.

Configure the plugin in the "General Tab" by adding an API Key

<table><thead><tr><th width="240.87109375">Setting</th><th>Description</th></tr></thead><tbody><tr><td><strong>Extended protocol</strong></td><td>Additional debug context in FILE_METADATA events</td></tr><tr><td><strong>API Type</strong></td><td>OPEN_AI / AZURE: choose Azure if you want to connect to your own LLM URL.</td></tr><tr><td><strong>Base URL</strong> </td><td><p>OPEN_AI:</p><p>Leave empty if using API Type OPEN_AI (default fallback)</p><p></p><p>AZURE / others:</p><ul><li>example: <code>https://{{ressource}}.openai.azure.com</code></li><li>Could be any Endpoint compliant with the <a href="https://developers.openai.com/api/reference/chat-completions/overview">Chat Completions API</a></li></ul></td></tr><tr><td><strong>API Key (Required)</strong></td><td>Enter your API key. Without API Key, requests will be denied. </td></tr><tr><td><strong>API Version</strong> (only with API Type AZURE)</td><td>Use the current version of your LLM</td></tr><tr><td><p><strong>Model</strong></p><ul><li><strong>free entry (deployment name)</strong></li><li><strong>list</strong></li></ul></td><td><p></p><ul><li>free entry: input your deployemt name</li><li>list: select from the list of OpenAI models</li></ul></td></tr><tr><td><strong>Used image size</strong></td><td>Controls the resolution of images sent to the LLM. The default of 600px is sufficient for most use cases.</td></tr><tr><td><strong>Instructions for the AI</strong></td><td><p>Describe the AI’s general style, role, and behavior here. These instructions apply to all answers, for example tone, perspective, or fixed phrasing rules. Do not enter the actual question here.</p><p><strong>Note:</strong> does currently not work with API Type AZURE</p></td></tr></tbody></table>

{% hint style="warning" %}
**It's currently not possible to save more than one LLM configuration.**
{% endhint %}

### Creating the prompts

Inside the plugin managers config tab, create one or more **named prompts** by clicking <img src="/files/sMUfgHb2N7NnhAZeARcE" alt="" data-size="line">.

The available inputs are mandatory.

<table><thead><tr><th width="141.953125">Setting</th><th width="302.953125">Description</th><th>Options</th></tr></thead><tbody><tr><td>Name</td><td>Name your prompt. This name will be used later to refer to this particular prompt.</td><td>-</td></tr><tr><td>Prompt</td><td>Your prompt to the LLM</td><td>-</td></tr><tr><td>Type</td><td>Output type</td><td><ul><li><em>Single line text</em></li><li><em>Single line text (multilingual)</em></li><li><em>List text</em></li><li><em>List text (multilingual)</em></li><li><em>Boolean</em></li><li><em>Date (ISO 8601)</em></li></ul></td></tr></tbody></table>

#### Choosing the Output Type

{% hint style="info" %}
**The output type matters when mapping the prompt to the object type later**, make sure you select the option that fits your needs and matches your object types fields.

If it doesn't, edit your datamodel accordingly and continue afterwards.
{% endhint %}

* *Single line text*
  * maps to[All Data Types](/for-administrators/tools/csv-importer/examples/all-data-types.md#single-line--multi-line-text-simple-text-string)
* *Single line text (multilingual)*
  * maps to [All Data Types](/for-administrators/tools/csv-importer/examples/all-data-types.md#single-line-and-multiline-text-multilingual)
* *List text*
  * maps to [All Data Types](/for-administrators/tools/csv-importer/examples/all-data-types.md#single-line--multi-line-text-simple-text-string) in a nested field.
* *List text (multilingual)*
  * maps to [All Data Types](/for-administrators/tools/csv-importer/examples/all-data-types.md#single-line-and-multiline-text-multilingual) in a nested field.
* *Boolean*
  * maps to Boolean field
* *Date (ISO 8601)*&#x20;
  * maps to [All Data Types](/for-administrators/tools/csv-importer/examples/all-data-types.md#date-date--time)

#### Examples

<figure><img src="/files/waVHFpB2oToGwfokTDOe" alt="Example prompts created for the ai-metadata plugin. The prompts display the usage of the different output type options while showcasing prompts in various complexities."><figcaption><p>Example prompts created for the ai-metadata plugin. The prompts display the usage of the different output type options while showcasing prompts in various complexities.</p></figcaption></figure>

{% hint style="info" %}
**Multilingual fields:** Make sure to include the languages of your expected output in your prompt.
{% endhint %}

Each named list entry in the Plugin Managers configuration tab of the ai-metadata plugin becomes available in the **Metadata Mapping** section of fylr. (see in *sidebar > Administration*)

As a start, you can upload the displayed configuration and modify as needed.

{% file src="/files/qCLexdWa4sdd54ozmh3s" %}

### Creating the Metadata Mapping

Create a new **IMPORT Mapping** by finding the <img src="/files/kwLgzERZBeiUdaOoiht0" alt="" data-size="line">-Icons in the bottom of the available Mappings and clicking the plus symbol, selecting IMPORT.

<table><thead><tr><th width="149.2109375">Option</th><th>Description</th></tr></thead><tbody><tr><td>Object Type</td><td>Select an Object Type to apply the mapping to</td></tr><tr><td>Name</td><td>Provide a descriptive name for your mapping</td></tr></tbody></table>

If the plugin is enabled and a prompt is configured in the previous step, you should now see the section **“ChatGPT Metadata”** listing your created prompts.

Per default all Object Types are listed, in the below example, only the fields for the selected Object Type are shown because the object type limits the mapping to be used on OT "ChatGPT Object"

To map the available fields to the created prompts, drag and drop the individual fields onto your created mapping. Right click a mapping to remove.

<figure><img src="/files/EeHekvMSAYjD5Vq42Iyg" alt="Example of a created Metadata Mapping in fylr. To map an object types fields to our created ChatGPT Prompts drag and drop the individual fields onto your created mapping."><figcaption><p>Example of a created Metadata Mapping in fylr using drag and drop.</p></figcaption></figure>

{% hint style="info" %}
Right-click on a mapping to remove it. Duplicate Mappings are allowed.
{% endhint %}

**Reload the frontend to make the mapping available for upload dialogs and record creation.**

### Usage while creating records

When creating a new record, select the metadata mapping that includes your prompt configuration.

The mapping can be used

* during **upload via the plus icon**

<figure><img src="/files/hseQawMz0VVngecmshrc" alt="Create New Records view in fylr app. Highlighted are our choosen Object Type and the metadata mapping using the ChatGPT prompts"><figcaption></figcaption></figure>

When clicking **Next**, you will see the Metadata Mapping being applied with a progress indicator.

<figure><img src="/files/62JpawzT9S3yx4qe9aqG" alt="" width="305"><figcaption><p>Metadata mapping status progress</p></figcaption></figure>

In the next view, see the prompt results arranged into your fields. After saving, the records will be accessible just like any other record.

{% hint style="info" %}
**Upload collections** can be configured to apply a custom metadata mapping to be applied to all newly added records
{% endhint %}

If not satisfied with the results, after updating your prompts with closer alignment to your requirements, restart the import process or continue reading to apply the mapping with a background task.

***

## Usage with Background Tasks

To run the **ai-metadata plugin** in background tasks, follow these steps:

### Create and Configure the Task

1. Open the **Background Tasks** section in the header bar of your fylr instance.

<figure><img src="/files/Yi5vNYLTEUvbgDhORk8R" alt="background tasks manager located in the header bar of fylr" width="375"><figcaption><p>background tasks manager located in the header bar of fylr</p></figcaption></figure>

2. Create a **new task** by using the <img src="/files/kwLgzERZBeiUdaOoiht0" alt="" data-size="line">-Icon and selecting the module **Metadata**
3. Define the schedule:

* **Manual time** (default: now)
* **Scheduled time** (future execution)

### Task Parameters

#### Configure the mapping

<figure><img src="/files/23KMuxWr2JVgDWL5wDGj" alt="" width="563"><figcaption><p>Example mapping configuration for using the ai-metadata plugin in a background task</p></figcaption></figure>

| Parameter            | Description                                                                         |
| -------------------- | ----------------------------------------------------------------------------------- |
| **Object Type**      | Apply only to the specified object type.                                            |
| **Metadata Mapping** | Select the metadata mapping to be applied.                                          |
| **Pool**             | Used when creating linked records with pool management during metadata mapping.     |
| **Mask**             | Define which mask to use for the task.                                              |
| **Field for Files**  | If multiple file fields exist, specify which one to use as source for yourn prompt. |

#### Creating a search result to apply the mapping to

For the task to have records to apply the mapping to, configure a search that finds the records you want to be filled by the LLM (selection of records / the entire search result).

{% hint style="warning" %}
If individual records are selected, only those will be mapped.

If an entire search result is selected using **"Select All", the same search will be reused** for the next scheduled run of the background task.
{% endhint %}

{% hint style="danger" %}
**Warning: An empty search will result in the background-task sending ALL records to the LLM. This may or may not result in a high usage of tokens.**
{% endhint %}

<figure><img src="/files/IAF7aMnJREBhWeActy13" alt="Example configuration of a search query showing results to be mapped by the ai-metadata plugin using the bg-tasks." width="563"><figcaption><p>Example configuration of a search query showing results to be mapped by the ai-metadata plugin using the bg-tasks.</p></figcaption></figure>

If the found records have no values in their fields yet, the **Override Values** checkbox might not be required.

After the created tasks next scheduled job has finished (now or later, depending on the amount of records to be mapped), confirm the changes made in the tasks log and the record itself.

### Customizable variables in prompts

Create customizable variables used in your prompts to be filled later during usage.

Configuration of a variable:

* **Name (string):**&#x20;
  * the name of your variable, will be visible later during usage. **No empty space characters allowed.**
  * Use the variable in your prompts by referencing it using the this syntax: `%variables.NAME%`
* **Description (string):** will be used as a label to your variables input
* **Changeable by user (boolean):** users can set the value of the variable during upload or while setting up a background task

### Simple usage examples

#### A customisable prompt to be created during usage

1. A variable prompt to be only defined during usage, not in the plugins prompt configuration:

<figure><img src="/files/441hFnaCZjmJipSnwnOS" alt=""><figcaption></figcaption></figure>

2. Create a placeholder prompt using only the variables content as the full prompt to be send to the LLM

<figure><img src="/files/7TG7TnISs3u8tSp4qEYB" alt=""><figcaption></figcaption></figure>

3. Create a metadata mapping that uses the the placeholder prompt to connect your variable is input for your placeholder prompt to your object types fields.
4. When selecting the metadata mapping that's using the placeholder prompt, notice the cog wheel. In the input behind the cog wheel provide a value for the variable (here: the entire prompt!)

<figure><img src="/files/r8jo7qQnBnOBlgLpPA8U" alt=""><figcaption><p>An example dialogue when creating a new record. A metadata mapping is selected, thus the parameter input is enabled (see cog-wheel icon)</p></figcaption></figure>

<figure><img src="/files/AbUO2n0lABI5i7FcNTNX" alt=""><figcaption><p>Providing the variable value during creation of a new record.</p></figcaption></figure>

#### A variable to control the length of a LLM response

**Setup:**

<figure><img src="/files/aNXzyv2yU54iYlbVrNoq" alt=""><figcaption><p>Example Setup of a variable controlling the length of a LLM response </p></figcaption></figure>

Reference this variable in another prompt:

<figure><img src="/files/fIQ682YStLtP7WFg4p8l" alt=""><figcaption><p>Usage of the amout_of_words variable in a LLM prompt</p></figcaption></figure>

Now during configuration of a prompt a user can control the amount of words used in the LLMs response.

{% hint style="info" %}
Those are examples with the goal to showcase the setup of the variable system in the ai-metadata plugin.&#x20;

The LLMs responses are only as good as the prompts provided.
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fylr.io/tutorials/ai-metadata-plugin.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.