> For the complete documentation index, see [llms.txt](https://docs.fylr.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fylr.io/for-developers/user-data-types/text-text_oneline.md).

# text, text\_oneline

`text` and `text_oneline` store characters in one language. In the server, these types are treated the same, text\_oneline is only a hint for frontends on how to display the data. So it is possible (over the API) to use newlines in a text\_online column.

Text are stored as received, so a leading or trailing space is preserved. Frontends are requested to trim texts before sending them to the API.

Text must be encoded in [UTF-8](https://en.wikipedia.org/wiki/UTF-8) and stores in all [normalization forms](https://unicode.org/reports/tr15/#Norm_Forms). There is not limit on the length of storable text.

## API

Text looks like this when send and received over the API:

```json
{
    "text": "this is the text"
}
```

The above example has the value *this is the text* for column `text`.

## Index

Normalization is performed as part of the indexer documentation creation where all text is run through a `icu_normalizer`.

In the indexer, text is stored using a custom analyzer `icu_text`, or `icu_text_folded` when **Remove Diacritics** is switched on for the default analyzer (see [Languages](https://docs.fylr.io/for-administrators/readme/languages.md#indexer)). They work as follows:

```json
{
  "analysis": {
   "icu_text": {
     "type": "custom",
     "tokenizer": "custom_icu_tokenizer",
     "filter": [],
     "char_filter": [
       "icu_normalizer"
     ]
  },
   "icu_text_folded": {
     "type": "custom",
     "tokenizer": "custom_icu_tokenizer",
     "filter": [
       "icu_folding"
     ],
     "char_filter": [
       "icu_normalizer"
     ]
  },
  "tokenizer": {
    "custom_icu_tokenizer": {
      "type": "pattern",
      "pattern": "([[^\\p{L}\\p{Digit}\\uD800-\\uDFFF\\u2F00-\\u2FDF]&&[^&%§\\$€]])"
    }
  }
}
```

Text is normalized using the [`icu_normalizer`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-normalization-charfilter.html) and tokenized into tokens using the above pattern.

What gets included in tokens:

* All alphabetic letters from any language.
* All numeric digits.
* Characters in the Unicode surrogate pair range and Kangxi Radicals.
* Symbols: `&`, `%`, `§`, `$`, and `€`.

What causes token separation:

* Punctuation marks (except the specified symbols).
* Whitespace characters.
* Other symbols and control characters not specified.

The `icu_normalizer` also turns all characters into lower case, so the token *Bär* is stored as *bär*, and a search for *bar* in the field does not find it. With **Remove Diacritics** switched on, the [`icu_folding`](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-folding.html) token filter also removes the diacritics: *Bär* is stored as *bar* and found by *bar* and *bär*. The switch is off by default, and a change applies to existing records only after a re-index.

For the full-text search, text is also indexed with the analyzer of every data language, so the **Remove Diacritics** setting of a data language makes the full-text search ignore diacritics in text too. Multi-language fields (`text_l10n`, `text_l10n_oneline`) are indexed with the analyzer of their data language.

Using the API, searches for text can be performed either on the analysed value (matching the terms), or on the unanalysed value, which is stored alongside with all terms. The unanalysed value stores the data *as is.* There is no normalisation taking place.

All text for indexed documents is split into chunks of 8000 UTF-8 characters. When matching full texts in analysed form, text cannot easily be matched if they exceed 8000 characters.

## Sorting

Sort strings are compiled using the Go library [collate](https://pkg.go.dev/golang.org/x/text/collate). It uses the first configured database language as assumption in what language the text is in. Numbers are recognised so that *Car 100* sorts after *Car 11*. Text is normalised by the collate library. Internally we use the hex representation of that string to work around anomalies in Elasticsearch. Some special replacement is [always done](https://docs.fylr.io/for-developers/user-data-types/text_l10n-text_l10n_oneline.md#sorting).

## Export

Text is exported as is, keeping spaces & normalisation.

The **XML export** looks like this for a column names `title` and a value *Title* for type `text_oneline`. The `column-api-id` in this example *29*.

```xml
<title type="text_oneline" column-api-id="29">Title</title>
```

For type text type is `text`.

Output in **CSV** is as is, same for **JSON**.
