> For the complete documentation index, see [llms.txt](https://docs.fylr.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.fylr.io/for-system-administrators/symptom-and-solution/too-many-s3-connections.md).

# too many S3 connections

Where fylr's connections to an S3 storage location come from, what limits apply, and how to count them

## Symptoms

* The S3 provider reports that a connection limit for your account or source IP is being reached.
* Under load — bulk imports, mass rendition production, large exports — uploads, renditions or downloads fail sporadically.
* Log messages about connection resets, timeouts or `context canceled` against the S3 endpoint.

## Where the connections come from

Connections to an S3 storage location are **not** all opened by the fylr process. Depending on your setup up to two different hosts talk to the S3 endpoint:

| Traffic                                                                                                                        | Opened by               | Goes to                                           |
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | ------------------------------------------------- |
| fylr's own reads and writes: file janitor, copies between locations, exports, every upload arriving at fylr and every download | the **fylr** process    | S3 endpoint                                       |
| Fetching the source file of a rendition job (`%_source.url%`)                                                                  | the **execserver** host | S3 endpoint (presigned GET)                       |
| Storing a produced rendition (`%_target.url%`)                                                                                 | the **execserver**      | fylr's backend — **not** S3; fylr writes it to S3 |

If the execserver runs on a separate machine, counting on the fylr host alone under-reports. Count on both.

In version 6.35.0, an S3 location with `allow_redirect: true` sent downloads from the browser straight to S3. The option is removed from version 6.35.1: every download is served by fylr and counts against fylr's own connections to S3.

## Which limits apply

fylr uses the AWS SDK for Go v2 with its default HTTP transport for every S3 storage location. Per fylr process and per storage location:

| Setting                           | Value |
| --------------------------------- | ----- |
| Maximum connections per host      | 2048  |
| Idle connections kept in the pool | 10    |
| Idle connection timeout           | 90 s  |

The important one is the second: **only about ten connections are pooled.** Above roughly ten concurrent S3 requests, every further request opens a fresh connection and closes it once the request is done. Under sustained load this produces a lot of connection churn, and each closed connection then sits in `TIME_WAIT` for 60 seconds on Linux.

{% hint style="info" %}
If your provider's limit counts **sockets** rather than **established connections**, the number they see can be several times the actual concurrency, because most of those sockets are in `TIME_WAIT` and no longer carrying data. Clarify which one is counted before you ask for a higher limit — see the state breakdown below.
{% endhint %}

There is no fylr configuration option to cap or widen the S3 connection pool. `fylr.debug.http` tunes fylr's general purpose outbound HTTP client, not the S3 client of a storage location. The `fylr.execserver.parallel` worker count is deprecated and no longer limits file work; the file dispatcher derives its admission from the capacity of the connected execservers and the CPU count.

## Counting the connections

Run these on the fylr host, and on the execserver host if it is a separate machine. All of them resolve the endpoint name to its current IP addresses and match the peer address against them, so they stay correct when the endpoint is DNS round-robin.

A point in time count of established connections:

```bash
S3HOST=s3.example.com; ss -Htn state established | grep -cE "$(getent ahostsv4 $S3HOST | awk '{print $1}' | sort -u | paste -sd'|' -)"
```

A live sampler that also tracks the peak — run this during a bulk import to find out what your instance actually needs:

```bash
S3HOST=s3.example.com; peak=0; while :; do RE=$(getent ahostsv4 "$S3HOST" | awk '{print $1}' | sort -u | paste -sd'|' -); est=$(ss -Htn state established | grep -cE "$RE"); all=$(ss -Htan | grep -cE "$RE"); (( est > peak )) && peak=$est; printf '%s established=%-5s all_sockets=%-5s peak=%s\n' "$(date +%T)" "$est" "$all" "$peak"; sleep 1; done
```

A breakdown by TCP state. This is the one that shows whether a reported number is real concurrency or mostly `TIME_WAIT`:

```bash
S3HOST=s3.example.com; ss -Htan | grep -E "$(getent ahostsv4 $S3HOST | awk '{print $1}' | sort -u | paste -sd'|' -)" | awk '{print $1}' | sort | uniq -c | sort -rn
```

### If fylr runs in Docker

With the default bridge networking the connections live in the container's network namespace, so `ss` on the host does not see them. Enter the namespace instead of using `docker exec` — the fylr image does not ship `ss`:

```bash
nsenter -t "$(docker inspect -f '{{.State.Pid}}' fylr)" -n ss -Htan state established
```

On Kubernetes the same works from the node, using the PID of the container process.

## Reducing the number

* **Run fewer file jobs at once.** Since 6.35 the file dispatcher sizes itself from the slots of the connected execservers and the CPU count: cap it with `fylr.execserver.maxInFlight` (the most file-queue items one fylr works on at once, `0` = automatic), or give the execservers fewer `slots`. Set `fylr.execserver.parallel: 0` on nodes that should not do file work at all (the one value of that key that still has an effect since 6.35). Fewer concurrent rendition jobs means fewer simultaneous fetches from S3.
* **Ask what is being counted.** A limit expressed in sockets and a limit expressed in established connections are very different numbers for the same workload.