Heuristic:PrefectHQ Prefect HTTP Connection Pool Tuning
| Knowledge Sources | |
|---|---|
| Domains | Networking, Optimization |
| Last Updated | 2026-02-09 22:00 GMT |
Overview
Limit HTTP connection pool to 16 max connections with 25s keepalive expiry to prevent instability with the Prefect Cloud load balancer.
Description
The Prefect orchestration client uses `httpx` for all API communication. Through production experience, the team discovered that allowing the client to open too many simultaneous connections causes instability. The pool is capped at 16 max connections and 8 keepalive connections. The keepalive expiry is set to 25 seconds, which is 5 seconds shorter than the Prefect Cloud load balancer's 30-second connection timeout. This prevents the client from trying to reuse a connection that the load balancer has already closed.
Usage
Apply this heuristic when configuring HTTP clients that communicate with the Prefect API server, especially through load balancers. This is relevant for any custom client configuration or when troubleshooting connection-related errors (unexpected closes, connection resets, timeout errors during high-concurrency flows).
The Insight (Rule of Thumb)
- Action: Configure httpx connection pool limits and keepalive expiry on the Prefect client.
- Value: `max_connections=16`, `max_keepalive_connections=8`, `keepalive_expiry=25` seconds.
- Trade-off: Lower max connections may throttle throughput under very high concurrency, but prevents connection instability. The 25s keepalive (vs 30s LB timeout) adds a 5-second safety buffer.
- Key Insight: Always set client keepalive expiry shorter than the load balancer timeout to avoid reusing already-closed connections.
Reasoning
The Prefect Cloud load balancer keeps connections alive for 30 seconds. If the client's keepalive exceeds this, it may attempt to send a request on a connection the LB has already terminated, causing unexpected connection errors. The 16-connection limit was determined empirically; the team observed instability (connection resets, timeouts) when allowing unlimited connections. HTTP/2 support is also available but negotiated per-server, further reducing the need for many parallel connections.
Code evidence from `src/prefect/client/orchestration/__init__.py:376-388`:
httpx_settings.setdefault(
"limits",
httpx.Limits(
# We see instability when allowing the client to open many connections at once.
# Limiting concurrency results in more stable performance.
max_connections=16,
max_keepalive_connections=8,
# The Prefect Cloud LB will keep connections alive for 30s.
# Only allow the client to keep connections alive for 25s.
keepalive_expiry=25,
),
)