stack: HTTP gateway opens a new upstream connection per request and exhausts ephemeral ports under load

[avallete]

Affected area

Local development

Supabase CLI version

develop @ 3704383 (experimental stack runtime)

Operating system

macOS + Docker Desktop

Command

SUPABASE_EXPERIMENTAL_STACK=1 supabase start --runtime docker

Actual output

Under sustained concurrent load, the stack's HTTP gateway (the host-side proxy behind API_URL) returns 502 Bad Gateway in waves. ~/.supabase/stacks/<stack-id>/owner.log shows:

Route rest GET upstream failed before responding, retrying
Route rest request failed { message: "socket hang up", code: "ECONNRESET", _tag: "HttpProxyError", responded: false }

The gateway opens a new upstream TCP connection for every proxied request. Each closed connection sits in TIME_WAIT, so at a few hundred requests per second the ephemeral port range fills up (macOS: 49152–65535). Once it is exhausted, upstream connects are reset and the gateway answers 502 until TIME_WAIT drains, then the cycle repeats.

Expected behavior

The gateway reuses keep-alive connections to each upstream, so sustained load does not exhaust ephemeral ports and requests keep succeeding.

Steps to reproduce

1. SUPABASE_EXPERIMENTAL_STACK=1 supabase start --runtime docker
2. Run 24 concurrent client loops for 120 s, each doing GET <API_URL>/rest/v1/ with the service-role key (roughly 480 req/s in total).
3. Sample netstat -an -p tcp | grep "127.0.0.1.<rest published port> " | grep -c TIME_WAIT every few seconds.

Observed: ~7.3k TIME_WAIT sockets after 20 s and ~15.5k after 40 s, which is the whole ephemeral range. About 64% of requests returned 502, clustered in windows (20–60 s and 100–120 s) that match the TIME_WAIT peaks. Storage routes behave the same way.

Control: hitting the service's published port directly with a keep-alive client at the same load produced zero errors, so the upstream services are fine.

Additional context

Root cause is in packages/stack/src/HttpProxy.ts: forward calls http.request with agent: false, which opens and closes one connection per request.

agent: false was introduced in #6897 because pooled connections could be reset by a just-woken backend (Studio on Docker), and only safe, bodyless requests are retried, so POSTs (e.g. MCP calls) surfaced the reset as a 502. A fix needs to keep that case working:

• Reuse a pooled keep-alive agent per upstream with a bounded socket count, applied in both native and container runtimes.
• Handle a reset on a reused socket before any response bytes arrive (the stale keep-alive race) by retrying on a fresh connection, without letting retries multiply new connections.
• Keep Connection: keep-alive on the wire (Bun ends a streamed upstream response early otherwise).
• Add an integration test that pushes a few thousand requests through a route and asserts the number of distinct upstream connections accepted by a test server stays bounded.

[jgoux]

Reviewed the diagnosis and the proposed fix against develop @ 3704383.

Diagnosis: confirmed. The owner process runs on Bun (the CLI is Bun-compiled and spawns the owner through process.execPath). With agent: false (HttpProxy.ts#L292), every forwarded attempt opens a new upstream connection: 200 sequential requests produced 200 connections on both Bun and Node. The proxy side closes each connection, so TIME_WAIT accumulates on the owner's own ephemeral ports. Native and container runtimes both route HTTP through a single direct hop to 127.0.0.1:<port>, and the client-facing listener already supports keep-alive.

Bun also supports what the fix needs: a keep-alive http.Agent reuses sockets, ClientRequest.reusedSocket is reported correctly, and the agent timeout expires idle pooled sockets without aborting a quiet in-flight streamed response. A single agent keeps separate pools per host and port.

The proposed remediation needs changes:

1. Retrying a request with a body is unsafe, and not implementable as written. A reset on a reused socket before response headers does not prove the upstream skipped the request: a POST can commit its side effect and then reset, so a retry can run it twice. The body is also streamed with request.pipe(outgoing) (#L318), so nothing is left to resend. Suggested direction: use the pooled agent only for safe, bodyless requests (the existing retry predicate at #L231-L248), keep fresh connections for everything else, and share one retry budget between the existing retry and stale-socket recovery. That keeps the fix(stack): open a fresh upstream connection per proxied request #6897 regression test valid (it expects two connections for two POSTs). The issue should state that write-heavy load (PostgREST writes and RPC, Storage uploads, function invocations) still opens one connection per request.

2. Drop the bounded socket count. With maxSockets = N, N long-lived streamed responses occupy the pool and further requests to that upstream queue with no timeout. A keep-alive agent with the default unbounded maxSockets already caps connections at peak concurrency, and maxFreeSockets plus timeout bound idle retention.

3. Own one agent in the proxy's scope. No per-upstream registry is needed: create a single agent in makeHttpProxy and destroy it in the listener's release (#L474-L488). Keep target acquisition per request so idle pooled sockets never hold the activity lease.

4. Establish what Studio actually did in fix(stack): open a fresh upstream connection per proxied request #6897. The fix(stack): open a fresh upstream connection per proxied request #6897 fixture resets every reused connection. If real Studio (or Docker Desktop's forwarder after a wake) behaves that way, every pooled GET to it fails once and retries on a fresh connection: no fewer connections, only extra latency and a warning per request. Confirm the real behavior before counting a win for that upstream, and add a per-upstream fallback only if persistent rejection is confirmed.

5. Tests. A connection count alone can pass while requests fail. Suggested coverage:

   • Count backend connection events at a controlled concurrency and assert every response's status and body.
   • Keep the fix(stack): open a fresh upstream connection per proxied request #6897 POST test passing.
   • A GET whose reused socket resets before headers succeeds through exactly one fresh attempt.
   • A POST whose backend reads the full body and then resets is delivered exactly once and returns 502.
   • A reused-socket failure followed by a fresh-connection failure returns 502 (retry budget exhausted).

Minor: causal chain. socket hang up / ECONNRESET does not by itself indicate local ephemeral-port exhaustion, which normally surfaces as EADDRNOTAVAIL on connect; the resets may come from Docker Desktop's port forwarder. Connection reuse is justified either way, but capturing the connect errors during a failing wave would confirm the exhaustion link. The existing GET retry also adds a second connection per failing request during those waves.

[avallete]

Thanks for the review. We gathered the missing data before changing the fix.

Causal chain. We re-ran the load on develop @ 3704383: 24 concurrent loops of GET /rest/v1/ for 45 s.

• Native: TIME_WAIT reached 16,327, which is the whole 49152–65535 range. Every failure was connect EADDRNOTAVAIL, so this is local ephemeral-port exhaustion, as you expected.
• Docker: no failure was EADDRNOTAVAIL. Once about 10.5k proxy-side TIME_WAIT sockets had built up, Docker Desktop's forwarder started accepting connections and then closing them. The TIME_WAIT then shows on the forwarder side, and we get socket hang up / ECONNRESET. So on docker the resets come from the forwarder under connection churn, not from our own ports.
• Retry cost: every 502 cost two connections, because the existing GET retry also failed.

What Studio did in #6897.

• Studio's POST /api/mcp closes the connection about 0.5 s after a 200 or 202 response, even though it sends Keep-Alive: timeout=5.
• A pooled reuse that lands 0.3–0.5 s after the previous response gets ECONNRESET. Through the pre-fix(stack): open a fresh upstream connection per proxied request #6897 gateway, that window gave up to 17 of 30 POSTs as 502. There were no failures outside it, including at 2 s and 7 s gaps.
• Studio does not reset every reused connection, and the failures have nothing to do with wake-up.
• Studio GETs follow Node's normal behaviour: the socket closes about 6 s after going idle, with a similar race right at that point.
• Other upstreams close idle connections much later: Storage after about 61 s, PostgREST after about 55 s, GoTrue after more than 100 s.
• Docker Desktop's forwarder passes the FIN through promptly, but it silently drops writes on a socket that has already been closed instead of resetting it.

Proposed fix, following your points:

1. One keep-alive agent per proxy. It is owned by the proxy's scope, with the default unbounded maxSockets. Target acquisition stays per request.
2. Idle timeout. The agent gets a timeout of about 4 s, so pooled sockets are dropped before Node upstreams close them at about 6 s.
3. Only safe, bodyless requests are pooled. GET, HEAD, OPTIONS and TRACE without a body use the pool. Everything else, including MCP POSTs and requests carrying an Idempotency-Key (our upstreams do not deduplicate on it), uses a fresh connection and is never replayed.
4. One retry in total. A safe, bodyless request that fails before response headers is retried once, always on a fresh connection.
5. Known limitation: write-heavy load (PostgREST writes and RPC, Storage uploads, function invocations) still opens one connection per request.

Tests.

• Bounded connection count at controlled concurrency, asserting every status and body.
• The fix(stack): open a fresh upstream connection per proxied request #6897 POST test, kept unchanged.
• A reused GET that resets gets exactly one fresh attempt.
• A POST (even with an Idempotency-Key) whose backend reads the body and then resets is delivered once and returns 502.
• A reused failure followed by a fresh failure returns 502.

With the fix, the earlier docker and native load runs gave no 502s, and TIME_WAIT stayed at 2 or below during load.

Edited: narrowed pooling to safe, bodyless requests, as you suggested. An earlier version of this comment also pooled and replayed Idempotency-Key requests and small safe-method bodies, which is unsafe because the upstreams don't deduplicate. Implemented in #6925.