Skip to content

fix(proxy): stop spending quota on h2 requests the client abandoned - #182

Open
ak2k wants to merge 4 commits into
KarpelesLab:masterfrom
ak2k:fix/h2-client-gone
Open

fix(proxy): stop spending quota on h2 requests the client abandoned#182
ak2k wants to merge 4 commits into
KarpelesLab:masterfrom
ak2k:fix/h2-client-gone

Conversation

@ak2k

@ak2k ak2k commented Aug 20, 2026

Copy link
Copy Markdown

Builds on #180 and the activity-entry PR. Review those first; the diff here is the last commit only.

A cancelled h2 request keeps consuming quota. With four accounts, if the first account answers a cancelled request with a quota rejection, the retry ladder tries the other three as well, and each retry is a real upstream call billed to that account's weekly bucket.

before: a cancelled h2 request spent 4 of 4 accounts
after:  it spends 1

Why

The retry guards check res.destroyed. Http2ServerResponse has no destroyed property, so on the MITM path the check reads undefined and never trips. The h2 equivalent is res.stream.destroyed.

The MITM path carries most traffic: HTTPS_PROXY pointed at teamclaude is how a claude instance connects, and the terminating server negotiates h2 with any client that offers it.

The fix

A clientGone(res) helper covers both transports: res.destroyed on h1, res.stream?.destroyed on h2. All of the res.destroyed reads move to it, including the retry-ladder rungs, the concurrency slot's abort probe, the late 429 and 502 answers, and the recovery's status choice.

A stranded handler on the streaming path

The inert check has a second consequence. res.write to a cancelled stream returns false, which sends streamResponse into its backpressure wait:

await new Promise(resolve => {
  const done = () => { res.off('drain', done); res.off('close', done); resolve(); };
  res.once('drain', done);
  res.once('close', done);
});

On a cancelled h2 stream, close has already fired before this listener is registered, and drain never fires on a closed stream, so the promise never settles. The check that should have exited the loop before the write is the inert one, so the handler never returns. Cancelling an h2 SSE stream after its first chunk, without the fix:

sawChunk=true  started=1  ended=0  returned=false

The activity entry stays open, undoing the accounting PR below this one on the streaming path, and the stranded handler sits on the event loop for the life of the process. With the fix, the same probe reports started=1 ended=1 returned=true.

Test design

The guards are mutually redundant: reverting any single call site outside the streaming path changes nothing, because another site catches the same cancellation. So the tests measure accounts spent and whether the handler returns, not which guards ran.

Reverting the h2 arm of clientGone itself restores both defects at once:

h2 arm dropped from the helper -> spent 4 of 4 accounts, and the handler strands again

Site-level results, for the record:

only the admit() abort probe converted -> spends 1 of 4
only the quota-429 rung converted      -> spends 1 of 4
both reverted, the rest converted      -> spends 4 of 4

The sites a quota-429 ladder does not exercise are converted for uniformity; the ladder test does not reach them. On the streaming path, reverting the check before the write turns the strand test red. The check after the backpressure wait is a backstop for a cancel that lands during the wait itself; that race cannot be driven reliably in a test, so it is noted here instead.

Tests

Two, in test/mitm-integration.test.js. Both drive a real CONNECT tunnel with h2 negotiated over ALPN and a real cancellation (NGHTTP2_CANCEL), and both verify the setup worked before asserting the result.

The ladder test uses four accounts whose upstream answers each with a quota rejection, confirms the upstream was actually reached, and waits for the ladder to stop growing rather than sleeping a fixed span, since a fixed sleep can read the count before the remaining rungs run.

The strand test cancels an SSE stream after its first chunk and asserts the handler returned, using the closed activity entry as the signal. It runs the proxy in a child process: a stranded handler would keep the test runner itself from exiting, so the parent enforces the timeout.

$ node --test --test-timeout=120000
ℹ tests 540
ℹ pass 540
ℹ fail 0

538 before, 540 after. npx eslint . clean.

Compatibility

h1 is unchanged: clientGone reads the same res.destroyed as before, and res.stream?.destroyed is undefined there. On h2, a live request sees no difference, since its stream is not destroyed; a cancelled one stops instead of continuing to retry.

ak2k added 4 commits August 20, 2026 07:19
…lient

The pin segment is percent-encoded by the client, and decodeURIComponent
throws URIError on a malformed escape. "/tc-acct/%/v1/messages",
"/tc-acct/%zz/v1/messages" and a truncated "/tc-acct/%E0%A4/v1/messages" are
all ordinary request lines, and all three throw out of the pin parsing into a
catch that logs and returns. The client never gets a response and waits until
its own timeout expires.

An undecodable pin is unusable for the same reason an unknown one is, so the
decode is guarded and falls through to the existing unknown-pin 404. The reply
quotes the token as it arrived, since there is no decoded form to show.

The new test races the request against a timer, so a hang shows up as a failed
assertion instead of a stuck run.
The 502 in forwardRequest covers the inner try only. The code above it (the
egress hold, pin parsing, body buffering, the activity hooks) runs under a
catch that logs and returns, and createProxyServer's request handler has the
same catch around the auth gate, the CSRF gate, the forward-proxy relay and
the status/reload/switch endpoints. A throw in either window leaves the
socket open and silent, and the client waits until its own timeout expires.
getStatusExtra is a hook the application installs, so the second window is
reachable from a plain GET.

Answering only before headersSent still leaves a hang: the status endpoint
serializes the hook's value after writeHead, so a hook returning something
JSON.stringify rejects (a cycle, a BigInt) throws with the 200 already sent,
and nothing ends the response.

Both catches now go through one answerUnhandled helper carrying the same
pair of arms forwardRequest already uses: a 502 while nothing has been
written, and destroy once something has. destroy rather than end() on the
second arm, because end() delivers truncated bytes as an apparently complete
reply and the client has no reason to retry. The headersSent guard matters
in the other direction too: the inner finally calls onRequestEnd after the
response has streamed, so a throw from that hook reaches the catch with the
headers long sent, and an unguarded writeHead would raise
ERR_HTTP_HEADERS_SENT from inside the catch.

Four new tests, each racing its request against a timer so a hang is a
failed assertion instead of a stuck run.
Every consumer of the activity hooks holds a request's row until it is told the
request ended. The TUI keeps it in `active` and keeps its animation running
while one is open; a headless consumer counts it as in flight. Nothing reclaims
a row that is never closed, so on a daemon that runs for weeks each leak is
permanent.

Only the inner path had a `finally`, so a throw above it opened a row that
nothing would ever close. The ordinary trigger is a client cancelling
mid-upload, which makes `for await (const chunk of req)` reject: Ctrl+C in
Claude Code does that routinely. A start hook that throws leaks the same way,
and that is the shipped hook's shape, since the TUI registers the row and then
renders, and the render can rethrow.

The listener now tracks the open entry in one marker. It is set before the
start hook, so a hook that registers its row and then throws is still accounted
for. Every closing site clears the marker before calling the end hook, because
a hook that throws would otherwise still look open to the outer catch, which
would then call that same hook a second time for one request. The outer catch
closes whatever is left, as 499 when the client is gone or the response is past
saying anything, and 502 when that is what it is about to write.

Its own call to the hook is guarded, because the throw that sent it there may
be that hook. Unguarded, the second throw escapes an async request listener
with nothing above it, which is an unhandled rejection, and crash-log.js turns
that into exit(1). A broken activity hook could take the daemon down.

The recovery also has to survive its own logging, because the console it logs
through is the same component whose failure it is recovering from. Under the TUI
the console is the TUI: `console.error` appends to the activity log and
repaints, so a render that throws makes the console throw. The report is the
first statement of each of these paths, so an unguarded one skips the whole
recovery: the row still leaks, the socket is never answered, and the throw
escapes as an unhandled rejection after all. Every report on a recovery path now
goes through one helper that falls back to stderr, the way the TUI already does
when its own activity stream fails.

The fallback writes with `writeSync` rather than `process.stderr.write`, because
it has to fail the way the helper promises. A stderr whose reader is gone makes
the stream surface EPIPE asynchronously, as an error event no `try` around the
call can see, and an uncaught EPIPE is fatal here. Written that way, a helper
meant to keep a broken render from killing the daemon would kill it on a closed
pipe instead. `writeSync` throws where it is called, so the catch is real.

Eight tests: an abort mid-body, a start hook that registers and then throws, an
end hook that throws, a hook that throws on every call, the blocklist's early
return, a console that throws on each of the two recovery paths, and a report
written to a stderr nobody is reading.
Every guard that asks whether the client is still there reads `res.destroyed`,
and `Http2ServerResponse` has no `destroyed` property. The read is `undefined`,
so the answer comes back "no" for every request on the MITM path, which is the
one carrying most of the traffic.

On the retry ladder that costs quota. A cancelled h2 request whose first account
answers with a quota rejection is retried on the next account, and the next, and
each retry is a real upstream call against that account's weekly bucket, for a
client that is not there to read any of it. Measured with four accounts: a
cancelled h2 request spent all four. It now spends one.

It also strands the request handler. Writing to a cancelled stream returns false,
which sends `streamResponse` into its backpressure wait, and that wait listens
for a `drain` or a `close` that has already happened and will not happen again.
The handler never returns, so its activity entry never closes and it holds the
event loop for as long as the process lives. Measured on a cancelled h2 SSE
stream: the entry opened and never closed.

`clientGone` asks the question where both shapes are known: the response's own
`destroyed` on h1, the underlying stream's on h2. Every read moves to it, so
there is no site left that asks this in a way that is inert on the busier
transport.

Held by the number of accounts spent and by whether the handler returns, rather
than by a count of guards. The rungs are individually redundant, since the abort
probe handed to `admit()` already stops the ladder on most paths, so a test
written against a guard count would measure something other than the cost.

The streaming test runs in a child. A stranded handler holds the event loop, so
in process it would leave the runner unable to exit whether it passed or failed.
@ak2k

ak2k commented Aug 25, 2026

Copy link
Copy Markdown
Author

Rebased onto #181's current head; the two branches had diverged by five comment-wording lines, which made the stack conflict with itself. No code change.

@ak2k
ak2k force-pushed the fix/h2-client-gone branch from 73e6631 to 4891b4a Compare August 25, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant