fix(proxy): stop spending quota on h2 requests the client abandoned - #182
Open
ak2k wants to merge 4 commits into
Open
fix(proxy): stop spending quota on h2 requests the client abandoned#182ak2k wants to merge 4 commits into
ak2k wants to merge 4 commits into
Conversation
…lient The pin segment is percent-encoded by the client, and decodeURIComponent throws URIError on a malformed escape. "/tc-acct/%/v1/messages", "/tc-acct/%zz/v1/messages" and a truncated "/tc-acct/%E0%A4/v1/messages" are all ordinary request lines, and all three throw out of the pin parsing into a catch that logs and returns. The client never gets a response and waits until its own timeout expires. An undecodable pin is unusable for the same reason an unknown one is, so the decode is guarded and falls through to the existing unknown-pin 404. The reply quotes the token as it arrived, since there is no decoded form to show. The new test races the request against a timer, so a hang shows up as a failed assertion instead of a stuck run.
The 502 in forwardRequest covers the inner try only. The code above it (the egress hold, pin parsing, body buffering, the activity hooks) runs under a catch that logs and returns, and createProxyServer's request handler has the same catch around the auth gate, the CSRF gate, the forward-proxy relay and the status/reload/switch endpoints. A throw in either window leaves the socket open and silent, and the client waits until its own timeout expires. getStatusExtra is a hook the application installs, so the second window is reachable from a plain GET. Answering only before headersSent still leaves a hang: the status endpoint serializes the hook's value after writeHead, so a hook returning something JSON.stringify rejects (a cycle, a BigInt) throws with the 200 already sent, and nothing ends the response. Both catches now go through one answerUnhandled helper carrying the same pair of arms forwardRequest already uses: a 502 while nothing has been written, and destroy once something has. destroy rather than end() on the second arm, because end() delivers truncated bytes as an apparently complete reply and the client has no reason to retry. The headersSent guard matters in the other direction too: the inner finally calls onRequestEnd after the response has streamed, so a throw from that hook reaches the catch with the headers long sent, and an unguarded writeHead would raise ERR_HTTP_HEADERS_SENT from inside the catch. Four new tests, each racing its request against a timer so a hang is a failed assertion instead of a stuck run.
Every consumer of the activity hooks holds a request's row until it is told the request ended. The TUI keeps it in `active` and keeps its animation running while one is open; a headless consumer counts it as in flight. Nothing reclaims a row that is never closed, so on a daemon that runs for weeks each leak is permanent. Only the inner path had a `finally`, so a throw above it opened a row that nothing would ever close. The ordinary trigger is a client cancelling mid-upload, which makes `for await (const chunk of req)` reject: Ctrl+C in Claude Code does that routinely. A start hook that throws leaks the same way, and that is the shipped hook's shape, since the TUI registers the row and then renders, and the render can rethrow. The listener now tracks the open entry in one marker. It is set before the start hook, so a hook that registers its row and then throws is still accounted for. Every closing site clears the marker before calling the end hook, because a hook that throws would otherwise still look open to the outer catch, which would then call that same hook a second time for one request. The outer catch closes whatever is left, as 499 when the client is gone or the response is past saying anything, and 502 when that is what it is about to write. Its own call to the hook is guarded, because the throw that sent it there may be that hook. Unguarded, the second throw escapes an async request listener with nothing above it, which is an unhandled rejection, and crash-log.js turns that into exit(1). A broken activity hook could take the daemon down. The recovery also has to survive its own logging, because the console it logs through is the same component whose failure it is recovering from. Under the TUI the console is the TUI: `console.error` appends to the activity log and repaints, so a render that throws makes the console throw. The report is the first statement of each of these paths, so an unguarded one skips the whole recovery: the row still leaks, the socket is never answered, and the throw escapes as an unhandled rejection after all. Every report on a recovery path now goes through one helper that falls back to stderr, the way the TUI already does when its own activity stream fails. The fallback writes with `writeSync` rather than `process.stderr.write`, because it has to fail the way the helper promises. A stderr whose reader is gone makes the stream surface EPIPE asynchronously, as an error event no `try` around the call can see, and an uncaught EPIPE is fatal here. Written that way, a helper meant to keep a broken render from killing the daemon would kill it on a closed pipe instead. `writeSync` throws where it is called, so the catch is real. Eight tests: an abort mid-body, a start hook that registers and then throws, an end hook that throws, a hook that throws on every call, the blocklist's early return, a console that throws on each of the two recovery paths, and a report written to a stderr nobody is reading.
Every guard that asks whether the client is still there reads `res.destroyed`, and `Http2ServerResponse` has no `destroyed` property. The read is `undefined`, so the answer comes back "no" for every request on the MITM path, which is the one carrying most of the traffic. On the retry ladder that costs quota. A cancelled h2 request whose first account answers with a quota rejection is retried on the next account, and the next, and each retry is a real upstream call against that account's weekly bucket, for a client that is not there to read any of it. Measured with four accounts: a cancelled h2 request spent all four. It now spends one. It also strands the request handler. Writing to a cancelled stream returns false, which sends `streamResponse` into its backpressure wait, and that wait listens for a `drain` or a `close` that has already happened and will not happen again. The handler never returns, so its activity entry never closes and it holds the event loop for as long as the process lives. Measured on a cancelled h2 SSE stream: the entry opened and never closed. `clientGone` asks the question where both shapes are known: the response's own `destroyed` on h1, the underlying stream's on h2. Every read moves to it, so there is no site left that asks this in a way that is inert on the busier transport. Held by the number of accounts spent and by whether the handler returns, rather than by a count of guards. The rungs are individually redundant, since the abort probe handed to `admit()` already stops the ladder on most paths, so a test written against a guard count would measure something other than the cost. The streaming test runs in a child. A stranded handler holds the event loop, so in process it would leave the runner unable to exit whether it passed or failed.
Author
|
Rebased onto #181's current head; the two branches had diverged by five comment-wording lines, which made the stack conflict with itself. No code change. |
ak2k
force-pushed
the
fix/h2-client-gone
branch
from
August 25, 2026 01:05
73e6631 to
4891b4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #180 and the activity-entry PR. Review those first; the diff here is the last commit only.
A cancelled h2 request keeps consuming quota. With four accounts, if the first account answers a cancelled request with a quota rejection, the retry ladder tries the other three as well, and each retry is a real upstream call billed to that account's weekly bucket.
Why
The retry guards check
res.destroyed.Http2ServerResponsehas nodestroyedproperty, so on the MITM path the check readsundefinedand never trips. The h2 equivalent isres.stream.destroyed.The MITM path carries most traffic:
HTTPS_PROXYpointed at teamclaude is how aclaudeinstance connects, and the terminating server negotiates h2 with any client that offers it.The fix
A
clientGone(res)helper covers both transports:res.destroyedon h1,res.stream?.destroyedon h2. All of theres.destroyedreads move to it, including the retry-ladder rungs, the concurrency slot's abort probe, the late 429 and 502 answers, and the recovery's status choice.A stranded handler on the streaming path
The inert check has a second consequence.
res.writeto a cancelled stream returns false, which sendsstreamResponseinto its backpressure wait:On a cancelled h2 stream,
closehas already fired before this listener is registered, anddrainnever fires on a closed stream, so the promise never settles. The check that should have exited the loop before the write is the inert one, so the handler never returns. Cancelling an h2 SSE stream after its first chunk, without the fix:The activity entry stays open, undoing the accounting PR below this one on the streaming path, and the stranded handler sits on the event loop for the life of the process. With the fix, the same probe reports
started=1 ended=1 returned=true.Test design
The guards are mutually redundant: reverting any single call site outside the streaming path changes nothing, because another site catches the same cancellation. So the tests measure accounts spent and whether the handler returns, not which guards ran.
Reverting the h2 arm of
clientGoneitself restores both defects at once:Site-level results, for the record:
The sites a quota-429 ladder does not exercise are converted for uniformity; the ladder test does not reach them. On the streaming path, reverting the check before the write turns the strand test red. The check after the backpressure wait is a backstop for a cancel that lands during the wait itself; that race cannot be driven reliably in a test, so it is noted here instead.
Tests
Two, in
test/mitm-integration.test.js. Both drive a real CONNECT tunnel with h2 negotiated over ALPN and a real cancellation (NGHTTP2_CANCEL), and both verify the setup worked before asserting the result.The ladder test uses four accounts whose upstream answers each with a quota rejection, confirms the upstream was actually reached, and waits for the ladder to stop growing rather than sleeping a fixed span, since a fixed sleep can read the count before the remaining rungs run.
The strand test cancels an SSE stream after its first chunk and asserts the handler returned, using the closed activity entry as the signal. It runs the proxy in a child process: a stranded handler would keep the test runner itself from exiting, so the parent enforces the timeout.
538 before, 540 after.
npx eslint .clean.Compatibility
h1 is unchanged:
clientGonereads the sameres.destroyedas before, andres.stream?.destroyedisundefinedthere. On h2, a live request sees no difference, since its stream is not destroyed; a cancelled one stops instead of continuing to retry.