Skip to content

feat: native passthrough behavior — byte-budget admission, fail-open routing, correct error types - #31

Closed
dean0x wants to merge 5 commits into
mainfrom
feat/native-passthrough-behavior
Closed

feat: native passthrough behavior — byte-budget admission, fail-open routing, correct error types#31
dean0x wants to merge 5 commits into
mainfrom
feat/native-passthrough-behavior

Conversation

@dean0x

@dean0x dean0x commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • Replace count-based 503 rejection with byte-budget admission + queueing (HTTP 529 only on queue exhaustion)
  • Colon-prefixed model names with unknown provider qualifiers now forward to Anthropic instead of returning a relay-invented 400
  • 413 response now uses request_too_large error type (spec-correct) instead of invalid_request_error

Changes

CHANGE 1 — Byte-budget admission gate (src/server.ts, src/config.ts)

The old maxConcurrentRequests: 32 gate rejected with 503 under realistic load (peak ~100 concurrent) and used the wrong status code. The new gate:

  • Bounds total in-flight request bytes (limits.maxInFlightBytes, default 2 GiB) rather than request count
  • Queues requests that would exceed the budget instead of rejecting them; emits 529 overloaded_error only when the queue itself is full
  • Grants single-request progress: an oversized request is always admitted when inFlightBytes === 0
  • Releases budget on res.once("finish") (keep-alive safe) plus res.once("close") with a once-guard
  • Handles client disconnect while queued — removes entry, no reservation leak
  • New knobs: limits.maxInFlightBytes, limits.maxQueueDepth (1000), limits.maxQueueWaitMs (60 000 ms)
  • anthropic.maxUpstreamSockets default raised 32 → 256
  • limits.maxConcurrentRequests deprecated but kept for backward compat
  • /__subswitch/* exempt from the gate

CHANGE 2 (L1) — Fail-open for unknown colon qualifiers (src/server.ts)

model="claude-sonnet-9:preview" was rejected with a relay-invented 400. The relay should behave as if the client were talking directly to api.anthropic.com. Fix: the unknown_provider switch arm now forwards to Anthropic and logs a warning. Real codex: prefix routing is unaffected.

CHANGE 3 (L4) — Correct 413 error type (src/errors.ts)

The body_too_large → invalid_request_error mapping is wrong per the Anthropic spec. Fixed to request_too_large. Also adds not_found_error and request_too_large to the AnthropicErrorType union.

Breaking Changes

None. limits.maxConcurrentRequests is deprecated (no longer enforced) but the config key is still accepted.

Reviewer Focus Areas

  • src/server.ts — acquireSlot / drainQueue / getReservationBytes / releaseSlot logic; the unknown_provider arm change
  • src/config.ts — new schema fields and defaults
  • test/integration/concurrency.test.ts — 7 suites covering all admission-gate invariants
  • CHANGELOG.md — heading ## [0.2.1] - 2026-08-19

dean0x and others added 5 commits August 19, 2026 02:42
@dean0x
fix(errors): 413 uses request_too_large type, add missing AnthropicEr…
12d05af 
…rorType variants

- Map body_too_large → request_too_large (was invalid_request_error) in
  proxyErrorToAnthropic; status code remains 413
- Add not_found_error and request_too_large to AnthropicErrorType union so
  callers can type-check against all Anthropic-spec error types
- Add unit test asserting the correct mapping (L4 regression test)
@dean0x
fix(routing): unknown colon-prefixed model qualifiers fail open to An…
e2104d4 
…thropic

When a model name contains a colon prefix that is not a registered provider
id (e.g. "claude-sonnet-9:preview"), the relay was returning a relay-invented
400 error.  This violates the passthrough principle — api.anthropic.com would
have accepted or rejected the request on its own terms.

Change: in the unknown_provider switch arm of dispatch(), forward to the
Anthropic upstream and emit a warn-level log instead of returning 400.
Real codex: prefix routing is unaffected.

Adds routing-behavior tests: L1 (fail-open for unknown qualifier) and the
existing F6g updated to assert the request reaches Anthropic, not 400.
@dean0x
feat(admission): replace count-based rejection with byte-budget queueing
f0b31d6 
Replace the 32-request concurrent-count gate (which could 503 under normal
load) with a byte-budget admission gate that queues rather than rejects:

- New config knobs: limits.maxInFlightBytes (default 2 GiB),
  limits.maxQueueDepth (1000), limits.maxQueueWaitMs (60 000 ms)
- limits.maxConcurrentRequests deprecated (kept for backward compat)
- anthropic.maxUpstreamSockets default raised from 32 → 256
- Admission uses content-length header; POST /v1/messages without
  content-length falls back to maxBodyBytes (conservative, no bypass)
- Single-request progress: an oversized request is always admitted when
  the server is idle (inFlightBytes === 0)
- Budget released on res.finish (keep-alive safe) AND res.close with a
  once-guard to prevent double-decrement
- Client disconnect while queued removes the entry and drains the queue
  without leaking the reservation
- Queue overflow returns HTTP 529 overloaded_error (not 503)
- /__subswitch/* exempt from the gate

Integration tests cover: admitted under budget, queueing-not-rejection,
single-request progress, disconnect-while-queued, counter-returns-to-zero,
queue-exhaustion 529, health never gated.
@dean0x @claude
fix(admission): FIFO guard, Result types, reconcile chunked reservati…
6deeb14 
…on, test non-vacuity

Required fixes from adversarial review:

1. FIFO / anti-starvation (BLOCKING): add queue.length === 0 guard to acquireSlot
   immediate-admission condition. New arrivals must join the queue when waiters
   are already present — prevents small requests from barging past a large queued
   one and causing starvation-to-529.

2. Rewrite vacuous b1 test (BLOCKING): previous "queueing" test used budget=100
   with 10-byte bodies (nothing ever queued) and asserted status < 529 (passes on
   504). New test: budget=10, asserts parkedRes===null before release (req2 must
   be queued), then asserts res2.status===200 and body.ok===true.

3. Add queued-state assertion in b2 test (BLOCKING): assert parked.parkedRes===null
   before controller1.abort() so the test fails when queueing is disabled (req2
   would be immediately admitted and parkedRes would be non-null).

4. Assert inFlightBytes invariant (BLOCKING): replace Math.max silent clamp with
   delta-check + logger.log("error", "inFlightBytes_underflow", ...) so a
   double-release is visible in operator logs. Still clamps to prevent counter
   going negative; justified in comment.

5. Reconcile chunked reservation (BLOCKING): for POST /v1/messages without
   content-length, getReservationBytes reserves maxBodyBytes (32 MiB) as an
   anti-bypass estimate. After bufferBody() returns, shrink inFlightBytes and
   the slot to the actual body size. reservationBytes is now let so the
   releaseSlot closure sees the updated amount. Documented in JSDoc, README,
   CHANGELOG.

Recommended fixes:

6. Request identity assertions: add distinct ?id= query params and assert
   parkedUrl?.includes(...) in tests b1, b2, d, e so tests verify WHICH
   request reached upstream, not just that something arrived.

7. Positive control in test (g): before the health check, fire a second POST
   and assert it queues (parkedRes===null after 30ms), proving the gate is
   engaged before verifying health bypasses it.

8. Real-registry L1 test: add "claude-sonnet-9:preview" through the real
   registry (no synthetic resolver), asserting Anthropic routing (200) and
   byte-identical response body forwarding.

9. acquireSlot returns Result<void, SlotError>: converts from throw/reject to
   discriminated Result type per codebase convention. DisconnectedWhileQueued
   sentinel class removed; kind "disconnected" now handled via Result.error.kind.
   Route logged as "disconnected_while_queued" to prevent spurious
   route:anthropic/status:200 log entries for unserved requests.

10. Dead code removed: QueueEntry.reject field (never called by drainQueue),
    not_found_error from AnthropicErrorType union (appeared only in type
    declaration, unused in src/ or test/).

11. Bad log line: DisconnectedWhileQueued early-return now sets
    route = "disconnected_while_queued" before returning so request_complete
    does not emit anthropic/200 for a request that was never served.

Non-vacuity verified: b1 queued-assertion fails with huge budget, b2 queued-
assertion fails with huge budget, h (FIFO) only-A-assertion fails without the
queue.length guard.

Co-Authored-By: Claude <noreply@anthropic.com>
@dean0x @claude
test(integration): fix stale 413 error type assertion in passthrough …
c20efd9 
…test

Commit 12d05af changed the synthesized 413 body to use `request_too_large`
(matching Anthropic's own API taxonomy) and updated the unit test, but the
matching assertion in test/integration/passthrough.test.ts was missed.

Update `body.error.type` assertion from `invalid_request_error` to
`request_too_large` to match the current source.

Co-Authored-By: Claude <noreply@anthropic.com>
@dean0x dean0x mentioned this pull request Aug 19, 2026
Merged
@dean0x

dean0x commented Aug 19, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #33. All commits from this branch are contained in wave/passthrough-hardening, which carries this work plus the transparency-hardening follow-up. Closing unmerged.

@dean0x dean0x closed this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dean0x