Skip to content

Record an error the upstream answered as a failed request; TurnView carries the status - #263

Merged
fylorn merged 1 commit into
mainfrom
fix/passthrough-failed-turns
Oct 2, 2026
Merged

fylorn merged 1 commit into
mainfrom
fix/passthrough-failed-turns

Conversation

@fylorn

@fylorn fylorn commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Problem

When an upstream answers 4xx (or 3xx) and the gateway hands that answer on to the client (the request's own fault, or no next upstream to fail over to), the ending was RequestFinished with that status and an empty error. Every place that counts failures reads error, so the request looked successful everywhere:

  • traffic list (/history, and the desktop app's live rows), history search's "failed" filter
  • overview (/summary failed), session list (errors), upstream check-up
  • session detail: TurnView had no error and no status, while the transcript (which reads the status) gave the turn no answer. The desktop app's conversation showed only the user's message, with no answer and no reason.

Fix

The gateway now ends these requests with RequestFailed (source upstream), so one rule holds wherever failures are counted: a request failed if and only if error is set. No SQL changes.

  • relay marks a non-2xx answer on the Ending (Ending::refused). The Ending keeps the head of the error body (whether or not bodies are stored) and, when the answer has been passed on, reports the failure with what the upstream said:
    • gw.upstream.status_message {upstream, status, message}: "Upstream `x` answered 400: prompt is too long: …", read in the upstream's format with the new tw_dialect::convert::error_message (now shared with Session::error); plain text is used as it is
    • an empty body or an HTML page falls back to the existing gw.upstream.status
    • the words are masked with the request's redaction, like the stored response body, and capped at 500 characters
  • A Bedrock credential refusal already replaces AWS's words with ours; that message is now the recorded reason. The [ThinkWatch] prefix moves out of those four messages into the body sent to the client, as for the gateway's own errors (codes and arguments unchanged).
  • Unchanged: the client still gets the upstream's answer byte for byte (no x-thinkwatch-error); a client that leaves while the error is passed on is still a cancellation; WebSocket upgrades (101); the circuit breaker (a client error is still not held against the upstream); the attempt chain (served with the status).

TurnView gains status: number | null (the upstream's status code; null when the request never got one), so a failed turn can say what the upstream answered. CONTROL_API_VERSION stays at 32 (unreleased); its note now mentions both changes.

For the desktop app: one new message code, gw.upstream.status_message (upstream, status, message), needs a translation.

Tests

  • ending.rs: the upstream's words; nothing readable (empty, whitespace, HTML) → status only; plain text and unknown JSON as they are; long words cut; a secret in the upstream's words is masked; our words for Bedrock; a client leaving during an error answer is still a cancellation
  • tests/endings.rs: a passed-on 400 reaches the client unchanged and ends exactly once as an upstream failure carrying the upstream's words
  • tests/bedrock.rs: the refusal's recorded reason names no account and has no prefix
  • recorder.rs: row, turns (with status), session errors and no-usage count, summary for a 200, a passed-on 400 and a cancelled turn
  • tw-control/tests/error_answers.rs (end to end through a real gateway and store): the same three turns in one session, read back from /sessions/{id}, /history, /summary, /sessions and the transcript; it fails without the gateway change
  • tw-dialect: error_message for each format; tw-api (ts): TurnView exports status: number | null

Local: cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings (and --lib), env -u HTTP_PROXY -u HTTPS_PROXY cargo test --workspace (2099 passed), cargo clippy -p tw-api --all-targets --features ts and cargo test -p tw-api --features ts.

🤖 Generated with Claude Code

…arries the status

When an upstream answered 4xx (or 3xx) and the gateway handed the answer
on to the client, the ending was RequestFinished with that status and an
empty error. Everything that counts failures reads `error`, so the
request looked successful everywhere: the traffic list, the overview's
failed count, the session's errors, the upstream check-up, the history
search's failed filter. Session detail (TurnView) treated the turn as
done while the transcript, which reads the status, gave it no answer:
the desktop app's conversation showed only the user's message, with no
answer and no reason.

The gateway now reports these as RequestFailed (source `upstream`):

- relay marks a non-2xx answer on the Ending (`Ending::refused`). The
  Ending keeps the head of the error body whether or not bodies are
  stored, and on `finished` reports the failure with what the upstream
  said: gw.upstream.status_message {upstream, status, message}
  ("Upstream `x` answered 400: prompt is too long..."), read in the
  upstream's format with tw_dialect::convert::error_message (now shared
  with Session::error). Plain text is used as it is; an empty body or an
  HTML page falls back to gw.upstream.status. The words are masked with
  the request's redaction, like the stored response body, and capped at
  500 characters.
- A Bedrock credential refusal already replaces AWS's words with ours;
  that message is now the recorded reason. The `[ThinkWatch]` prefix
  moves out of those four messages into the body sent to the client, as
  with the gateway's own errors.
- A client that leaves while the error is being passed on is still a
  cancellation. WebSocket upgrades (101) are unchanged.

The row's status still comes from RequestHeaders, so one rule holds in
every place that counts: a request failed if and only if `error` is set.
No SQL changes, and cancellations are counted as before.

TurnView gains `status` (the upstream's status code, null when the
request never got one), so a failed turn can say what the upstream
answered. CONTROL_API_VERSION stays at 32 (unreleased); its note
mentions both changes.

Tests: the Ending (the upstream's words, nothing readable, masking, our
words for Bedrock, a cancellation during an error answer); a passed-on
400 reaches the client unchanged and ends once as a failure; the Bedrock
refusal's recorded reason; rows, turns, session and summary in the
recorder; and an end-to-end control-plane test through a real gateway
and store with a 200, a passed-on 400 and a cancelled turn in one
session, read back from /sessions/{id}, /history, /summary, /sessions
and the transcript.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fylorn
fylorn merged commit 104559c into main Oct 2, 2026
4 checks passed
@fylorn
fylorn deleted the fix/passthrough-failed-turns branch October 2, 2026 15:08
fylorn added a commit that referenced this pull request Oct 3, 2026
Release for #251, #252, #263, #265, #268 and #272, with the pre-release
fixes in #273.

- Bump the workspace version to 0.58.0
- release-notes/0.58.0.md

The control-plane protocol (34) and the request store schema (25) are
already at their release values on main and do not change here.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant