proxy: native Anthropic/Gemini capture, usage auto-injection, retry classification - #5
Merged
Merged
Conversation
…lassification Three proxy-layer improvements that close honest limitations documented in the README, plus the schema work they require. 1. Multi-provider usage parsing. parseUsageJSON now detects the OpenAI, Anthropic (/v1/messages) and Gemini (usageMetadata) wire shapes and maps them onto a provider-neutral tokenUsage. Streaming usage is merged last-non-zero-wins so Anthropic's split message_start/message_delta events reconcile like OpenAI's single final block. Gemini's model is read from the URL path. This lets an Anthropic/Gemini agent point base_url at the proxy directly instead of needing the callback shim. Anthropic separates cache-read from cache-write (cache_creation_input_tokens, ~1.25x input), which the old Usage model couldn't express. Added a CacheWriteTokens bucket end to end (trace, cost.Usage/ModelPrice/Breakdown, pricing loader, pricing.yaml, aggregate) so cache writes are billed at their real premium rather than silently under-counted. 2. Auto-inject stream_options.include_usage on OpenAI streaming chat requests that don't opt in, so streamed calls are captured exactly instead of as a zero-token row. Surgical merge preserves every other request field; scoped to the OpenAI chat path so it never reaches Anthropic/Gemini. Toggle with --inject-usage=false. 3. Retry classification. The proxy hashes each request body per (scenario, run) and labels a byte-identical repeat trace.KindRetry vs KindInitial — the observable signature of a client-library retry. Surfaced as retries/run in the aggregate. Fan-out is deliberately not claimed (call concurrency isn't visible at the HTTP layer). Classification and injection key off the original request body, so record/replay still regenerate identical traces. Tests cover all three plus cache-write billing; README limitations updated to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three proxy-layer improvements that close honest limitations the README already documented, plus the schema work they require. All backward compatible;
go build+go vet+go test ./... -count=1green (mirrors CI).1. Multi-provider usage capture (
#1)parseUsageJSONnow detects the OpenAI, Anthropic (/v1/messages) and Gemini (usageMetadata) wire shapes and maps them onto a provider-neutraltokenUsage. Streaming usage is merged last-non-zero-wins so Anthropic's splitmessage_start/message_deltaevents reconcile like OpenAI's single final block; Gemini's model is read from the URL path.Anthropic separates cache-read from cache-write (
cache_creation_input_tokens, ~1.25× input), which the oldUsagemodel couldn't express. Added aCacheWriteTokensbucket end to end (trace→cost.Usage/ModelPrice/Breakdown→ pricing loader →pricing.yaml→aggregate) so cache writes are billed at their real premium instead of being silently under-counted.This lets an Anthropic/Gemini agent that can set a
base_urlpoint straight at the proxy instead of needing the callback shim.2. Auto-inject
stream_options.include_usage(#2)OpenAI only emits a streaming usage block when the request opts in. The proxy now injects that flag surgically (preserving every other field), scoped to the OpenAI chat path so it never reaches Anthropic/Gemini. Streamed calls are captured exactly instead of as a zero-token row. Toggle:
--inject-usage=false.3. Retry vs call classification (
#3)The proxy hashes each request body per
(scenario, run)and labels a byte-identical repeattrace.KindRetryvsKindInitial— the observable signature of a client-library retry. Surfaced as retries/run in the aggregate. Fan-out is deliberately not claimed (call concurrency isn't visible at the HTTP layer).Classification and injection both key off the original request body, so record/replay still regenerate identical traces.
Tests
New coverage for Anthropic/Gemini parsing, streaming merge,
modelFromPath, injection, retry classification, and cache-write billing. README "honest limitations" updated to match.