Skip to content

Archive every model call as encrypted, deduplicated daily traces with verified off-site upload - #12

Merged
rune-gh merged 3 commits into
mainfrom
feat/trace-archive
Nov 20, 2026
Merged

rune-gh merged 3 commits into
mainfrom
feat/trace-archive

Conversation

@rune-gh

@rune-gh rune-gh commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Every prompt, reasoning trace and tool call that passes through the hub was thrown away. The only way to keep them was request-log, which writes debug dumps at about 1.2 GB per hour because agent clients resend the whole conversation on every turn. This PR adds a long-term trace archive. It keeps a durable record of every model call for evals, fine-tuning, and replaying old prompts against future models.

What gets recorded (opt-in; trace-archive.enabled defaults to false):

  • The client request, plus the final upstream request after translation (what the model actually saw).
  • The response, rebuilt into whole messages: text, Claude thinking with signatures and redacted_thinking, tool calls, and Responses reasoning items with encrypted_content stored verbatim so the conversation can be replayed.
  • Metadata: model, alias and response model, account auth index (never tokens), session lineage, latency and TTFT, attempts, status, token usage.
  • Coverage: Claude Messages, OpenAI Responses (HTTP and Codex websocket, including steering and compaction) and Chat Completions. Other protocols are stored as raw frames.

How it's stored:

  • Capture never blocks or fails a request. A bounded queue drops and counts on overflow.
  • System prompts, tools and messages are content-addressed. Each is stored once per daily segment, which deduplicates a growing conversation about 13×.
  • Each day becomes one self-contained JSONL segment, compressed with zstd and encrypted with age. A recipient is required to enable the archive. The hub holds only the public key, so a compromised hub cannot read sealed history.
  • rclone uploads sealed segments off-site (e.g. Google Drive) and verifies the SHA-256 by reading each file back. Retention never deletes data that has not been verified as uploaded.
  • MCP authorization and authorization_token fields and credential headers are masked before hashing.
  • The default directory is <auth-dir>-traces, outside the credential tree.

Tools:

  • t3code-proxy traces export|verify|stats commands.
  • /v8/management/trace-archive/{settings,status,seal-now} endpoints; /v0 is unchanged.
  • A Logging section in the web panel.
  • docs/trace-archive.md specifies the format precisely enough to decode with only age, zstd and jq.

Proof:

  • make verify passes, apart from two TUI tests that also fail on main under WSL; CI is the arbiter.
  • Race tests pass for tracearchive, api and the Codex duplex executor.
  • I ran a manual smoke test of the built binary against a fake Claude SSE upstream:
    • A streamed /v1/messages call was recorded and sealed.
    • traces verify and traces export reproduced the thinking block, its signature, the tool call {"city":"Seoul"} and tokens 42/37.
    • The standalone age CLI plus Python zstd decoded the segment, and the footer SHA-256 matched.
    • No client, upstream or management key bytes appeared in the decrypted archive or the export.

Reviewed with sol-loop: 3 rounds with GPT-6.2 Sol (high). It found 14 issues; 13 are fixed, each with a regression test. Sol did not approve the final state because two narrow Codex duplex cases remain, listed below as follow-ups.

Known follow-ups (rare, and client traffic is unaffected):

  • A queued create's 401 that arrives while the initial response is still active also marks the initial capture as 401.
  • A credential rejection with no response ID, during a duplex turn that has a queued create, is not archived.

Created with GPT-6.2 Astra (implementation, Codex) and Claude Opus 5.5 (coordination, review, verification) in T3 Code.

🤖 Generated with Claude Code

@rune-gh
rune-gh merged commit 84e93a0 into main Nov 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant