Skip to content

feat: check Prax's traces against the secrets proxy's wire record, arguments included - #250

Merged
praxagent merged 1 commit into
mainfrom
feat/wire-record-check
Oct 3, 2026
Merged

praxagent merged 1 commit into
mainfrom
feat/wire-record-check

Conversation

@praxagent

@praxagent praxagent commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Update: traces now record argument hashes, so swapped arguments are caught too

TJ asked whether traces should record arguments for completeness. Now they do, as hashes only:

  • requested_args_sha256 on each tool span: the arguments the model asked for, taken from its response (on_llm_end) and matched by tool_call_id.
  • args_sha256: the arguments the tool actually ran with, taken from the innermost on_tool_start. A first version took the outer wrapper's arguments, which would have hidden a wrapper that rewrote them; the new test caught that.
  • Both hashes use exactly the wire record's canonical form (a parsed JSON string, sorted keys, compact separators), checked against a fixed vector.

check_wire_record.py now matches on the hashes and reports two new findings:

  • ARGS DIFFER: the trace misreports what the model asked for.
  • CHANGED BEFORE RUNNING: the tool ran with other arguments than the model's.

Older traces fall back to name plus time window.

Tests:

  • An integration test through build_agent_loop with a scripted model and Prax's real bind_tool_user_context wrapper: both hashes recorded.
  • A rewriting wrapper is detected.
  • Save/load round-trip; spans without hashes stay compact.
  • 3 new checker cases.
  • make ci green: 4069. Proxy requests during the run were TJ's own TeamWork chat at 18:18–18:20 PDT.

Idea credit: NVIDIA's Open Agent Safety Platform: check the agent's own account against the model path, which the agent doesn't control.

Companion to praxagent/prax-secrets-proxy#3. That PR has the proxy write a hash-chained record of the tool calls each model response asked for: names and argument hashes, never text.

What

scripts/check_wire_record.py WIRE.jsonl [--caller prax-prod] [--graphs DIR] [--hours 24] [--slack 300]

  • First it verifies the record's hash chain. A deleted, edited or reordered line means BROKEN CHAIN, exit 1.
  • Then it lists every tool call on the wire that has no matching tool span in Prax's traces shortly afterwards: activity Prax didn't account for (exit 1).
  • Each span can account for only one call.
  • --caller filters to one proxy caller, for when dev and prod share the proxy.
  • It is read-only and changes nothing.

Honest limits (in the script and the docs)

  • Matching is by tool name and time window, not argument hash, because traces don't record the model's raw arguments. So it catches a hidden or dropped call, not one whose arguments were silently swapped.
  • A call Prax legitimately refused (a floor, a budget) still has a span, so it matches.

Verified

  • 5 tests: all accounted for; a call missing from the traces; one span can't cover two calls; other callers ignored; tampered record.
  • make ci green: 4015 passed.
  • 0 new secrets-proxy requests during the run (678 before, 678 after).
  • Not run against a live wire record yet; that needs proxy chore(main): release 0.1.0 #3 deployed.

Idea credit: NVIDIA's Open Agent Safety Platform — check the agent's account
against the model path, which the agent doesn't control.

scripts/check_wire_record.py verifies the proxy's hash chain, then lists every
tool call the model returned on the wire that Prax's own traces don't show
within a window — activity Prax didn't account for. Read-only; per-caller so
dev and prod sharing the proxy stay apart. Matches on name and time, not
arguments (traces don't hold the model's raw arguments); both limits are in
the docs.
@praxagent
praxagent merged commit cf61e2f into main Oct 3, 2026
1 check passed
@praxagent
praxagent deleted the feat/wire-record-check branch October 3, 2026 00:56
@praxagent praxagent changed the title feat: check Prax's traces against the secrets proxy's wire record feat: check Prax's traces against the secrets proxy's wire record, arguments included Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant