Repository navigation
feat: check Prax's traces against the secrets proxy's wire record, arguments included - #250
Merged
Merged
Conversation
Idea credit: NVIDIA's Open Agent Safety Platform — check the agent's account against the model path, which the agent doesn't control. scripts/check_wire_record.py verifies the proxy's hash chain, then lists every tool call the model returned on the wire that Prax's own traces don't show within a window — activity Prax didn't account for. Read-only; per-caller so dev and prod sharing the proxy stay apart. Matches on name and time, not arguments (traces don't hold the model's raw arguments); both limits are in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update: traces now record argument hashes, so swapped arguments are caught too
TJ asked whether traces should record arguments for completeness. Now they do, as hashes only:
requested_args_sha256on each tool span: the arguments the model asked for, taken from its response (on_llm_end) and matched bytool_call_id.args_sha256: the arguments the tool actually ran with, taken from the innermoston_tool_start. A first version took the outer wrapper's arguments, which would have hidden a wrapper that rewrote them; the new test caught that.check_wire_record.pynow matches on the hashes and reports two new findings:Older traces fall back to name plus time window.
Tests:
build_agent_loopwith a scripted model and Prax's realbind_tool_user_contextwrapper: both hashes recorded.make cigreen: 4069. Proxy requests during the run were TJ's own TeamWork chat at 18:18–18:20 PDT.Idea credit: NVIDIA's Open Agent Safety Platform: check the agent's own account against the model path, which the agent doesn't control.
Companion to praxagent/prax-secrets-proxy#3. That PR has the proxy write a hash-chained record of the tool calls each model response asked for: names and argument hashes, never text.
What
scripts/check_wire_record.py WIRE.jsonl [--caller prax-prod] [--graphs DIR] [--hours 24] [--slack 300]--callerfilters to one proxy caller, for when dev and prod share the proxy.Honest limits (in the script and the docs)
Verified
make cigreen: 4015 passed.