Repository navigation
Count searches where the calls are actually visible - #35
Merged
Merged
Conversation
Cross-reading the two production traces settles what the event stream can and cannot show. The earlier run's stderr proves `webrun` was called twice, because the hook refused both. That same run's event stream contains thread.started, turn.started, two agent_message items, and turn.completed. Nothing else. Two confirmed web tool calls produced no item at all. So `codex exec --json` does not report a web call, the search counter reads the event stream, and it would have reported "no search observed" on a run that searched perfectly. It never could have worked. The hook is the one component every call passes through, and it is now the record: tool name, decision, and query, one line per call. Every error while writing that line is swallowed, so observation cannot change the decision -- a test proves the gate still allows and denies correctly when the log path is unwritable. The worker merges the hook's record with anything the event stream happened to show, so a future Codex that does emit an item cannot double count. A researched run that recorded no search now names the tools that were refused, or states that no tool was called at all, which is itself the finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxxEQsJEXq9hZmjpNH5PxY
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cross-reading the two production traces settles something the search counter was always going to get wrong.
Trace 1 — stderr proves
webrunwas called twice (the hook refused both). That run's event stream contained:Two confirmed web tool calls. Zero items in the event stream.
Trace 2 (after the hook fix) — no refusals, and the same shape:
thread.started, turn.started, agent_message, turn.completed.What that proves
codex exec --jsondoes not emit an item for a web tool call. The search counter reads the event stream. It would report "no search observed" on a run that searched perfectly, and it never could have worked. Every "no search observed" reading so far is uninformative about whether a search happened.The fix
The hook is the one component every tool call demonstrably passes through — that is how
webrunwas identified at all. It is now the record:Errors while writing that line are swallowed, because observation must never be able to change a security decision. A test proves the gate still allows
webrunand deniesshellcorrectly when the log path is unwritable.The worker merges the hook's record with anything the event stream happens to show, so a future Codex that does emit items cannot double-count.
The trace now distinguishes the two cases it was conflating
Previously both produced the same uninformative "no search observed".
Tests
115 unit tests (114 pass / 1 Windows-only skip). New in
tool-gate.test.ts: the hook records two allowedwebruncalls with their queries plus one refusedapply_patch; and gating is unaffected when recording fails. Mocked e2e, lint, build clean.What to expect
After deploying, a researched review gives one of three readings, all actionable: N searches with the queries (working), refused: X (name the tool and I permit it), or called no tool at all (the model isn't attempting one — a prompt problem, not plumbing).
Generated by Claude Code