Repository navigation
Search in a mode the sandbox can actually reach - #33
Merged
Merged
Conversation
A real transcript settled this. The reviewer announced it would check the decision's external premises against primary sources, worked for 115 seconds, and concluded: "The web-search attempt returned no usable source, so no external source was relied upon." The tool existed, the policy permitted it, the model used it, and it came back empty. The mode was `live`, which fetches pages from the open web. This sandbox runs read-only with no outbound egress, so those fetches find nothing. `cached` and `indexed` are served from an OpenAI maintained index over the connection the model already holds, which is why they work here. The default is now cached, and SOL_RESEARCH_MODE accepts indexed or live for a deployment whose sandbox has real egress. Cached is also the narrower channel: pre-indexed results rather than arbitrary live pages, so the prompt injection surface is smaller than what the security model previously described. The same transcript carried no search item at all, despite a search having been attempted. A search that returns nothing need not be reported as an event, so the trace can no longer claim the reviewer chose not to search. Zero observed searches is now reported as nothing recorded, alongside the mode used and the review's own account of what it tried. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxxEQsJEXq9hZmjpNH5PxY
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A real event stream settled this. Two lines from it:
So: the tool existed, the policy permitted it, the reviewer used it, and it came back empty. Every previous diagnosis in this chain was looking at the wrong end of the pipe.
Cause
The mode was
live, which fetches pages from the open web. Reviews run--sandbox read-onlyinside a Vercel Sandbox with no outbound egress, so those fetches find nothing.cached(Codex default)indexedliveI picked
liveback when I reasoned "live is the only one that reaches the web." It reaches the web from a machine that can't.cachedreaches the web through OpenAI, over the connection the model already holds.Default is now
cached.SOL_RESEARCH_MODEacceptsindexedorlivefor a deployment whose sandbox has real egress.This also narrows the security surface.
docs/SECURITY.mdwarned that retrieved pages are untrusted input; withcachedthe reviewer reads pre-indexed results rather than arbitrary live pages, so the prompt-injection channel is smaller than what was documented. Updated accordingly.The trace was also lying
That same stream contains no
web_searchitem at all — despite a search having been attempted and reported by the model itself. A search that returns nothing need not be emitted as an event.So "No search ran / the reviewer chose not to search" was an unfounded claim about the model's intent. It now reads:
Same correction applied to the cited-sources-without-a-search case, which was accusing the reviewer of fabrication on the same faulty inference.
Tests
applyWebSearchModewritescachedfor research,disabledotherwise, and passeslivethrough intact when a deployment asks for it107 unit tests (106 pass / 1 Windows-only skip), mocked e2e, lint, build clean.
Generated by Claude Code