Skip to content

Search in a mode the sandbox can actually reach - #33

Merged
cabibbz merged 1 commit into
mainfrom
claude/pwa-chatgpt-model-selection-o38216
Jul 27, 2026
Merged

cabibbz merged 1 commit into
mainfrom
claude/pwa-chatgpt-model-selection-o38216

Conversation

@cabibbz

@cabibbz cabibbz commented Jul 27, 2026

Copy link
Copy Markdown
Owner

A real event stream settled this. Two lines from it:

item_2 — "I'm checking the decision's external premises against primary/official sources, especially Windows capture semantics and the cited graphics-kernel proof of concept."

item_3, 115 seconds later — "The web-search attempt returned no usable source, so no external source was relied upon."

So: the tool existed, the policy permitted it, the reviewer used it, and it came back empty. Every previous diagnosis in this chain was looking at the wrong end of the pipe.

Cause

The mode was live, which fetches pages from the open web. Reviews run --sandbox read-only inside a Vercel Sandbox with no outbound egress, so those fetches find nothing.

Mode Served from Works here
cached (Codex default) OpenAI-maintained index, pre-indexed results ✅
indexed External access gated through the search index ✅
live Fetches the most recent data from the open web ❌ needs egress

I picked live back when I reasoned "live is the only one that reaches the web." It reaches the web from a machine that can't. cached reaches the web through OpenAI, over the connection the model already holds.

Default is now cached. SOL_RESEARCH_MODE accepts indexed or live for a deployment whose sandbox has real egress.

This also narrows the security surface. docs/SECURITY.md warned that retrieved pages are untrusted input; with cached the reviewer reads pre-indexed results rather than arbitrary live pages, so the prompt-injection channel is smaller than what was documented. Updated accordingly.

The trace was also lying

That same stream contains no web_search item at all — despite a search having been attempted and reported by the model itself. A search that returns nothing need not be emitted as an event.

So "No search ran / the reviewer chose not to search" was an unfounded claim about the model's intent. It now reads:

No search observed — Research was allowed and no search was recorded. That can mean the reviewer did not need one, or that a search returned nothing and was never reported as an event. The review's own text usually says which.

Search mode was "cached".

Same correction applied to the cited-sources-without-a-search case, which was accusing the reviewer of fabrication on the same faulty inference.

Tests

  • The worker validates the mode against the documented variants and defaults to one that works without egress
  • applyWebSearchMode writes cached for research, disabled otherwise, and passes live through intact when a deployment asks for it
  • The trace never claims a choice was made

107 unit tests (106 pass / 1 Windows-only skip), mocked e2e, lint, build clean.


Generated by Claude Code

A real transcript settled this. The reviewer announced it would check
the decision's external premises against primary sources, worked for
115 seconds, and concluded: "The web-search attempt returned no usable
source, so no external source was relied upon." The tool existed, the
policy permitted it, the model used it, and it came back empty.

The mode was `live`, which fetches pages from the open web. This
sandbox runs read-only with no outbound egress, so those fetches find
nothing. `cached` and `indexed` are served from an OpenAI maintained
index over the connection the model already holds, which is why they
work here. The default is now cached, and SOL_RESEARCH_MODE accepts
indexed or live for a deployment whose sandbox has real egress.

Cached is also the narrower channel: pre-indexed results rather than
arbitrary live pages, so the prompt injection surface is smaller than
what the security model previously described.

The same transcript carried no search item at all, despite a search
having been attempted. A search that returns nothing need not be
reported as an event, so the trace can no longer claim the reviewer
chose not to search. Zero observed searches is now reported as
nothing recorded, alongside the mode used and the review's own
account of what it tried.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxxEQsJEXq9hZmjpNH5PxY
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sol-review-gate Building Building Preview, Comment Jul 27, 2026 2:23pm

Request Review

@cabibbz
cabibbz merged commit fa6d7f7 into main Jul 27, 2026
2 of 3 checks passed

This branch was successfully deployed

1 active deployment
Preview — b08cc997 Deployed Jul 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants