[MaxGhenis]

We add a web+pe3 arm to scripts/tool_pilot_argon.py and pin the confirmatory runner's treatment to the commit that adds it. On 2026-11-03 Max amended d2229 to add a pre-registered stage to the confirmatory run. The stage compares web with web+pe3 for GPT-6 Luna, Gemini 3.5 Flash-Lite and Claude Sonnet 5.5, on the calculator stage's 120 households. The runner refuses to start unless the harness's treatment code matches TREATMENT_COMMIT, so the arm has to be in that commit. The runner's web stage comes in a later pull request.

The arm

web+pe3 offers the provider's web search beside the version 3 calculator. It relates to pe3 as web+pe relates to pe. The arm takes pe3's notice and names web search in its opening sentence, and the benchmark-search rule applies. with_web_opening makes that change, and it refuses a notice that doesn't open with the calculator exactly once.

• ARMS, WEB_ARMS, ARM_SUFFIX (+web+pe-tool-v3) and FEATURES_BY_ARM (every calculator feature) gain the arm. uses_web reads WEB_ARMS.
• Tests cover the arm on the Responses path, the Gemini chat path (googleSearch with server-side tool invocations) and the Anthropic chat path (web_search). Each checks the notice, that the calculator schema hash equals pe3's, and the record's treatment. The chat-path tests fail if the arm loses web search.
• Help text and docstrings name the arm. tool_version stays 1 for pe3 and web+pe3. Manifests record it, and ToolPool falls back on it only when a caller passes no tool_features. The harness always passes the arm's features, so they decide what pe3's and web+pe3's calculator does.

Since the first review

• c0a2885 merges v2-microcosm-batches at 27a1903, which carries the confirmatory runner (Confirmatory runner: lanes, one engine pool per model, admission and the ceiling #244) and Max's prompt rulings (v2 prompt: carry out d1970, d1971, d1972 and d1029 #249). Git stopped on one conflict, in the test file's imports: each side added one, and both stay.
• 07ee27f moves TREATMENT_COMMIT from 1c8c645 to c0a2885, the last commit that touches the treatment's code (scripts/tool_pilot_argon.py, policybench/ and the fix module). It also adds the test that v2 prompt: carry out d1970, d1971, d1972 and d1029 #249's delta review suggested. The test runs the preflight's own comparison, so a commit that changes the treatment's code without moving the pin fails in CI. We take the test out after the confirmatory run, before v2-microcosm-batches reaches main.

Invariants

• No other arm changes. We built 13,770 profiles at 27a1903 and at 07ee27f, and each matches byte for byte in notice, profile_treatment, labels and flags:

  • each of the 51 models in policybench.config.MODELS;
  • each arm 27a1903 has, and each feature arm (pe: plus any nonempty set of the seven features);
  • both prompt versions, and the benchmark-search rule on and off where an arm takes it.

  The 3,264 profiles that ask for concise without explain raise the same error at both.

• web+pe3 uses pe3's calculator: the same features and schema hash, for every model.

• web+pe3's notice differs from pe3's only in the opening sentence and the benchmark-search rule.

• TREATMENT_COMMIT names the last commit that touches the treatment's code, and the new test fails when HEAD's treatment code differs from that commit's.

Plan

• Add the arm. An independent Claude Opus 5.5 review approved 294e98f, and ba29023 addresses its low findings
• Merge v2-microcosm-batches (c0a2885) and move the pin (07ee27f)
• Run the tests at 07ee27f on Python 3.12 and policyengine-us 2.41.0. The pin test passes, and it fails when we set the old pin in memory. The 15 web tests pass, and so do all 235 in tests/test_confirmatory_run.py. The whole tests/test_tool_pilot_argon.py had passed 43 tests when it reached the 2 GB limit we set for local runs, so CI runs that file
• Run CI at the head. GitHub runs ci.yml only on pushes to main and on pull requests into it, so ci/pr246-07ee27fb held 07ee27f plus one commit that added that branch to the push trigger. Run 41741496792 passed lint, app and test, and we deleted the branch
• Get an independent review of c0a2885 and 07ee27f. GPT-6.1 Sol compared 07ee27f with 27a1903 and approved it. It found two overstatements in this description, and we corrected both: what the pin test checks, and what reads tool_version
• Merge into v2-microcosm-batches with a merge commit, never a squash, since the pin names c0a2885 and needs it in the branch's history

#240 (v2-microcosm-batches into main) stays a draft until after the confirmatory run.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Nov 3, 2026 11:36pm UTC

[MaxGhenis]

Independent review, GPT-6.1 Sol, round 1: APPROVE at 07ee27f, compared with v2-microcosm-batches at 27a1903. It raised no blocking or should-fix findings.

• Merge and arms. The merge keeps both sides' imports, v2 prompt: carry out d1970, d1971, d1972 and d1029 #249's prompt rulings and the US-only guard. web+pe3 composes pe3's notice, states the engine and appends the benchmark-search rule, and it stays outside LEGACY_TOOL_ARMS.
• Pin. c0a2885 is the last first-parent commit that changes the treatment, and its diff to HEAD across TREATMENT_PATHS is empty. No 1c8c645 reference remains in scripts, tests or docs.
• New test. It runs the preflight's own committed-tree comparison and skips only when the pinned commit object is missing. CI checks out full history, so it runs there.
• Two nits, both in this description: what the pin test checks, and what reads tool_version. We corrected both.

CI ran at this head on ci/pr246-07ee27fb (run 41741496792: lint, app and test passed). We merge with a merge commit, since the pin names c0a2885.

🤖 Generated with Claude Code