[MaxGhenis]

WIP draft. The driver passed review; the release is being staged. Nothing is frozen or published.

The next board release, dashboard-data-20261027, adds OpenAI's GPT-6.1 Terra (card: #245, merged) to dashboard-data-20261014, taking the board to 51 models. The same release installs the reviewed 2026-10-22 reference build (#242), which moves the references from policyengine-us 2.41.0 to 2.60.0.

What this PR adds

• scripts/finish_terra.py: the release driver. It is the Gemini 4 Argon driver (finish_argon.py) with Muse Wonder's install-references step. It prepares Terra's pinned run, installs the reference build, re-judges the cases that Terra or the moved references re-open, triages, and exports a hash-bound stage.
• scripts/freeze_terra.py: freezes that bound stage, re-running every gate.
• docs/terra/: the release spec, the seed digest of release 20261014's audit, and the design note. The design note's result lines are placeholders for the lead.
• tests/test_finish_terra.py and tests/test_freeze_terra.py, on synthetic fixtures.

Review so far

• Round 1 (GPT-6.1 Sol): REQUEST_CHANGES. The commits after 9c3fa2e address its four findings:

  1. The judge identity, runner and UTC timestamp are now required for every verdict this release judges, not only when Terra is among the wrong models. The same check over the staged audits of releases 20261014 and 20261013 passes every verdict they re-judged.
  2. The judge provenance note is generated from the record's entries, and export and the freeze refuse any other note.
  3. The freeze, and every step that resumes from stage.json, refuse a stage whose early or partial flag is anything but the boolean false. (install-references and a first bind-seed run before a stage receipt exists.)
  4. Failing-case tests now cover the release gates in the review's table. One synthetic non-dry freeze runs through the real gates. In 42 one-gate-off mutants, 39 make a test fail.

• Rounds 2a and 2b (GPT-6.1 Sol): REQUEST_CHANGES. Round 2a agreed findings 1 to 3 are resolved and finding 4 is partial. Round 2b, a gate-coverage audit, found three more gaps. The 13 commits after d2027b5 address both rounds:

  1. Export and the freeze reject any by_model path before reading predictions, so every reader uses the bound combined predictions.csv.
  2. A case's parse_failure_only flag is derived from the predictions; a flag that disagrees in either direction stops export and the freeze.
  3. Every verdict this release judges must carry the hardened runner's isolation, token authentication and a transcript bound to its prompt and answer. Removing the isolation field no longer skips those checks.
  4. The export receipt binds the repository files the freeze reads (dependency files, the reference build's timing record, the snapshot manifest, data.versions.json and the rendered paper), and scoring's spec and weights files must equal the base commit's.
  5. Export builds its outputs in a scratch copy and promotes them only after every check passes; a refused export leaves the stage byte for byte unchanged.
  6. The provenance record is verified as a temporary candidate before an atomic replace, and each entry's group must match its sidecar's declared account.
  7. The run-state evidence-kind check and the applied-adjudication check each have their own failing case, and the baseline loaders are tested against real committed blobs.

  Each new gate has a reversal check: turning it off makes its new tests fail (10 reversals, 31 failures). The build also checked releases 20261014 and 20261013 against their committed records. Neither release has a by_model path, a parse-only flag that disagrees with its predictions, or a verdict judged without isolation.

• Round 3 (GPT-6.1 Sol): REQUEST_CHANGES at 50f023d. It agreed the round-2 findings are fixed except two write-ordering paths, and found one new gap. The 7 commits after 50f023d address them:

  1. Export and the freeze refuse any verdict-bearing directory outside the manifest's judged cases, and any directory symlink under the cases. The published provenance tally reads only the validated, receipt-bound cases.
  2. Every audit consumer reads predictions through an explicit reader that opens only the bound combined file, never looks for per-model files and never writes. A by_model file that appears during the reference read is refused, and the combined file is left unchanged.
  3. CLI export runs triage and every later check in a disposable copy of the stage, then promotes the accepted outputs together. A refusal at any step leaves every stage file unchanged, its diagnostics, annotations and release receipt included.
  4. The design note lists the freeze's writes in order and says which failures after the first write need the stage restored or re-exported. A pointer-only failure, or an identical repeated freeze, leaves the bindings valid.

  Each fix has a reversal check: turning it off makes its new tests fail (4, 1 and 2 failures). The three shared modules gain keyword-only parameters whose defaults keep earlier releases' behaviour. At 8d01221, tests/test_finish_terra.py has 670 passed and 1 skipped, and tests/test_freeze_terra.py has 221 passed, both run serially.

• Round 4 (GPT-6.1 Sol): AGREE at 8d01221, with no findings. It re-ran round 3's probes against the export and freeze gates, traced every audit consumer to the combined-only reader, checked the freeze's write order against the design note, and compared the shared modules' defaults with releases 20261014 and 20261013.

Staging so far

• docs/terra/input_pins.json pins the board run's predictions and run state (b0cfa0d).
• pyproject.toml and uv.lock pin policyengine-us 2.60.0, the reference build's engine (0805c28).
• The reference build is installed and the stage is prepared. The Opus 5.5 judge is running on the isolated judge login.

Plan

1. Pass round 4. Done.
2. Stage the release from Terra's run, re-judge, triage and export, then freeze.
3. Rerun the publication check on the newest policyengine-us release.
4. Fill the design note's result lines and the paper's release prose.
5. Publish only on Max's go.

Invariants the gates enforce

• Every incumbent keeps release 20261014's serialized stats byte for byte on the base references; only the four approved reference moves may change a score.
• A stage's every input is hash-bound in its receipt, and the freeze rebuilds the payload from those bound inputs before writing anything.
• No rehearsal stage (early or partial) can become a release.
• Every verdict judged for this release names the release's judge, runner and a UTC timestamp, and ran isolated under token authentication.
• Export and the freeze read predictions only from the bound combined file.
• A refused export, including a refusal in the CLI's triage step, leaves every stage file byte for byte unchanged. The freeze's own release gates run before it writes. Checks inside the shared freezer still run after its writes begin, and the design note says which of those failures need a restore or a re-export.
• The published judge provenance counts only the validated, receipt-bound cases.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 27, 2026 3:06pm UTC

[MaxGhenis]

CI note at 9a3bd31: the one failing test, tests/test_reference_upgrade.py::test_the_reference_engine_is_the_last_upgrades[working_tree], is expected at this head. 0805c28 pins policyengine-us 2.60.0 ahead of the freeze, and the freeze is the step that writes the 2.60.0 upgrade's reference refresh and bundle record into the working tree (scripts/freeze_terra.py, the installed engine upgrade's regenerated_at_utc and policyengine_bundles). The rest of the suite passes (6,719 passed, 89 skipped). The test must pass at the freeze head before this PR leaves draft.

🤖 Generated with Claude Code