Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
47aaf73
WIP: start the Fable 5.2 release driver from finish_terra.py
MaxGhenis Nov 10, 2026
e746bcd
Remove the install path from the Fable 5.2 driver and add a scope check
MaxGhenis Nov 10, 2026
022c770
Add the Fable 5.2 release spec and d2180's exclusion record
MaxGhenis Nov 10, 2026
bf9eb96
Add the Fable 5.2 release design doc
MaxGhenis Nov 10, 2026
a09dee4
Add the Fable 5.2 release freeze
MaxGhenis Nov 10, 2026
67e6ebd
Add the Fable 5.2 judge-verdict dating step
MaxGhenis Nov 10, 2026
107d34e
Write the stage_upgrade docstring in the active voice
MaxGhenis Nov 10, 2026
0de026d
Add the Fable 5.2 driver tests (work in progress, not yet run)
MaxGhenis Nov 10, 2026
f888676
Fix the Fable 5.2 driver tests and name field changes in the scope check
MaxGhenis Nov 10, 2026
d7f7399
Add the Fable 5.2 freeze tests (work in progress, not yet run)
MaxGhenis Nov 10, 2026
d2076d0
Spell out the manifest pins the freeze tests hold to the base
MaxGhenis Nov 10, 2026
6460b75
Add the Fable 5.2 dating tests (work in progress, not yet run)
MaxGhenis Nov 10, 2026
a27594f
Say where the scope check fails closed in the Fable 5.2 design doc
MaxGhenis Nov 10, 2026
9d07233
Reword two MOCK docstrings that read like a real answer (M17)
MaxGhenis Nov 10, 2026
763299c
Let the Fable 5.2 tests pass on the release's later trees (M5)
MaxGhenis Nov 10, 2026
eada3f6
Make --finalize require a current publication check (M6)
MaxGhenis Nov 10, 2026
5c6b019
Test the judge's identity and isolation on the Fable 5.2 driver (M1)
MaxGhenis Nov 10, 2026
a7a474d
Test the ruled-entry gate on the Fable 5.2 driver (M3, part 1; not ye…
MaxGhenis Nov 10, 2026
c896203
Run triage itself on the d2180 record in the Fable 5.2 tests (M3, par…
MaxGhenis Nov 10, 2026
6a3e765
Hold the Fable 5.2 copies of Terra's gates to Terra's code (M2; not y…
MaxGhenis Nov 10, 2026
c80f223
Map Terra's own spelling of two restatement messages (M2 fix; not yet…
MaxGhenis Nov 10, 2026
6e66c93
Freeze a MOCK no-build stage through every gate in the Fable 5.2 test…
MaxGhenis Nov 10, 2026
9ffe587
Freeze the no-build stage end to end in the Fable 5.2 tests (M4, part 2)
MaxGhenis Nov 10, 2026
5077a68
Hold the staged references and explanations at every later step (M7)
MaxGhenis Nov 10, 2026
2679d0f
Test the reference pins at export's and prepare's call sites (M8)
MaxGhenis Nov 10, 2026
3e1765f
Reword MOCK lines that read like Claude Fable 5.2's real answers (as …
MaxGhenis Nov 10, 2026
c812dd1
Hold the scope export to the base outside modelStats (M9)
MaxGhenis Nov 10, 2026
a2cb616
Test every branch of the scope check and fix four properties (M10)
MaxGhenis Nov 10, 2026
a362dbc
Build the live version description from the base's words (M12)
MaxGhenis Nov 10, 2026
880062b
Hold the committed references at every finalize (M13)
MaxGhenis Nov 10, 2026
3e347ab
Hold the dating step to the base's evidence and the frozen day (M14)
MaxGhenis Nov 10, 2026
ed694d7
Read the ruled wave's one-record sentence in the shared record test (…
MaxGhenis Nov 10, 2026
7648129
Name the lead's step when no decided case re-opened (M14 ruling)
MaxGhenis Nov 10, 2026
66786e7
Hold each wave's written day to its verdicts, today and its wave (M16)
MaxGhenis Nov 10, 2026
3ff5d06
Check d2180's record against its audit's files (M18)
MaxGhenis Nov 10, 2026
c9ef5d1
State the lead's step when the publication check moves the ruled outp…
MaxGhenis Nov 10, 2026
13b1ce5
Test verify_usage's refusals through prepare (M11, part 1)
MaxGhenis Nov 10, 2026
2d85f69
Find the ruled record by its key in the record-pin property
MaxGhenis Nov 10, 2026
bafd470
Isolate the dating step's wave check in its day property (M11, part 2)
MaxGhenis Nov 10, 2026
2fb19c3
Let check (a) pass a decided case's classes and case note (N1)
MaxGhenis Nov 10, 2026
67a0295
Hold the values the inherited gates read to Terra's and the runner's …
MaxGhenis Nov 10, 2026
c0e28c6
N3: let a stopped install-exclusions run again
MaxGhenis Nov 10, 2026
df79e18
N4: bind the dating step to export's provenance; preview it in the dr…
MaxGhenis Nov 10, 2026
cba32b9
N5: compute both dating files before writing either
MaxGhenis Nov 10, 2026
f9adc66
N3 ruling: name the recovery for a stopped reinstall
MaxGhenis Nov 10, 2026
ca89f04
N6: describe check (a) as N1 fixed it
MaxGhenis Nov 10, 2026
dfee4a0
N8: hold the export spy's first gate to its argument
MaxGhenis Nov 10, 2026
6fc9797
Q1, Q3: carry release 20261027's wording onto kept cases (not yet run)
MaxGhenis Nov 10, 2026
7395fec
Q1: fix the driver suite's new kept-case tests (not yet run)
MaxGhenis Nov 10, 2026
41556cf
Q1: give the freeze harness a MOCK release 20261027 (not yet run)
MaxGhenis Nov 10, 2026
37054f4
Q2: split the committed-spec pin by the state each part holds in (not…
MaxGhenis Nov 10, 2026
91d58ed
Q4: follow alias reads in the value walk and scan for value changes (…
MaxGhenis Nov 10, 2026
eed1cb6
Q6: put three MOCK lines in "on the MOCK board" wording (not yet run)
MaxGhenis Nov 10, 2026
cc6aecd
Q1: say where the real-data section's frames come from (not yet run)
MaxGhenis Nov 10, 2026
d13b66a
Q1: count Terra's text amendments apart from its reasoning ones (not …
MaxGhenis Nov 10, 2026
43ad9ae
Q8: drop the freeze docstring's claim that every listed wording is pu…
MaxGhenis Nov 10, 2026
f1b8ecf
Q1: hold the kept gate to doubled rows and notes, and the re-opened c…
MaxGhenis Nov 10, 2026
88c8b32
Review r4 P3-3: point the design note's check (a) bullet up to "A kep…
MaxGhenis Nov 10, 2026
0e8a163
Pin the Claude Fable 5.2 run's input hashes
MaxGhenis Nov 11, 2026
73139f5
Record the Claude Fable 5.2 judge provenance and fill the spec
MaxGhenis Nov 11, 2026
9d99c15
Name the release dashboard-data-20261111 and set its snapshot date
MaxGhenis Nov 11, 2026
14b401d
Freeze release dashboard-data-20261111 (before the render)
MaxGhenis Nov 11, 2026
1bf5ec6
Record the publication check on policyengine-us 2.116.21 and date the…
MaxGhenis Nov 11, 2026
4701d0f
docs/benchmark_card.md, docs/paper.md: state release dashboard-data-2…
MaxGhenis Nov 11, 2026
570cea5
paper/index.qmd: narrate the 2026-11-01 exclusion ruling and name the…
MaxGhenis Nov 11, 2026
2deb89a
paper/index.qmd: name Claude Fable 5.2 among the Claude rows that rej…
MaxGhenis Nov 11, 2026
2867053
sensitivity/data: rescore the thinking summaries and per-program asse…
MaxGhenis Nov 11, 2026
b22bb2a
docs/benchmark_card.md: state the newest-engine check of 2026-11-11
MaxGhenis Nov 11, 2026
fe46484
App and paper code: add Claude Fable 5.2 to the model lists and refre…
MaxGhenis Nov 11, 2026
f60d80e
Notes: draft the release note and its tests (unit in progress)
MaxGhenis Nov 11, 2026
b1b6887
Notes: drop two serving sentences from the release note draft and its…
MaxGhenis Nov 12, 2026
2fa0354
Notes: restore the default-effort sentence in the release note draft …
MaxGhenis Nov 12, 2026
7bdfa56
Tests: check the adjudication record after the dating step
MaxGhenis Nov 12, 2026
8ded88f
Move the test pins to the 52-model snapshot
MaxGhenis Nov 12, 2026
976dbf7
App: show Claude Fable 5.2 as Anthropic's frontier model
MaxGhenis Nov 12, 2026
b8b1975
Sensitivity note: restate the board comparisons on the 52-model snapshot
MaxGhenis Nov 12, 2026
615f4e2
Render the paper and bind the rendered artifacts
MaxGhenis Nov 12, 2026
579f090
Tests: read a date broken at a line end in the PDF's text
MaxGhenis Nov 12, 2026
977c098
Tests: hold release 20261027's freeze tests to that release's files
MaxGhenis Nov 12, 2026
9e2ced4
Record the review of the judge's texts, its errata and the car-loan a…
MaxGhenis Nov 12, 2026
5f4b0a3
Finish the release note and the design note's completion record
MaxGhenis Nov 12, 2026
3d1fe89
App: keep Claude Fable 5.1 as Anthropic's frontier column
MaxGhenis Nov 12, 2026
921a788
Add the Claude effort sensitivity and the option that sends an effort…
MaxGhenis Nov 12, 2026
bc640f5
Release note: state the effort sensitivity's four comparisons and lin…
MaxGhenis Nov 12, 2026
db40817
Errata: apply the release review's corrections
MaxGhenis Nov 12, 2026
44214be
Wording review record: apply the release review's corrections
MaxGhenis Nov 12, 2026
2c50d29
Design note: name commits in place of private paths; pin the check-sc…
MaxGhenis Nov 12, 2026
01c9a48
Car-loan audit: state the statute's phase-out beside the engine's; la…
MaxGhenis Nov 12, 2026
c20caeb
Errata: say what the SSI correction rests on
MaxGhenis Nov 12, 2026
0bdf80c
App: show each correction under the judge's sentence it corrects
MaxGhenis Nov 12, 2026
08bc2d9
Paper: name the effort sensitivity beside the Claude rows that reject…
MaxGhenis Nov 13, 2026
56e1f6f
Design note: record the release review and name each reviewer's model…
MaxGhenis Nov 13, 2026
ef4112a
Wording review record: second round of review corrections
MaxGhenis Nov 13, 2026
8784615
Effort sensitivity: tighten the option's guards and publish the plan …
MaxGhenis Nov 13, 2026
7512b09
Render the paper again and bind the rendered artifacts
MaxGhenis Nov 13, 2026
e218e0e
Release note: apply the reviews' findings and date it 2026-11-13
MaxGhenis Nov 13, 2026
be6b961
App: say that PolicyBench has reworded some judge notes; errata: two …
MaxGhenis Nov 13, 2026
2c68890
Release note and records: leave the open questions open, record the t…
MaxGhenis Nov 13, 2026
697b22f
Effort sensitivity: record fixes from the third review; header test h…
MaxGhenis Nov 13, 2026
b9a234b
Correct the review record after the fourth round: four rounds, and wh…
MaxGhenis Nov 13, 2026
da69e68
Design record: the final tests, the last check and the publication line
MaxGhenis Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Binary file modified app/public/paper/policybench.pdf
Binary file not shown.
Binary file modified app/public/paper/web/figures/positive_zero_scatter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
949 changes: 486 additions & 463 deletions app/public/paper/web/index.html

Large diffs are not rendered by default.

49 changes: 49 additions & 0 deletions app/src/components/JudgeTextCorrections.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import {
JUDGE_TEXT_ERRATA_URL,
rowSpan,
type JudgeTextErratum,
} from "../lib/judgeTextErrata";

/**
* PolicyBench's corrections to statements in a judge's text, rendered
* beneath the text they correct. Pass `model` for a row sentence, so a row
* with its own quoted span shows that span. The judge's text itself is left
* as published.
*/
export default function JudgeTextCorrections({
errata,
model,
}: {
errata: JudgeTextErratum[];
model?: string;
}) {
if (errata.length === 0) return null;
return (
<div className="mt-2 space-y-2">
{errata.map((entry) => {
const span = model ? rowSpan(entry, model) : entry.span;
return (
<section
key={`${entry.where}|${entry.span}`}
className="rounded-lg border border-border-subtle bg-surface px-4 py-3"
data-testid="judge-text-correction"
>
<div className="text-[10px] uppercase tracking-[0.14em] text-text-muted font-medium">
Correction from PolicyBench
</div>
<p className="mt-2 text-sm text-text-secondary leading-relaxed">
<span className="text-text-muted">On &ldquo;{span}&rdquo;:</span>{" "}
{entry.correction}{" "}
<a
href={JUDGE_TEXT_ERRATA_URL}
className="underline underline-offset-2 hover:text-text"
>
Errata
</a>
</p>
</section>
);
})}
</div>
);
}
58 changes: 55 additions & 3 deletions app/src/components/ScenarioExplorer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,21 @@ import {
} from "../lib/predictionStatus";
import { modelPageHref } from "../lib/boardScope";
import { mergeScenarioExplanations } from "../lib/explanations";
import {
JUDGE_TEXT_ERRATA_URL,
auditJudgeNames,
isJudgeWrittenNote,
judgeTextErrataForCell,
rowErrata,
type JudgeTextErratum,
} from "../lib/judgeTextErrata";
import {
useExplanations,
type ExplanationsStatus,
} from "../lib/useExplanations";
import frozenServingConfig from "../model-serving-config.json";
import ExclusionNote from "./ExclusionNote";
import JudgeTextCorrections from "./JudgeTextCorrections";
import ProviderMark from "./ProviderMark";
import RequestShapeNotice from "./RequestShapeNotice";

Expand Down Expand Up @@ -272,6 +281,15 @@ export default function ScenarioExplorer({
selectedCell.variable,
)
: undefined;
// PolicyBench's corrections to the judge's texts for the selected output.
const selectedErrata = selectedCell
? judgeTextErrataForCell(
resolvedScenarioId,
selectedCell.variable,
versionId,
liveVersionId,
)
: [];

const dialogRef = useRef<HTMLDialogElement | null>(null);
const householdDialogRef = useRef<HTMLDialogElement | null>(null);
Expand Down Expand Up @@ -874,6 +892,7 @@ export default function ScenarioExplorer({
ref={dialogRef}
selectedCell={selectedCell}
exclusion={selectedExclusion}
errata={selectedErrata}
predictions={filteredPredictions}
country={country}
currencySymbol={currencySymbol}
Expand Down Expand Up @@ -947,6 +966,7 @@ const HouseholdDialog = React.forwardRef<HTMLDialogElement, HouseholdDialogProps
type DetailDialogProps = {
selectedCell: { variable: string; model: string } | null;
exclusion?: ReferenceExclusion;
errata?: JudgeTextErratum[];
predictions: Record<string, Record<string, ScenarioPrediction>>;
country: CountryCode;
currencySymbol: "$" | "£";
Expand All @@ -961,6 +981,7 @@ const DetailDialog = React.forwardRef<HTMLDialogElement, DetailDialogProps>(
{
selectedCell,
exclusion,
errata,
predictions,
country,
currencySymbol,
Expand Down Expand Up @@ -992,6 +1013,7 @@ const DetailDialog = React.forwardRef<HTMLDialogElement, DetailDialogProps>(
<DetailContent
selectedCell={selectedCell}
exclusion={exclusion}
errata={errata}
predictions={predictions}
country={country}
currencySymbol={currencySymbol}
Expand All @@ -1009,6 +1031,8 @@ const DetailDialog = React.forwardRef<HTMLDialogElement, DetailDialogProps>(
export type DetailContentProps = {
selectedCell: { variable: string; model: string };
exclusion?: ReferenceExclusion;
/** The errata entries for this output (judgeTextErrataForCell). */
errata?: JudgeTextErratum[];
predictions: Record<string, Record<string, ScenarioPrediction>>;
country: CountryCode;
currencySymbol: "$" | "£";
Expand All @@ -1025,6 +1049,7 @@ export type DetailContentProps = {
export function DetailContent({
selectedCell,
exclusion,
errata = [],
predictions,
country,
currencySymbol,
Expand Down Expand Up @@ -1071,6 +1096,12 @@ export function DetailContent({
formatFailureLabel(pred.failureSource),
pred.failureSubtype ? pred.failureSubtype.replaceAll("_", " ") : null,
].filter(Boolean) as string[];
const annotationErrata = rowErrata(errata, model, pred.annotation);
const judgeWrittenNote = isJudgeWrittenNote(
pred.annotation,
versionId,
liveVersionId,
);

return (
<div className="relative px-5 py-5">
Expand Down Expand Up @@ -1213,9 +1244,30 @@ export function DetailContent({
</p>
) : null}
{pred.annotation ? (
<p className="mt-2 text-sm text-text-secondary leading-relaxed whitespace-pre-wrap">
{pred.annotation}
</p>
<>
<p className="mt-2 text-sm text-text-secondary leading-relaxed whitespace-pre-wrap">
{pred.annotation}
</p>
<JudgeTextCorrections errata={annotationErrata} model={model} />
{judgeWrittenNote ? (
<p
className="mt-2 text-xs text-text-muted leading-relaxed"
data-testid="judge-note-disclosure"
>
This note began as text that a judge model (
{auditJudgeNames()}) wrote after scoring; PolicyBench has
since reworded some of these notes and has checked only some
of them, and the{" "}
<a
href={JUDGE_TEXT_ERRATA_URL}
className="underline underline-offset-2 hover:text-text"
>
errata
</a>{" "}
list the statements this release&rsquo;s check found wrong.
</p>
) : null}
</>
) : correct || excluded ? null : explanationsStatus !== "ready" ? (
<PendingExplanationNote status={explanationsStatus} />
) : (
Expand Down
8 changes: 4 additions & 4 deletions app/src/data.artifact.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"asset": "dashboard-data.json",
"bytes": 138702498,
"bytes": 141238562,
"repo": "PolicyEngine/policybench",
"sha256": "67cb0d14ca93cefaeaf2678ef6e6f81aea1858bb7b23b1b61d24c46fe378021b",
"tag": "dashboard-data-20261027",
"url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20261027/dashboard-data.json",
"sha256": "b171cf6c7f62283c3dd499e78d4e54be7834ecbf1ed6f2c1061cf664e86c2b1d",
"tag": "dashboard-data-20261111",
"url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20261111/dashboard-data.json",
"version": 1
}
4 changes: 2 additions & 2 deletions app/src/data.versions.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
"artifact": {
"pointer": "live"
},
"description": "Scored reference outputs from policyengine-us 2.60.0 (the 49 excluded outputs keep the values they were decided on: 29 from policyengine-us 1.755.4, 18 from 2.15.17, 2 from 2.38.6, and the 16 that move on 2.60.0 were re-reviewed and stay excluded); same household facts and requested outputs for every model; answer transport and request shape per model as recorded in the serving-configuration table - 51 models",
"description": "Scored reference outputs from policyengine-us 2.60.0 (the 50 excluded outputs keep the values they were decided on: 29 from policyengine-us 1.755.4, 18 from 2.15.17, 2 from 2.38.6, 1 from 2.60.0, and the 16 that move on 2.60.0 were re-reviewed and stay excluded); same household facts and requested outputs for every model; answer transport and request shape per model as recorded in the serving-configuration table - 52 models",
"id": "1.1",
"label": "1.1",
"snapshotLabel": "Snapshot 2026-10-27"
"snapshotLabel": "Snapshot 2026-11-11"
},
{
"artifact": {
Expand Down
Loading
Loading