Skip to content

Add Claude Fable 5.2 and stop scoring scenario_122's Minnesota refundable credits (release, 52 models) - #260

Merged
MaxGhenis merged 101 commits into
mainfrom
release-claude-fable-5.2
Nov 13, 2026
Merged

MaxGhenis merged 101 commits into
mainfrom
release-claude-fable-5.2

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Max approved publication on 2026-11-13 UTC, on condition that the final suite passed and that the last review left nothing blocking and nothing that needed his decision. The data release dashboard-data-20261111 went up at 07:28 UTC. The site reads its board from the pointer in this PR, so it keeps release 20261027 until this PR merges.

This release adds Anthropic's Claude Fable 5.2 to dashboard-data-20261027, taking the board to 52 models, and stops scoring one Minnesota output. It changes no engine: the references stay on policyengine-us 2.60.0. The model card landed in #259.

Results

Answers right Rank Bounded score Bounded rank
Claude Fable 5.2 91.95% #5 of 52 97.81 #3

Claude Fable 5.2 answered every output for all 100 households, at $31.69 in all and a median 47.7 seconds per household. The board's responses run from 2026-06-12 to 2026-11-09, and the snapshot date is 2026-11-11.

The release stops scoring scenario_122's Minnesota state_refundable_credits for every model. Each of the 51 earlier models' shares moves by less than 0.01 points: the 42 that answered $0 fall, and the 9 that did not rise. No earlier model passes another, and each of the 47 earlier models below Claude Fable 5.2 moves down one place. Claude Fable 5.2 also answered $0, and it ranks #5 with or without the output scored. Every model is scored on 1,934 of its 1,984 requested outputs; the release excludes 50, one of them new, and restores none.

What the release does

  • Run. PolicyBench ran Claude Fable 5.2 on Anthropic's API on 2026-11-09. Its API rejects forced tool calls, so the row answers as a JSON object, as the Claude Fable 5.1, Claude Opus 5.5 and Claude Sonnet 5.5 rows do. The board sends no reasoning setting, so the row ran at Anthropic's default effort, high; Claude Opus 5.5's default is medium. On 6 households the answer stopped at the 16,384-token starting budget, and PolicyBench asked again with twice the budget; one of those calls took 325 seconds, inside the row's 600-second timeout (docs/fable52/escalated_calls.json). docs/fable52/input_pins.json pins the run's predictions and state.

  • Driver. scripts/finish_fable52.py starts from Terra's driver and drops its reference install. At every step it checks that the references, their sidecar and the scenarios keep release 20261027's bytes, that the engine pins name policyengine-us 2.60.0, and that the publication record names those references and an engine that moves no scored output. It adds a second scope check: the release payload may differ from an export on release 20261027's exclusion record only in what the added record explains. scripts/freeze_fable52.py rebuilds the payload from the bound stage and checks the frozen outputs, and scripts/date_fable52_judge_verdicts.py writes the judge evidence (docs/fable52/judge_verdicts_20261111.json). docs/fable52/design.md records each command and its times.

  • Exclusion. Under d2180, ruled on 2026-11-01, scenario_122's Minnesota state_refundable_credits leaves scoring. Minnesota's renter's credit requires a Certificate of Rent Paid, the prompt lists the household's rent and no certificate, and policyengine-us 2.60.0 counts an unlisted certificate as absent, so the reference is $0; with a certificate, the output is $1,921 (reference_audit/2026-10-29-mn-renters-crp/, docs/fable52/proposed_exclusions.json). The exclusion record goes from 49 records to 50. Of the 52 models, 43 answer $0, 8 give a positive amount, none of them $1,921, and 1 gives no answer.

  • References. The release installs no reference build. policyengine-us 2.116.21, PyPI's newest at the check's read at 17:29 UTC on 2026-11-11, gives 2.60.0's value for every scored output and moves three excluded outputs, which keep their decided values: scenario_057's ssi by -$0.0010, and federal_income_tax_before_refundable_credits for scenario_120 by -$92.97 and scenario_121 by +$10.98 (reference_audit/2026-10-22-engine-upgrade/verification/publication_check_2_116_21.csv). The check read release 20261027's 49-record exclusion record, so its 1,935 scored outputs include the Minnesota output. PyPI's newest reached 2.118.9 by publication (read at 07:27 UTC on 2026-11-13), and the release did not run the check again.

  • Judging. The audit covers 681 cases. It carries 559 verdicts from release 20261027, and Claude Opus 5.5 judged the 122 cases Claude Fable 5.2 joined on 2026-11-11, on the isolated judge login (docs/fable52/judge_provenance.json). The judge flagged seven references. Six already had decisions, which the release keeps: four are excluded outputs and two are references earlier releases affirmed. The release restates the judge fields of 39 committed decisions on re-opened cases and keeps their decisions. It applies no wording amendment: kept cases carry release 20261027's published wording, and the judge's new texts are published as written.

  • Triage. Three developer decisions of 2026-11-11 (docs/fable52/spec.json) keep their outputs scored. The release affirms the seventh flagged reference, scenario_100's Montana state_refundable_credits: the 2023 bill that proposed a Montana child tax credit never became law, and the reference is Montana's earned income credit, 20% of the federal one for 2026. On the federal income tax of scenario_039 and scenario_101, the judge put eight models' rows on each in the class for references that rest on law published after the freeze. Those rows apply the older rates and deductions that were due to return in 2026; P.L. 119-21, enacted on July 4, 2025, before the 2026-07-03 freeze, made the current ones permanent, so the release classes the rows as model errors. Claude Fable 5.2 misses all three outputs, so each decision keeps a reference that the new model's answer misses.

  • Wording review. Claude Opus 5.5 readers, the judge's model, read the judge's new texts on the 122 cells Claude Fable 5.2 answered wrong, and GPT-6.1 Sol readers read the 559 judged cells it answered right. The first read raised 24 questions about a prompt or a reference, and the second raised 21. The release changes no reference and no scored output on the reviews' findings: the first read covered only Claude Fable 5.2's misses. The questions stay open for every model, and PolicyBench has set no date for deciding them. Of the 21 open questions with a stated other value, scoring their cells against that value would raise Claude Fable 5.2's share in 7, lower it in 7 and leave it unchanged in 7. Together, the 21 values would raise its share; the 20 other than the one on auto loan interest would lower it. The wording review record lists each question and where it stands. The errata list the 36 first-read findings of false or unsupported statements that Claude Opus 5.5 checkers upheld, as 105 entries in 31 cells. The app shows each upheld correction to a model's row sentence under the sentence it corrects; corrections to case notes appear only in the errata. The car-loan audit measures one question: the prompts list auto loan interest without saying whether the loan qualifies for the federal deduction for qualified passenger vehicle loan interest, and counting it as qualifying moves 11 scored outputs by more than $1. The audit gives the board with those outputs excluded, and in an engine sensitivity, with them scored against the changed values.

  • Effort sensitivity. PolicyBench ran Claude Fable 5.2 at effort medium and Claude Opus 5.5 at effort high, once each over the board's 100 households on 2026-11-12, under a plan written before the first call (sensitivity/claude-effort-2026-11-plan.md, which also records every departure from it). The runs change no board row. On the share of answers right, paired by household, with 95% intervals from 10,000 resamples of the households:

    1. Claude Fable 5.2 at medium minus its board row at high: +1.6 points (+0.3 to +3.1).
    2. Claude Opus 5.5 at high minus its board row at medium: +0.2 points (-0.9 to +1.3).
    3. At medium, Claude Fable 5.2 minus Claude Opus 5.5: -0.5 points (-1.5 to +0.5).
    4. At high, Claude Fable 5.2 minus Claude Opus 5.5: -2.3 points (-3.9 to -0.8).

    Each arm is one run, and the intervals resample households, not runs. The two runs and their probes cost $21.82 against a $45 cap. POLICYBENCH_REASONING_EFFORT_OVERRIDE sends the effort for one named model. The option resolves the effort once per run and uses that value for the run's record and every request, keeps a request's other output_config members, and stops the run on an empty or malformed value, on a level litellm refuses for these models (xhigh, max) or on litellm's drop_params. Each effort request has its own cache namespace, and the snapshot freeze, every release freezer's treatment check and the board exports refuse to run while the variable is set. Both runs used the option as committed in 921a788. The guards came after the runs, in 8784615, and PolicyBench checked offline, for all 100 households of both runs, that each first request's body and headers are unchanged. sensitivity/claude-effort-2026-11.md gives the results, costs and limits.

  • Rulings. scenario_122's Minnesota refundable credits leave scoring under d2180. The four r24 cells (scenarios 002, 107 and 121 federal income tax, and 121 state income tax) stay excluded while d1878 is open.

  • Site and paper. tests/test_notes.py recomputes the facts of release note 2026-11-13-claude-fable-5-2-joins-the-board. The note dates the public board's change to 2026-11-13 and names the 2026-11-11 snapshot. It also records the correction of eleven notes' score definition (Compare America.gov with GPT-5.5 and GPT-6 Sol in its note #254 and Define the score correctly in ten PolicyBench notes #256), which moved no score. The release rescores the thinking sensitivity's summaries on the 52-model board, updates the benchmark card, and re-renders the paper (PDF sha256 62dde0c4).

  • A change made and reverted. 976dbf7 made Claude Fable 5.2 Anthropic's frontier column in the app, and 3d1fe89 reverted it. The frontier list follows no one rule across providers: xAI's column is Grok 4.5, although Grok 4.6 and Grok 4.7 joined later, and GPT-6.1 Terra joined without becoming OpenAI's. The list stays as release 20261027 had it, and a test pins that Claude Fable 5.2 is not a frontier column.

Invariants (tested)

  • With release 20261027's exclusion record put back, every incumbent's modelStats entry, heatmap row and scenarioPredictions cell, and the payload's country, scenario, bundle, weight and exclusion sections, equal release 20261027's payload as serialized; only annotation fields on cases the release re-opened or decided may differ (test_with_the_bases_record_every_incumbent_must_be_byte_identical).
  • With the release's record, the excluded output's 52 cells turn unscored, each model's n falls by one, and nothing moves that the added record does not explain (test_for_any_answers_the_ruled_record_moves_only_its_own_output, test_excluding_any_one_output_never_changes_another_outputs_hit). Every model is scored on 1,934 outputs (test_every_incumbent_and_fable52_are_scored_on_1934_outputs).
  • The staged and committed references, their sidecar, the scenarios and their sidecar keep release 20261027's bytes at every step, and the release's exclusion record is release 20261027's with the one ruled record added (test_any_one_byte_change_to_a_reference_file_is_refused, test_the_release_record_is_the_base_with_the_ruled_record_before_its_tail).
  • A case the release keeps carries release 20261027's published rows and case note in every column (test_the_freeze_holds_every_kept_case_to_release_20261027s_files).
  • No early or partial stage becomes a release (test_an_early_rehearsal_scores_the_release_record_and_can_never_release).
  • The export receipt hash-binds every stage input, and the freeze rebuilds the payload from those inputs. Finalize requires a publication record rewritten since export, whose engine was PyPI's newest at its read, on or after the snapshot date, and moved no scored output (test_at_finalize_the_record_checks_pypis_newest_on_the_snapshot_day).
  • The snapshot freeze and the board exports refuse to run while the effort variable is set (test_the_freeze_and_the_board_exports_refuse_while_the_variable_is_set). tests/test_effort_sensitivity_evidence.py recomputes the sensitivity note's scores, intervals, costs, output tokens, latencies and run dates from the committed data and the frozen snapshot.
  • The car-loan audit's board as released equals the frozen board (tests/test_car_loan_sensitivity.py).
  • The note's facts equal a recomputation from the frozen release (test_release_20261111_note, test_release_20261111_note_claims).

Checks

  • CI at da69e68: lint, test, app, paper and both Vercel checks passed. The test job, which leaves out the tests that carry a slow mark, reported 8,458 passed and 92 skipped. The app job's tests: 204 passed, none failed.
  • The local Python suite, one pytest process per file, ran at 697b22f: 100 files, 8,511 tests passed, 77 skipped, none failed. At b9a234b it ran again with the same result: 100 files, 8,511 tests passed, 77 skipped, none failed.
  • The head, da69e68, changes only docs/fable52/design.md from b9a234b. tests/test_notes.py, the one test file that reads that record, ran on the head's text: 50 passed, 2 skipped. The app suite ran at the head: 204 passed, none failed, in 24 files.
  • The digests GitHub records for the release assets equal the pins: dashboard-data.json b171cf6c, predictions.csv.gz b057d873.

Review

The session that operated this release runs on Claude Fable 5.2, the model the release adds, and it wrote the effort sensitivity's plan and started its runs. A Claude Opus 5.5 agent built the release driver. A Claude Opus 5.5 agent also wrote the effort option and its scripts, sent the probes and ran the sensitivity's scoring. PolicyBench ruled on the Minnesota output (d2180) on 2026-11-01, eight days before Anthropic released Claude Fable 5.2.

  • Serving card. Claude Opus 5.5, an Anthropic model, reviewed it, its 600-second timeout included, and approved it at 3531192 in a second round on 2026-11-09.
  • Driver. Claude Opus 5.5 readers other than its builder reviewed it in four rounds from 2026-11-10 to 2026-11-11; the last found no P1 or P2 against the scripts.
  • Triage decisions. A Claude Opus 5.5 adjudicator proposed each of the three from the law, and a second Claude Opus 5.5 reader tried to refute each and could not.
  • Wording review. Claude Opus 5.5 made the first read, 14 of round one's 16 adjudications, all of round two's adjudications and the errata check. GPT-6.1 Sol, an OpenAI model, made the second read, tried to refute each of round one's 14 Claude Opus 5.5 adjudications and adjudicated two questions itself. The briefs for the second read and for the adjudications of round one and of the first seven second-read batches said a dollar answer had to match the reference to the cent; PolicyBench counts an amount within $1 as right. PolicyBench has rechecked round one under the $1 rule.
  • Release review. Claude Opus 5.5 reviewers read the release in four rounds. P1 marks what must be fixed before publication, P2 what should be and P3 what is worth noting.
    • Round one, on 2026-11-12: six reviewers at 9e2ced4, each followed by a Claude Opus 5.5 reader who tried to refute its P1 and P2 findings, made 60 findings (6 P1, 14 P2, 40 P3). After the refuters, 2 of the 20 P1 and P2 findings stood at P1, 10 at P2 and 7 at P3, and 1 was refuted.
    • Round two, from 2026-11-12 to 2026-11-13: four reviewers at 2c50d29, each followed by a refuter, made 37 findings (1 P1, 12 P2, 24 P3). After the refuters, 9 of the 13 P1 and P2 findings stood at P2 and 4 at P3.
    • Round three, on 2026-11-13: three reviewers read the release at e218e0e for the changes since round two, and after each a Claude Opus 5.5 reader tried to refute each finding rated P1 or P2 or marked for Max's decision and said whether it blocked publication. They made 20 findings (4 P2, 16 P3, 2 of the P3s marked for Max's decision). Of the 6 the readers checked, they let 5 stand and upheld 1 in part, leaving 1 at P1, 2 at P2 and 3 at P3. One blocked publication: the app's new sentence under each judge-written row note said a judge model wrote the note, though PolicyBench had reworded some of those notes, nearly all in the previous release. be6b961 fixed that finding and two points in the errata, and 2c68890 and 697b22f answered most of the others, among them the two marked for Max's decision: the texts no longer say that PolicyBench will decide the open questions in a later release, and the plan file's header explains the API-key tool it names. The release does not make three changes the round asked for, which "Open at publication" lists, and two findings asked for no change.
    • Round four, on 2026-11-13: three reviewers read the release at 697b22f for the changes since round three, with a Claude Opus 5.5 refuter after the texts and hygiene reviews. They made 14 findings (3 P2, 11 P3), none marked for Max's decision. The 3 P2s make two points, and the readers let all three stand and said both points blocked publication: the note and the design record said the release review ran in three rounds while the fourth was running, and the record said round three's other points were fixed, though three were not. b9a234b corrects both. Three P3 points stay as they are, which "Open at publication" lists.
  • GPT-6.1 Sol. On 2026-11-12, GPT-6.1 Sol reviewed a text bundle holding drafts of the effort sensitivity's plan, code change, note and summary data, the release note, the wording review record, a sample of the errata, the car-loan audit and the design note's disclosures and completion record. It did not read the repository, and it has not read the published versions. Its verdict was "AGREE WITH CHANGES", with 20 findings (9 P1, 11 P2), and PolicyBench made changes in response. The design record lists the commits that answered the first two rounds and GPT-6.1 Sol, from 5f4b0a3 to e218e0e.
  • Last check. After round four, a Claude Opus 5.5 reader checked b9a234b. It found nothing in that commit that blocked publication and nothing that needed Max's decision. It made four findings (1 P2, 3 P3): the P2 objected to the wording planned for the design record's sentence on CI, which now says what CI runs; one P3 asked for the app-test runs after 7512b09, which the record now gives; and two P3s on the record's wording stay as they are. The head, da69e68, is the design record's last edit: it fills in that result, the final tests and the publication line, and had no review of its own.

No model from a lab other than Anthropic took part in the first read or the errata check, and none has checked the three triage decisions or round two's adjudications. No model has checked the second read's 86 findings of false or unsupported statements.

Open at publication

From the wording review:

  • The open questions, for every model: PolicyBench has decided none of them and has set no date for deciding them. No other lab has tried to refute round two's adjudications, no adjudicator has ruled on the questions the errata check raised or on the 403(b) question in scenario_023 ssi, the classifications that leaned on the cent rule have not been rechecked under the $1 rule beyond round one's, and the scored cells the partnership and educational-assistance questions reach need an engine run for their values under the other reading.
  • The four first-read findings on words used only in Claude Fable 5.2's rows ("invented", "made up" twice, "chose to leave out"), which the errata check did not uphold because their sentences state no false law. The release publishes those rows as the judge wrote them.
  • GPT-6.1 Sol's further requested checks, which have not run, among them a check of the second read's 86 findings and a review of those four findings under the first read's standard of equal treatment.

From the release review, by decision:

  • The paper's word "differ" for the 16 re-reviewed excluded outputs: 18 differ from their decided values, and 16 move.
  • The n of 1,984 in the earlier sensitivity summaries beside scores computed on 1,934 outputs.
  • The subtypes the three triage decisions changed on 71 rows of models already on the board, which no public file mentions.
  • The naming of the person who ruled in the frozen exclusion record and in a case note, which only a new freeze could change.
  • The working-process wording of some commit messages.
  • Five test gaps: the exclusion-record pin's unchecked derivation; the GPT-6.1 Terra freeze tests' read of the timing record in the working tree; no test that this release's committed verdicts are the dating script's output on its stage; five pre-existing export tests that always skip; and suite logs without skip reasons.
  • What rounds three and four leave as it is:
    • Neither the model card nor the usage table says that the table's count of zero reasoning tokens for Claude Fable 5.2 means the tokens were not counted, as the release note does.
    • The effort sensitivity's summary takes the runs' litellm version from the interpreter that builds it, and its script checks that version only against probe records, which the note's Reproducing command does not pass.
    • The sensitivity's records hold the offline check that the tightened option sends the same requests only as it ran on an earlier version of the harness than the one committed; a reviewer's rerun on the code as committed at e218e0e passed.
    • The sensitivity's summary data still say, without the qualifier the note and the plan now carry, that litellm 1.95.0 adds the beta header to a request carrying the effort field. That holds for the runs, which used its bundled model map.
    • A phrase in the sensitivity note on that model map can be read as attached to the effort field.
    • 697b22f's commit message leaves out some of its changes.

After merge

  • The Seal release workflow moves dashboard-data-20261111's tag to the squash commit and marks the release Latest. Check its run and the tag, then check the board, the note, the model page and the paper PDF on policybench.org.

🤖 Generated with Claude Code

Checkpoint of scripts/finish_fable52.py, forked from finish_terra.py, with
the release constants, the 20261027 base pins and the no-engine-change block
that replaces Terra's reference install, plus docs/fable52/seed_digest.csv.
The driver's remaining upgrade paths, the spec, the freeze, the dating
script and the tests follow in later commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 13, 2026 7:31am UTC

Request Review

This release changes no engine, so the driver drops finish_terra's
install-references step and the gates that served only it. Prepare
checks the base's engine pins and the audit's publication record in
their place, and every later step re-checks them through stage_upgrade.

The export keeps the scope check from Haiku 5.5's and Terra's drivers:
with release 20261027's exclusion record put back, every incumbent's
modelStats must equal the base's byte for byte. A new check,
exclusion_scope_problems, compares the release payload with that scope
export and refuses any difference the ruled records do not explain,
the new model's cells included.

An early rehearsal stages the release's record under a provisional
stage.json, as Terra's staged its build. bind-seed before prepare now
only checks the seed and grounding against their pins, because prepare
binds the seed itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 9, 2026 19:42
docs/fable52/proposed_exclusions.json holds the one record Max's
d2180 rules: scenario_122's Minnesota state_refundable_credits, whose
$0 reference rests on a Certificate of Rent Paid the prompt never
states (reference_audit/2026-10-29-mn-renters-crp). It uses the newest
record field order, that of the 2026-10-10 wave.

docs/fable52/spec.json pins that file and quotes the rulings: the
engine (no change; the references stay 20261027's on policyengine-us
2.60.0), d2180 and r24 as Terra's spec states it. The spec carries the
ruled adjudication (prompt_ambiguity, unlisted_input, state_local_rule),
the derivation sentence, the wave sentence and empty triage
adjudications. snapshot_date and adjudications_written_on hold [LEAD:]
placeholders that the driver and the freeze refuse until the lead
fills them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The doc covers mechanics only: the base and input hashes (each
recomputed), what the driver keeps from Terra's, what goes with the
install and what checks the base's state instead, the ruled exclusion
as release 20261010's mechanics apply it, the two disclosures the
release note carries, the lead's command sequence with real paths,
the steps the lead does by hand, and [LEAD: ...] placeholders for every
result. It marks the export's exclusion_scope_problems check as new
code and explains why the publication check takes the committed build
as its snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/freeze_fable52.py forks freeze_terra.py for a release that installs
no reference build. It keeps every gate Terra's no-upgrade branches run and
replaces the install gates with verify_no_upgrade: a release stage
(require_release_stage), no installed build (driver.stage_upgrade, which
re-checks the base's engine pins and the audit's publication record), and
every staged scoring source at release 20261027's pins with the release's
exclusion record.

The live description counts the ruled record on policyengine-us 2.60.0
after the base's three engines, and the record's engines must be exactly
those four. The record gate passes the spec's triage decisions, as triage
does. The freeze states the references' base pins in the frozen manifest,
checks the snapshot's us_reference_outputs.csv and the explanations pin,
requires real YYYY-MM-DD days in the spec, and preflights the freezer's
engine-setup and provenance reads before writing any destination.
--finalize re-gates a rerun publication check through stage_upgrade.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 8 commits November 9, 2026 20:09
The release adds a wave, Max's 2026-11-01 ruling (d2180), so the evidence in
reference_audit/2026-09-28/verification/judge_verdicts.json needs it:
scripts/date_fable52_judge_verdicts.py takes Terra's restatement file and
Argon's uncommitted-wave update. Release 20261027 decided no wave, so its
seven waves all name their commits and nothing waits to be filled. The new
wave joins uncommitted with the spec's adjudications_written_on, and its
decision gains the stage verdict it names. The script requires the record's
new waves to be exactly the spec's ruled wave (and a triage wave only when
the spec lists triage decisions), each deciding exactly the spec's outputs on
a real YYYY-MM-DD day on or after the wave.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Synthetic stages, verdicts and mock boards; the real exporter runs the scope
check on a three-household MOCK board.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The step tests moved driver.ROOT before reading the base's exclusion
record; they now read it once under the checkout's root, as Terra's tests
did. The mock boards keep a parsed answer in every model's output group,
since the exporter's heatmap MAE is NaN without one. The any-one-output
property now states when the scope check refuses an exclusion: one that
reorders the exporter's groups, drops programStats' mape or excludes an
eligibility output. The ruled output does none of these, which a new test
checks on the committed references.

The scope check reports a changed field list on its own line instead of
folding it into the n/nParsed message; it refuses the same payloads. The
dating script's docstring no longer spells the release tag, which the
drivers define once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The heatmap and programStats bullet now says the group's entries keep
their fields. The paragraph after the bullets names the three kinds of
exclusion the check refuses (a group's first scored row, a group's last
parsed answer on a nonzero reference, an eligibility output), says that
d2180's output makes none of them, and names the tests that check it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two fixture docstrings in the Fable 5.2 driver tests named the real model
and the real ruled output, so a reader of the public repository could take
them for the unpublished result. They now say "the new model", "MOCK
answers" and "MOCK verdict". The fixtures' data stays synthetic and
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 3 commits November 9, 2026 22:46
Seven tests pinned the working tree as it stands before the lead fills the
spec, freezes and dates, so they would fail on the release PR's last head.

- The committed-spec test now accepts each day as the lead's placeholder or
  a real UTC day on or after the ruling's, and any triage list (the gates
  hold its form). The placeholder refusals run on filled copies of the spec:
  the record gate's test, and a parametrized freeze test that replaces the
  one reading the committed spec.
- Each suite's filled spec carries no triage decision, so a triage decision
  the lead adds leaves the ruled-wave tests as they are.
- The dating tests read release 20261027's adjudication record and evidence
  from git at the base commit, since the freeze and the dating step rewrite
  the working tree's copies.

No gate changes. With the spec's days filled in the working tree, the
dating and freeze suites and the spec-reading driver tests pass too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--finalize gated the publication record again only when sweep_timing.json
had changed since export, so the 2026-10-29 record (policyengine-us 2.78.0)
bound the release whenever the lead skipped the publication check or the
check stopped before it wrote.

What --finalize now refuses that it accepted before:
- a stage whose sweep_timing.json is unchanged since export (it finalized
  such a stage without re-gating the record);
- a publication record whose engine is not PyPI's newest at the check's
  read (pypi.newest_at_read), or whose PyPI read is not a UTC stamp on or
  after the spec's snapshot_date: the committed 2026-10-29 record stops it;
- a missing or placeholder snapshot day.

verify_publication_record takes the day as checked_on_or_after, and only
finalize passes it, so every earlier step gates the record as before. The
freeze's main hands finalize the spec's snapshot_date. A record of the
reference engine itself (PyPI's newest still 2.60.0) passes, as the check
writes it.

Tests: the finalize test that expected no re-gate is now a refusal test;
finalize passes the day to the gate, stops on the base's record through the
real gate, and refuses a missing day; the driver's currency check has
parametrized cases on the base's record from git and a Hypothesis property
(any read day against any snapshot day). Z 112 passed; the driver's 39
publication-record and engine tests passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate_verdicts alone refuses a new or re-opened verdict whose judge was
not Claude Opus 5.5, did not run through scripts/run_audit_claude.sh, or is
not stamped in UTC, and no test of this driver's copy reached that block:
removing it left the suite green.

New tests, each on an otherwise valid MOCK verdict:
- 16 identity defects (the new model requested or reported, both models
  reported, none, another runner, a stamp without an offset or off UTC, a
  bare day, no stamp): a new verdict goes pending, and remove_invalid sets
  it aside with the identity reason;
- the same defects on a case judged anew whose wrong models are incumbents
  alone (the not-carried arm), and on a seed-carried verdict that names the
  new model (the other arm), which stops the step;
- 12 isolation defects (no judge_isolation, another login, no transcript, a
  second or another user message, an answer other than verdict.json, a
  second accepted answer, another tool call, a git working directory, a
  turn at another effort): pending at validate_verdicts, and refused by
  verify_judge_provenance, which export runs;
- UTC stamps written with Z and with +00:00 pass.

No gate changes. The judge, verdict and isolation subset: 91 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 9, 2026 23:27
…t run)

Review r1's M3: no test ran the adjudication gate this release changed.
This ports Haiku 5.5's tests of ruled_adjudication_problems to the d2180
entry the adjudicate-exclusions step writes, and runs triage's record gate
(stage_adjudications) and the freeze's (verify_adjudication_record)
without a stub:

- the step appends one entry and keeps every base entry's bytes and place
  (64 decisions, 50 exclusions), and excludes exactly the release record's
  50 outputs;
- the gate passes the exact built entry and refuses it missing or edited by
  hand (a Hypothesis property over a value, the key order, an added and a
  dropped field), and each of 13 named field edits;
- stage_adjudications passes the step's record with and without the ruled
  case re-opened, and refuses the ruled entry edited after the step with
  "Staged adjudications of the ruled outputs", a changed, dropped,
  reordered or added base decision, a re-opened case's decision field and
  a decision the spec does not list;
- the freeze's record gate passes the step's record and refuses the edited
  entry and a record that disagrees with the staged exclusion record;
- the one-to-one check refuses a spec with no adjudication for its record,
  the record decided twice, or an adjudication with no record, and the
  restated-list check refuses a restatement of an undecided output.

Part 2 ports Terra's triage_stage tests through driver.triage itself.
The Mac ran on battery, so these tests have not run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t 2; not yet run)

Review r1's M3 asked for Terra's triage_stage tests with the d2180 entry.
The triage_stage fixture takes the stage after --step adjudicate-exclusions,
installs the release's exclusion record, re-opens the ruled case and gives
driver.triage MOCK collected annotations, one MOCK wrong row for each of
the 64 decided cases. driver.triage then runs its real record gates:

- it accepts the step's own record unchanged, rewrites the ruled case's
  note to the decided class and leaves no flag or unresolved row;
- it refuses an exclusion record the decisions do not make (the base's 49),
  a decision the spec does not list, judge fields written by hand on a
  re-opened case, any other change to a recorded decision, a dropped
  decision, an edited date convention, the ruled entry edited after the
  step, and a duplicate key the entry gate cannot see;
- it stops on a pending verdict, a missing or hedged one, and a case whose
  wrong_model_count disagrees with its rows.

The Mac ran on battery, so these tests have not run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et run)

Review r1's M2: the equal-treatment gates ran in no test of this copy.

(a) A differential source test. For each function the build report lists
as kept byte for byte (driver 60, freeze 17, dating 4) the Fable 5.2
source must equal Terra's line for line. For each one kept with names
only (driver 20, freeze 7), its code, docstrings aside, must equal Terra's
after a committed rename map: ndiff.py's map plus two temporary-folder
prefixes and two verify_restatements messages that now name the base by
its role. Terra's suites then cover these bodies here, and an edit to one
fails until it gets tests of its own. A guard test shows the comparison
catches a one-token edit (">= 0" to "> 0" in verify_usage).

(c) Two of the report's names-only functions are off the lists:
finalize, which M6 changed and tested, and provenance_note, whose note
now gives the new model as a case's only reason to re-open. Terra's
round-trip property of the note is ported for it, with a test of the
reason sentence.

(b) A spy test that export runs require_combined_predictions,
verify_verdict_inventory, stage_upgrade, both verify_reference_pins (the
committed pins with the release's hash once the pointer names this
release), verify_parse_failure_flags, verify_new_model_inputs and
verify_judge_provenance before build_payload, with their arguments; that
an early or partial export skips only its named gates; and that a
refusal by any gate leaves the payload unbuilt and nothing written.

The Mac ran on battery, so these tests have not run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… run)

Terra's verify_restatements names its base as 20261014; the M2 rename map
named it by the 20261027 spelling ndiff prints after its own renames, so
the differential test would have failed on verify_restatements. A read of
the three scripts against the lists (no test run) now finds every listed
function equal: 81 byte for byte, and 27 equal in code after the map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 24 commits November 12, 2026 13:25
pdftotext joins the halves of a date that the PDF breaks at a hyphen, so the
rendered-PDF check of the engine-timing sentences now accepts that one form.
The HTML check of the same sentences stays exact.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The GPT-6.1 Terra freeze tests read the live pointer and the snapshot's
reference files from the working tree, which this release's freeze moved.
They now read both at release 20261027's merge commit, as the earlier
releases' freeze tests read theirs.

The exclusion-record tests count the 2026-11-01 ruling's record: 50 records,
43 on unlisted inputs, one decided on 2026-11-01 on policyengine-us 2.60.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…udit

docs/fable52/wording_review.md records two independent reads of the judge's
texts for release dashboard-data-20261111, the adjudication of the doubts
they raised, and the open questions. The release changes no reference and no
scored output on them; PolicyBench will decide them for every model in a
later release.

docs/fable52/judge_text_errata.md lists the statements in the re-judged
cells that a check against primary sources found wrong. The texts are
published as written.

reference_audit/2026-11-11-car-loan-interest/ computes how the board would
read if the listed auto loan interest counted as qualifying for the federal
deduction, from the frozen files.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The note states the row's default effort beside the other Claude rows'
defaults, the second calls and the one call past 300 seconds, the wording
reviews and what the release does with them, the car-loan question with the
board under each treatment, and who operated and reviewed the release.
docs/fable52/escalated_calls.json holds the two-call households' call
records the note cites.

The design note's completion record is filled except for the final suite,
the release review and publication.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Reverts 976dbf7. The frontier list does not follow one rule across
providers (xAI's column is Grok 4.5 although Grok 4.6 and 4.7 joined later,
and GPT-6.1 Terra joined without becoming OpenAI's), so this release leaves
the list as release 20261027 had it and pins that Claude Fable 5.2 is not a
frontier column, as that release pinned GPT-6.1 Terra. Which model leads
each provider's group is left for a rule applied to every provider.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… value

PolicyBench sends no reasoning setting, so Claude Fable 5.2's board row runs
at Anthropic's default effort, high, and Claude Opus 5.5's at medium. The
sensitivity runs each model once at the other's default, under a plan fixed
before the first call, and changes no board row.

POLICYBENCH_REASONING_EFFORT_OVERRIDE sends Anthropic's output_config.effort
for one named model and records it in the run fingerprint; with the variable
unset the request is the board's.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…k it

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
One entry no longer faults a reading the exclusion record treats as open,
the entries of each cell name the same omissions for every row, entries that
note one model's match with a recorded alternative give the count of other
models that match it too, and the header says who read, who checked and what
the page does not cover.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The record now says exactly what the errata cover and what no check has
verified, gives each reading of the car-loan question its reasons, states
each adjudication's own confidence, adds the recorded points against the
values that match Claude Fable 5.2's answers, and says which way each open
question with a stated value would move that model's score (some raise it,
some lower it). It names who read what without calling the reads
independent, and calls the review partitions batches.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ope sentence

The design note's command sequence and by-hand steps now name the commits
that did each step, in the past tense, and carry no local path or internal
queue command. tests/test_paper_results.py pins the paper's sentence on the
newest-engine check's scored count from the frozen files, and one docstring
in tests/test_adjudications.py says what its test covers.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…bel the sensitivity

The README now gives the engine's phase-out start by filing status next to
the statute's text, which gives the higher start to joint returns only; no
household that states auto loan interest files as a surviving spouse, so the
difference changes nothing here. Board (c) is labelled an engine sensitivity
that counts all stated interest as qualifying, and the limits say what the
audit does not trace.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The three scenario_023 entries now state that the reference counts the
listed 403(b) distributions as unearned income on the stated facts, and
cite the rule under which a withdrawal from an account counted as a resource
would not be income.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The scenario explorer's prediction dialog now shows, under a model's audit
sentence, the corrections docs/fable52/judge_text_errata.md lists for that
row, with a link to the errata, and one sentence saying that a judge model
wrote the note and that PolicyBench has checked only some of them. The
judge's text is unchanged. app/src/judgeTextErrata.json is the page's
entries in machine-readable form; a test holds the two equal and checks that
every quoted span occurs in the frozen text.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… forced tool use

The sentence said none of the three rows has a sensitivity run; it now says
none has a tool_choice "auto" run and points to the effort sensitivity. The
paper is rendered again before the release.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and lab

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The tally of which way each open question would move Claude Fable 5.2's
score now covers every question with a stated other value. The record says
what PolicyBench rechecked under the $1 rule and calls the adjudications
provisional, states that the readers did not see the prompts, makes the
Oklahoma SNAP reason conditional, adds the SSI question raised in review,
and says what the app shows and what GPT-6.1 Sol's review read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…with its record

The option now resolves the effort once for a run's record and every request,
keeps other output_config members, stops on an empty or malformed value or on
litellm's drop_params, gives effort requests their own cache namespace, and
the snapshot freeze, every freezer's treatment check and the board exports
refuse to run while the variable is set. The two runs were made before this
change and nothing was rerun; the note says how PolicyBench checked that the
tightened option builds the same requests for them.

sensitivity/claude-effort-2026-11-plan.md gives the plan's rules word for
word and a record of the runs' times and every departure from the plan. The
note names who wrote the plan, who ran what and who reviewed it, cites the
documented effort defaults, and words each comparison as a difference between
runs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The paper's sentence on the Claude rows that reject forced tool use now names
the effort sensitivity. paper/render_paper.py with Quarto 1.9.36, then
scripts/freeze_snapshot.py --rendered-only and scripts/freeze_fable52.py
--finalize.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The note states the effort defaults in its first paragraph, gives the limits
of the effort comparisons, counts which way each open question with a stated
value would move Claude Fable 5.2's share and their net direction, says what
the car-loan audit shows for every model and for this one, and lists who
reviewed what, including what no model from another lab has checked. Its
tests pin the new facts and sentences and refuse a note that still carries a
placeholder marker. The publication date moves to 2026-11-13.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…corrections

The line under a row's note said a judge model wrote it. Release 20261027
reworded some notes in place, so the line now says the note began as a
judge model's text and that PolicyBench has since reworded some notes. A
test renders a row whose whole note is an amendment's text.

The errata header says what GPT-6.1 Sol read of the page, and one entry no
longer cites a model's own explanation, which no other entry does.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…hird review

The note, the wording review record and the car-loan audit said PolicyBench
would decide the wording review's open questions in a later release. Nothing
on record sets that schedule, so the texts now say what holds today: the
questions stay open for every model and PolicyBench has set no date for
deciding them. A test keeps the promise from coming back.

The note's last paragraph says the release review ran in three rounds and
that GPT-6.1 Sol read drafts, not the published versions. The design record
adds the second render, the final suite at e218e0e and the third review
round; three markers wait for the publication commit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…olds on either model map

The plan file's header says what `claude-credits exec` is, and its record
gives the reason the comparisons stayed on the share of answers right: a
comparison added after the results were in view would be a post hoc choice,
and each row's bounded score is in the note's table.

tests/test_reasoning_effort_override.py asserted the effort beta header,
which litellm adds only when its model map does not mark the model as
adaptive. CI loads the remote map, so the test now sets the map state itself
and checks both states: the header with the runs' state, and no header with
the model marked adaptive.

The harness change is comments only. The sensitivity's data files are
unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…at round three left undone

The fourth review round, at 697b22f, found two errors in the release's own
record. The note and the design record said the release review ran in three
rounds while a fourth was running: both now say four, and the tests pin it.
The design record said the third round's other points were fixed: it now
says which were answered, names the three things this release does not do
and the two findings that asked for no change, and records the fourth round.

On that round's smaller points: the plan file's header says the API-key tool
is one the release's operator uses, kept outside this repository, and that
the paragraph it leaves out records the operating session dropping its own
bar on running the sensitivity; departure 6 dates its reason; the car-loan
audit gives its reason for changing nothing; the design record corrects the
order and attribution of the commits that answered GPT-6.1 Sol; and the
no-promise test's description matches what it checks, with four more phrases
in its list.

The design record keeps three markers for the publication commit: the last
check's result, the final tests and the publication line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit fills the record's three markers and changes nothing else.

The full suite passed at 697b22f and again at b9a234b (100 files, 8,511
tests passed and 77 skipped each time), and the app's tests passed at
b9a234b. A further Claude Opus 5.5 reader checked b9a234b and found
nothing that blocked publication and nothing that needed Max's decision;
the record gives its four findings. tests/test_notes.py, the one test file
that reads this record, passed on this text before the commit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 13, 2026 07:29
@MaxGhenis MaxGhenis changed the title Add Claude Fable 5.2 to the board and exclude scenario_122's Minnesota refundable credits Add Claude Fable 5.2 and stop scoring scenario_122's Minnesota refundable credits (release, 52 models) Nov 13, 2026
@MaxGhenis
MaxGhenis merged commit e61eea4 into main Nov 13, 2026
6 checks passed
@MaxGhenis
MaxGhenis deleted the release-claude-fable-5.2 branch November 13, 2026 08:12
MaxGhenis added a commit that referenced this pull request Nov 13, 2026
Brings in the Claude Fable 5.2 release (#260). One conflict, in
sensitivity/claude-thinking-2026-08.md: in the Claude Haiku 5.5 section,
the merge keeps this branch's clause about the harness not yet reading
Anthropic's thinking tokens and takes main's count of 1,934 scored
outputs. policybench/eval_no_tools.py merged without conflict: main's
effort override and this branch's thinking-token capture touch separate
hunks, and both survive unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — da69e68e Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant