Repository navigation
Add Claude Fable 5.2 and stop scoring scenario_122's Minnesota refundable credits (release, 52 models) - #260
Merged
Merged
Conversation
Checkpoint of scripts/finish_fable52.py, forked from finish_terra.py, with the release constants, the 20261027 base pins and the no-engine-change block that replaces Terra's reference install, plus docs/fable52/seed_digest.csv. The driver's remaining upgrade paths, the spec, the freeze, the dating script and the tests follow in later commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This release changes no engine, so the driver drops finish_terra's install-references step and the gates that served only it. Prepare checks the base's engine pins and the audit's publication record in their place, and every later step re-checks them through stage_upgrade. The export keeps the scope check from Haiku 5.5's and Terra's drivers: with release 20261027's exclusion record put back, every incumbent's modelStats must equal the base's byte for byte. A new check, exclusion_scope_problems, compares the release payload with that scope export and refuses any difference the ruled records do not explain, the new model's cells included. An early rehearsal stages the release's record under a provisional stage.json, as Terra's staged its build. bind-seed before prepare now only checks the seed and grounding against their pins, because prepare binds the seed itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/fable52/proposed_exclusions.json holds the one record Max's d2180 rules: scenario_122's Minnesota state_refundable_credits, whose $0 reference rests on a Certificate of Rent Paid the prompt never states (reference_audit/2026-10-29-mn-renters-crp). It uses the newest record field order, that of the 2026-10-10 wave. docs/fable52/spec.json pins that file and quotes the rulings: the engine (no change; the references stay 20261027's on policyengine-us 2.60.0), d2180 and r24 as Terra's spec states it. The spec carries the ruled adjudication (prompt_ambiguity, unlisted_input, state_local_rule), the derivation sentence, the wave sentence and empty triage adjudications. snapshot_date and adjudications_written_on hold [LEAD:] placeholders that the driver and the freeze refuse until the lead fills them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The doc covers mechanics only: the base and input hashes (each recomputed), what the driver keeps from Terra's, what goes with the install and what checks the base's state instead, the ruled exclusion as release 20261010's mechanics apply it, the two disclosures the release note carries, the lead's command sequence with real paths, the steps the lead does by hand, and [LEAD: ...] placeholders for every result. It marks the export's exclusion_scope_problems check as new code and explains why the publication check takes the committed build as its snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/freeze_fable52.py forks freeze_terra.py for a release that installs no reference build. It keeps every gate Terra's no-upgrade branches run and replaces the install gates with verify_no_upgrade: a release stage (require_release_stage), no installed build (driver.stage_upgrade, which re-checks the base's engine pins and the audit's publication record), and every staged scoring source at release 20261027's pins with the release's exclusion record. The live description counts the ruled record on policyengine-us 2.60.0 after the base's three engines, and the record's engines must be exactly those four. The record gate passes the spec's triage decisions, as triage does. The freeze states the references' base pins in the frozen manifest, checks the snapshot's us_reference_outputs.csv and the explanations pin, requires real YYYY-MM-DD days in the spec, and preflights the freezer's engine-setup and provenance reads before writing any destination. --finalize re-gates a rerun publication check through stage_upgrade. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release adds a wave, Max's 2026-11-01 ruling (d2180), so the evidence in reference_audit/2026-09-28/verification/judge_verdicts.json needs it: scripts/date_fable52_judge_verdicts.py takes Terra's restatement file and Argon's uncommitted-wave update. Release 20261027 decided no wave, so its seven waves all name their commits and nothing waits to be filled. The new wave joins uncommitted with the spec's adjudications_written_on, and its decision gains the stage verdict it names. The script requires the record's new waves to be exactly the spec's ruled wave (and a triage wave only when the spec lists triage decisions), each deciding exactly the spec's outputs on a real YYYY-MM-DD day on or after the wave. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Synthetic stages, verdicts and mock boards; the real exporter runs the scope check on a three-household MOCK board. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The step tests moved driver.ROOT before reading the base's exclusion record; they now read it once under the checkout's root, as Terra's tests did. The mock boards keep a parsed answer in every model's output group, since the exporter's heatmap MAE is NaN without one. The any-one-output property now states when the scope check refuses an exclusion: one that reorders the exporter's groups, drops programStats' mape or excludes an eligibility output. The ruled output does none of these, which a new test checks on the committed references. The scope check reports a changed field list on its own line instead of folding it into the n/nParsed message; it refuses the same payloads. The dating script's docstring no longer spells the release tag, which the drivers define once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The heatmap and programStats bullet now says the group's entries keep their fields. The paragraph after the bullets names the three kinds of exclusion the check refuses (a group's first scored row, a group's last parsed answer on a nonzero reference, an eligibility output), says that d2180's output makes none of them, and names the tests that check it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two fixture docstrings in the Fable 5.2 driver tests named the real model and the real ruled output, so a reader of the public repository could take them for the unpublished result. They now say "the new model", "MOCK answers" and "MOCK verdict". The fixtures' data stays synthetic and unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven tests pinned the working tree as it stands before the lead fills the spec, freezes and dates, so they would fail on the release PR's last head. - The committed-spec test now accepts each day as the lead's placeholder or a real UTC day on or after the ruling's, and any triage list (the gates hold its form). The placeholder refusals run on filled copies of the spec: the record gate's test, and a parametrized freeze test that replaces the one reading the committed spec. - Each suite's filled spec carries no triage decision, so a triage decision the lead adds leaves the ruled-wave tests as they are. - The dating tests read release 20261027's adjudication record and evidence from git at the base commit, since the freeze and the dating step rewrite the working tree's copies. No gate changes. With the spec's days filled in the working tree, the dating and freeze suites and the spec-reading driver tests pass too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--finalize gated the publication record again only when sweep_timing.json had changed since export, so the 2026-10-29 record (policyengine-us 2.78.0) bound the release whenever the lead skipped the publication check or the check stopped before it wrote. What --finalize now refuses that it accepted before: - a stage whose sweep_timing.json is unchanged since export (it finalized such a stage without re-gating the record); - a publication record whose engine is not PyPI's newest at the check's read (pypi.newest_at_read), or whose PyPI read is not a UTC stamp on or after the spec's snapshot_date: the committed 2026-10-29 record stops it; - a missing or placeholder snapshot day. verify_publication_record takes the day as checked_on_or_after, and only finalize passes it, so every earlier step gates the record as before. The freeze's main hands finalize the spec's snapshot_date. A record of the reference engine itself (PyPI's newest still 2.60.0) passes, as the check writes it. Tests: the finalize test that expected no re-gate is now a refusal test; finalize passes the day to the gate, stops on the base's record through the real gate, and refuses a missing day; the driver's currency check has parametrized cases on the base's record from git and a Hypothesis property (any read day against any snapshot day). Z 112 passed; the driver's 39 publication-record and engine tests passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate_verdicts alone refuses a new or re-opened verdict whose judge was not Claude Opus 5.5, did not run through scripts/run_audit_claude.sh, or is not stamped in UTC, and no test of this driver's copy reached that block: removing it left the suite green. New tests, each on an otherwise valid MOCK verdict: - 16 identity defects (the new model requested or reported, both models reported, none, another runner, a stamp without an offset or off UTC, a bare day, no stamp): a new verdict goes pending, and remove_invalid sets it aside with the identity reason; - the same defects on a case judged anew whose wrong models are incumbents alone (the not-carried arm), and on a seed-carried verdict that names the new model (the other arm), which stops the step; - 12 isolation defects (no judge_isolation, another login, no transcript, a second or another user message, an answer other than verdict.json, a second accepted answer, another tool call, a git working directory, a turn at another effort): pending at validate_verdicts, and refused by verify_judge_provenance, which export runs; - UTC stamps written with Z and with +00:00 pass. No gate changes. The judge, verdict and isolation subset: 91 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t run) Review r1's M3: no test ran the adjudication gate this release changed. This ports Haiku 5.5's tests of ruled_adjudication_problems to the d2180 entry the adjudicate-exclusions step writes, and runs triage's record gate (stage_adjudications) and the freeze's (verify_adjudication_record) without a stub: - the step appends one entry and keeps every base entry's bytes and place (64 decisions, 50 exclusions), and excludes exactly the release record's 50 outputs; - the gate passes the exact built entry and refuses it missing or edited by hand (a Hypothesis property over a value, the key order, an added and a dropped field), and each of 13 named field edits; - stage_adjudications passes the step's record with and without the ruled case re-opened, and refuses the ruled entry edited after the step with "Staged adjudications of the ruled outputs", a changed, dropped, reordered or added base decision, a re-opened case's decision field and a decision the spec does not list; - the freeze's record gate passes the step's record and refuses the edited entry and a record that disagrees with the staged exclusion record; - the one-to-one check refuses a spec with no adjudication for its record, the record decided twice, or an adjudication with no record, and the restated-list check refuses a restatement of an undecided output. Part 2 ports Terra's triage_stage tests through driver.triage itself. The Mac ran on battery, so these tests have not run yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t 2; not yet run) Review r1's M3 asked for Terra's triage_stage tests with the d2180 entry. The triage_stage fixture takes the stage after --step adjudicate-exclusions, installs the release's exclusion record, re-opens the ruled case and gives driver.triage MOCK collected annotations, one MOCK wrong row for each of the 64 decided cases. driver.triage then runs its real record gates: - it accepts the step's own record unchanged, rewrites the ruled case's note to the decided class and leaves no flag or unresolved row; - it refuses an exclusion record the decisions do not make (the base's 49), a decision the spec does not list, judge fields written by hand on a re-opened case, any other change to a recorded decision, a dropped decision, an edited date convention, the ruled entry edited after the step, and a duplicate key the entry gate cannot see; - it stops on a pending verdict, a missing or hedged one, and a case whose wrong_model_count disagrees with its rows. The Mac ran on battery, so these tests have not run yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et run)
Review r1's M2: the equal-treatment gates ran in no test of this copy.
(a) A differential source test. For each function the build report lists
as kept byte for byte (driver 60, freeze 17, dating 4) the Fable 5.2
source must equal Terra's line for line. For each one kept with names
only (driver 20, freeze 7), its code, docstrings aside, must equal Terra's
after a committed rename map: ndiff.py's map plus two temporary-folder
prefixes and two verify_restatements messages that now name the base by
its role. Terra's suites then cover these bodies here, and an edit to one
fails until it gets tests of its own. A guard test shows the comparison
catches a one-token edit (">= 0" to "> 0" in verify_usage).
(c) Two of the report's names-only functions are off the lists:
finalize, which M6 changed and tested, and provenance_note, whose note
now gives the new model as a case's only reason to re-open. Terra's
round-trip property of the note is ported for it, with a test of the
reason sentence.
(b) A spy test that export runs require_combined_predictions,
verify_verdict_inventory, stage_upgrade, both verify_reference_pins (the
committed pins with the release's hash once the pointer names this
release), verify_parse_failure_flags, verify_new_model_inputs and
verify_judge_provenance before build_payload, with their arguments; that
an early or partial export skips only its named gates; and that a
refusal by any gate leaves the payload unbuilt and nothing written.
The Mac ran on battery, so these tests have not run yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… run) Terra's verify_restatements names its base as 20261014; the M2 rename map named it by the 20261027 spelling ndiff prints after its own renames, so the differential test would have failed on verify_restatements. A read of the three scripts against the lists (no test run) now finds every listed function equal: 81 byte for byte, and 27 equal in code after the map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pdftotext joins the halves of a date that the PDF breaks at a hyphen, so the rendered-PDF check of the engine-timing sentences now accepts that one form. The HTML check of the same sentences stays exact. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The GPT-6.1 Terra freeze tests read the live pointer and the snapshot's reference files from the working tree, which this release's freeze moved. They now read both at release 20261027's merge commit, as the earlier releases' freeze tests read theirs. The exclusion-record tests count the 2026-11-01 ruling's record: 50 records, 43 on unlisted inputs, one decided on 2026-11-01 on policyengine-us 2.60.0. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…udit docs/fable52/wording_review.md records two independent reads of the judge's texts for release dashboard-data-20261111, the adjudication of the doubts they raised, and the open questions. The release changes no reference and no scored output on them; PolicyBench will decide them for every model in a later release. docs/fable52/judge_text_errata.md lists the statements in the re-judged cells that a check against primary sources found wrong. The texts are published as written. reference_audit/2026-11-11-car-loan-interest/ computes how the board would read if the listed auto loan interest counted as qualifying for the federal deduction, from the frozen files. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The note states the row's default effort beside the other Claude rows' defaults, the second calls and the one call past 300 seconds, the wording reviews and what the release does with them, the car-loan question with the board under each treatment, and who operated and reviewed the release. docs/fable52/escalated_calls.json holds the two-call households' call records the note cites. The design note's completion record is filled except for the final suite, the release review and publication. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Reverts 976dbf7. The frontier list does not follow one rule across providers (xAI's column is Grok 4.5 although Grok 4.6 and 4.7 joined later, and GPT-6.1 Terra joined without becoming OpenAI's), so this release leaves the list as release 20261027 had it and pins that Claude Fable 5.2 is not a frontier column, as that release pinned GPT-6.1 Terra. Which model leads each provider's group is left for a rule applied to every provider. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… value PolicyBench sends no reasoning setting, so Claude Fable 5.2's board row runs at Anthropic's default effort, high, and Claude Opus 5.5's at medium. The sensitivity runs each model once at the other's default, under a plan fixed before the first call, and changes no board row. POLICYBENCH_REASONING_EFFORT_OVERRIDE sends Anthropic's output_config.effort for one named model and records it in the run fingerprint; with the variable unset the request is the board's. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…k it Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
One entry no longer faults a reading the exclusion record treats as open, the entries of each cell name the same omissions for every row, entries that note one model's match with a recorded alternative give the count of other models that match it too, and the header says who read, who checked and what the page does not cover. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The record now says exactly what the errata cover and what no check has verified, gives each reading of the car-loan question its reasons, states each adjudication's own confidence, adds the recorded points against the values that match Claude Fable 5.2's answers, and says which way each open question with a stated value would move that model's score (some raise it, some lower it). It names who read what without calling the reads independent, and calls the review partitions batches. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ope sentence The design note's command sequence and by-hand steps now name the commits that did each step, in the past tense, and carry no local path or internal queue command. tests/test_paper_results.py pins the paper's sentence on the newest-engine check's scored count from the frozen files, and one docstring in tests/test_adjudications.py says what its test covers. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…bel the sensitivity The README now gives the engine's phase-out start by filing status next to the statute's text, which gives the higher start to joint returns only; no household that states auto loan interest files as a surviving spouse, so the difference changes nothing here. Board (c) is labelled an engine sensitivity that counts all stated interest as qualifying, and the limits say what the audit does not trace. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The three scenario_023 entries now state that the reference counts the listed 403(b) distributions as unearned income on the stated facts, and cite the rule under which a withdrawal from an account counted as a resource would not be income. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The scenario explorer's prediction dialog now shows, under a model's audit sentence, the corrections docs/fable52/judge_text_errata.md lists for that row, with a link to the errata, and one sentence saying that a judge model wrote the note and that PolicyBench has checked only some of them. The judge's text is unchanged. app/src/judgeTextErrata.json is the page's entries in machine-readable form; a test holds the two equal and checks that every quoted span occurs in the frozen text. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… forced tool use The sentence said none of the three rows has a sensitivity run; it now says none has a tool_choice "auto" run and points to the effort sensitivity. The paper is rendered again before the release. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and lab Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The tally of which way each open question would move Claude Fable 5.2's score now covers every question with a stated other value. The record says what PolicyBench rechecked under the $1 rule and calls the adjudications provisional, states that the readers did not see the prompts, makes the Oklahoma SNAP reason conditional, adds the SSI question raised in review, and says what the app shows and what GPT-6.1 Sol's review read. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…with its record The option now resolves the effort once for a run's record and every request, keeps other output_config members, stops on an empty or malformed value or on litellm's drop_params, gives effort requests their own cache namespace, and the snapshot freeze, every freezer's treatment check and the board exports refuse to run while the variable is set. The two runs were made before this change and nothing was rerun; the note says how PolicyBench checked that the tightened option builds the same requests for them. sensitivity/claude-effort-2026-11-plan.md gives the plan's rules word for word and a record of the runs' times and every departure from the plan. The note names who wrote the plan, who ran what and who reviewed it, cites the documented effort defaults, and words each comparison as a difference between runs. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The paper's sentence on the Claude rows that reject forced tool use now names the effort sensitivity. paper/render_paper.py with Quarto 1.9.36, then scripts/freeze_snapshot.py --rendered-only and scripts/freeze_fable52.py --finalize. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The note states the effort defaults in its first paragraph, gives the limits of the effort comparisons, counts which way each open question with a stated value would move Claude Fable 5.2's share and their net direction, says what the car-loan audit shows for every model and for this one, and lists who reviewed what, including what no model from another lab has checked. Its tests pin the new facts and sentences and refuse a note that still carries a placeholder marker. The publication date moves to 2026-11-13. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…corrections The line under a row's note said a judge model wrote it. Release 20261027 reworded some notes in place, so the line now says the note began as a judge model's text and that PolicyBench has since reworded some notes. A test renders a row whose whole note is an amendment's text. The errata header says what GPT-6.1 Sol read of the page, and one entry no longer cites a model's own explanation, which no other entry does. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…hird review The note, the wording review record and the car-loan audit said PolicyBench would decide the wording review's open questions in a later release. Nothing on record sets that schedule, so the texts now say what holds today: the questions stay open for every model and PolicyBench has set no date for deciding them. A test keeps the promise from coming back. The note's last paragraph says the release review ran in three rounds and that GPT-6.1 Sol read drafts, not the published versions. The design record adds the second render, the final suite at e218e0e and the third review round; three markers wait for the publication commit. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…olds on either model map The plan file's header says what `claude-credits exec` is, and its record gives the reason the comparisons stayed on the share of answers right: a comparison added after the results were in view would be a post hoc choice, and each row's bounded score is in the note's table. tests/test_reasoning_effort_override.py asserted the effort beta header, which litellm adds only when its model map does not mark the model as adaptive. CI loads the remote map, so the test now sets the map state itself and checks both states: the header with the runs' state, and no header with the model marked adaptive. The harness change is comments only. The sensitivity's data files are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…at round three left undone The fourth review round, at 697b22f, found two errors in the release's own record. The note and the design record said the release review ran in three rounds while a fourth was running: both now say four, and the tests pin it. The design record said the third round's other points were fixed: it now says which were answered, names the three things this release does not do and the two findings that asked for no change, and records the fourth round. On that round's smaller points: the plan file's header says the API-key tool is one the release's operator uses, kept outside this repository, and that the paragraph it leaves out records the operating session dropping its own bar on running the sensitivity; departure 6 dates its reason; the car-loan audit gives its reason for changing nothing; the design record corrects the order and attribution of the commits that answered GPT-6.1 Sol; and the no-promise test's description matches what it checks, with four more phrases in its list. The design record keeps three markers for the publication commit: the last check's result, the final tests and the publication line. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit fills the record's three markers and changes nothing else. The full suite passed at 697b22f and again at b9a234b (100 files, 8,511 tests passed and 77 skipped each time), and the app's tests passed at b9a234b. A further Claude Opus 5.5 reader checked b9a234b and found nothing that blocked publication and nothing that needed Max's decision; the record gives its four findings. tests/test_notes.py, the one test file that reads this record, passed on this text before the commit. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
November 13, 2026 07:29
MaxGhenis
added a commit
that referenced
this pull request
Nov 13, 2026
Brings in the Claude Fable 5.2 release (#260). One conflict, in sensitivity/claude-thinking-2026-08.md: in the Claude Haiku 5.5 section, the merge keeps this branch's clause about the harness not yet reading Anthropic's thinking tokens and takes main's count of 1,934 scored outputs. policybench/eval_no_tools.py merged without conflict: main's effort override and this branch's thinking-token capture touch separate hunks, and both survive unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Max approved publication on 2026-11-13 UTC, on condition that the final suite passed and that the last review left nothing blocking and nothing that needed his decision. The data release
dashboard-data-20261111went up at 07:28 UTC. The site reads its board from the pointer in this PR, so it keeps release 20261027 until this PR merges.This release adds Anthropic's Claude Fable 5.2 to
dashboard-data-20261027, taking the board to 52 models, and stops scoring one Minnesota output. It changes no engine: the references stay on policyengine-us 2.60.0. The model card landed in #259.Results
Claude Fable 5.2 answered every output for all 100 households, at $31.69 in all and a median 47.7 seconds per household. The board's responses run from 2026-06-12 to 2026-11-09, and the snapshot date is 2026-11-11.
The release stops scoring scenario_122's Minnesota
state_refundable_creditsfor every model. Each of the 51 earlier models' shares moves by less than 0.01 points: the 42 that answered $0 fall, and the 9 that did not rise. No earlier model passes another, and each of the 47 earlier models below Claude Fable 5.2 moves down one place. Claude Fable 5.2 also answered $0, and it ranks #5 with or without the output scored. Every model is scored on 1,934 of its 1,984 requested outputs; the release excludes 50, one of them new, and restores none.What the release does
Run. PolicyBench ran Claude Fable 5.2 on Anthropic's API on 2026-11-09. Its API rejects forced tool calls, so the row answers as a JSON object, as the Claude Fable 5.1, Claude Opus 5.5 and Claude Sonnet 5.5 rows do. The board sends no reasoning setting, so the row ran at Anthropic's default effort, high; Claude Opus 5.5's default is medium. On 6 households the answer stopped at the 16,384-token starting budget, and PolicyBench asked again with twice the budget; one of those calls took 325 seconds, inside the row's 600-second timeout (
docs/fable52/escalated_calls.json).docs/fable52/input_pins.jsonpins the run's predictions and state.Driver.
scripts/finish_fable52.pystarts from Terra's driver and drops its reference install. At every step it checks that the references, their sidecar and the scenarios keep release 20261027's bytes, that the engine pins name policyengine-us 2.60.0, and that the publication record names those references and an engine that moves no scored output. It adds a second scope check: the release payload may differ from an export on release 20261027's exclusion record only in what the added record explains.scripts/freeze_fable52.pyrebuilds the payload from the bound stage and checks the frozen outputs, andscripts/date_fable52_judge_verdicts.pywrites the judge evidence (docs/fable52/judge_verdicts_20261111.json).docs/fable52/design.mdrecords each command and its times.Exclusion. Under d2180, ruled on 2026-11-01, scenario_122's Minnesota
state_refundable_creditsleaves scoring. Minnesota's renter's credit requires a Certificate of Rent Paid, the prompt lists the household's rent and no certificate, and policyengine-us 2.60.0 counts an unlisted certificate as absent, so the reference is $0; with a certificate, the output is $1,921 (reference_audit/2026-10-29-mn-renters-crp/,docs/fable52/proposed_exclusions.json). The exclusion record goes from 49 records to 50. Of the 52 models, 43 answer $0, 8 give a positive amount, none of them $1,921, and 1 gives no answer.References. The release installs no reference build. policyengine-us 2.116.21, PyPI's newest at the check's read at 17:29 UTC on 2026-11-11, gives 2.60.0's value for every scored output and moves three excluded outputs, which keep their decided values: scenario_057's
ssiby -$0.0010, andfederal_income_tax_before_refundable_creditsfor scenario_120 by -$92.97 and scenario_121 by +$10.98 (reference_audit/2026-10-22-engine-upgrade/verification/publication_check_2_116_21.csv). The check read release 20261027's 49-record exclusion record, so its 1,935 scored outputs include the Minnesota output. PyPI's newest reached 2.118.9 by publication (read at 07:27 UTC on 2026-11-13), and the release did not run the check again.Judging. The audit covers 681 cases. It carries 559 verdicts from release 20261027, and Claude Opus 5.5 judged the 122 cases Claude Fable 5.2 joined on 2026-11-11, on the isolated judge login (
docs/fable52/judge_provenance.json). The judge flagged seven references. Six already had decisions, which the release keeps: four are excluded outputs and two are references earlier releases affirmed. The release restates the judge fields of 39 committed decisions on re-opened cases and keeps their decisions. It applies no wording amendment: kept cases carry release 20261027's published wording, and the judge's new texts are published as written.Triage. Three developer decisions of 2026-11-11 (
docs/fable52/spec.json) keep their outputs scored. The release affirms the seventh flagged reference, scenario_100's Montanastate_refundable_credits: the 2023 bill that proposed a Montana child tax credit never became law, and the reference is Montana's earned income credit, 20% of the federal one for 2026. On the federal income tax of scenario_039 and scenario_101, the judge put eight models' rows on each in the class for references that rest on law published after the freeze. Those rows apply the older rates and deductions that were due to return in 2026; P.L. 119-21, enacted on July 4, 2025, before the 2026-07-03 freeze, made the current ones permanent, so the release classes the rows as model errors. Claude Fable 5.2 misses all three outputs, so each decision keeps a reference that the new model's answer misses.Wording review. Claude Opus 5.5 readers, the judge's model, read the judge's new texts on the 122 cells Claude Fable 5.2 answered wrong, and GPT-6.1 Sol readers read the 559 judged cells it answered right. The first read raised 24 questions about a prompt or a reference, and the second raised 21. The release changes no reference and no scored output on the reviews' findings: the first read covered only Claude Fable 5.2's misses. The questions stay open for every model, and PolicyBench has set no date for deciding them. Of the 21 open questions with a stated other value, scoring their cells against that value would raise Claude Fable 5.2's share in 7, lower it in 7 and leave it unchanged in 7. Together, the 21 values would raise its share; the 20 other than the one on auto loan interest would lower it. The wording review record lists each question and where it stands. The errata list the 36 first-read findings of false or unsupported statements that Claude Opus 5.5 checkers upheld, as 105 entries in 31 cells. The app shows each upheld correction to a model's row sentence under the sentence it corrects; corrections to case notes appear only in the errata. The car-loan audit measures one question: the prompts list auto loan interest without saying whether the loan qualifies for the federal deduction for qualified passenger vehicle loan interest, and counting it as qualifying moves 11 scored outputs by more than $1. The audit gives the board with those outputs excluded, and in an engine sensitivity, with them scored against the changed values.
Effort sensitivity. PolicyBench ran Claude Fable 5.2 at effort medium and Claude Opus 5.5 at effort high, once each over the board's 100 households on 2026-11-12, under a plan written before the first call (
sensitivity/claude-effort-2026-11-plan.md, which also records every departure from it). The runs change no board row. On the share of answers right, paired by household, with 95% intervals from 10,000 resamples of the households:Each arm is one run, and the intervals resample households, not runs. The two runs and their probes cost $21.82 against a $45 cap.
POLICYBENCH_REASONING_EFFORT_OVERRIDEsends the effort for one named model. The option resolves the effort once per run and uses that value for the run's record and every request, keeps a request's otheroutput_configmembers, and stops the run on an empty or malformed value, on a level litellm refuses for these models (xhigh,max) or on litellm'sdrop_params. Each effort request has its own cache namespace, and the snapshot freeze, every release freezer's treatment check and the board exports refuse to run while the variable is set. Both runs used the option as committed in 921a788. The guards came after the runs, in 8784615, and PolicyBench checked offline, for all 100 households of both runs, that each first request's body and headers are unchanged.sensitivity/claude-effort-2026-11.mdgives the results, costs and limits.Rulings. scenario_122's Minnesota refundable credits leave scoring under d2180. The four r24 cells (scenarios 002, 107 and 121 federal income tax, and 121 state income tax) stay excluded while d1878 is open.
Site and paper.
tests/test_notes.pyrecomputes the facts of release note2026-11-13-claude-fable-5-2-joins-the-board. The note dates the public board's change to 2026-11-13 and names the 2026-11-11 snapshot. It also records the correction of eleven notes' score definition (Compare America.gov with GPT-5.5 and GPT-6 Sol in its note #254 and Define the score correctly in ten PolicyBench notes #256), which moved no score. The release rescores the thinking sensitivity's summaries on the 52-model board, updates the benchmark card, and re-renders the paper (PDF sha256 62dde0c4).A change made and reverted. 976dbf7 made Claude Fable 5.2 Anthropic's frontier column in the app, and 3d1fe89 reverted it. The frontier list follows no one rule across providers: xAI's column is Grok 4.5, although Grok 4.6 and Grok 4.7 joined later, and GPT-6.1 Terra joined without becoming OpenAI's. The list stays as release 20261027 had it, and a test pins that Claude Fable 5.2 is not a frontier column.
Invariants (tested)
modelStatsentry,heatmaprow andscenarioPredictionscell, and the payload's country, scenario, bundle, weight and exclusion sections, equal release 20261027's payload as serialized; only annotation fields on cases the release re-opened or decided may differ (test_with_the_bases_record_every_incumbent_must_be_byte_identical).nfalls by one, and nothing moves that the added record does not explain (test_for_any_answers_the_ruled_record_moves_only_its_own_output,test_excluding_any_one_output_never_changes_another_outputs_hit). Every model is scored on 1,934 outputs (test_every_incumbent_and_fable52_are_scored_on_1934_outputs).test_any_one_byte_change_to_a_reference_file_is_refused,test_the_release_record_is_the_base_with_the_ruled_record_before_its_tail).test_the_freeze_holds_every_kept_case_to_release_20261027s_files).test_an_early_rehearsal_scores_the_release_record_and_can_never_release).test_at_finalize_the_record_checks_pypis_newest_on_the_snapshot_day).test_the_freeze_and_the_board_exports_refuse_while_the_variable_is_set).tests/test_effort_sensitivity_evidence.pyrecomputes the sensitivity note's scores, intervals, costs, output tokens, latencies and run dates from the committed data and the frozen snapshot.tests/test_car_loan_sensitivity.py).test_release_20261111_note,test_release_20261111_note_claims).Checks
docs/fable52/design.mdfrom b9a234b.tests/test_notes.py, the one test file that reads that record, ran on the head's text: 50 passed, 2 skipped. The app suite ran at the head: 204 passed, none failed, in 24 files.dashboard-data.jsonb171cf6c,predictions.csv.gzb057d873.Review
The session that operated this release runs on Claude Fable 5.2, the model the release adds, and it wrote the effort sensitivity's plan and started its runs. A Claude Opus 5.5 agent built the release driver. A Claude Opus 5.5 agent also wrote the effort option and its scripts, sent the probes and ran the sensitivity's scoring. PolicyBench ruled on the Minnesota output (d2180) on 2026-11-01, eight days before Anthropic released Claude Fable 5.2.
No model from a lab other than Anthropic took part in the first read or the errata check, and none has checked the three triage decisions or round two's adjudications. No model has checked the second read's 86 findings of false or unsupported statements.
Open at publication
From the wording review:
scenario_023 ssi, the classifications that leaned on the cent rule have not been rechecked under the $1 rule beyond round one's, and the scored cells the partnership and educational-assistance questions reach need an engine run for their values under the other reading.From the release review, by decision:
nof 1,984 in the earlier sensitivity summaries beside scores computed on 1,934 outputs.derivation; the GPT-6.1 Terra freeze tests' read of the timing record in the working tree; no test that this release's committed verdicts are the dating script's output on its stage; five pre-existing export tests that always skip; and suite logs without skip reasons.After merge
dashboard-data-20261111's tag to the squash commit and marks the release Latest. Check its run and the tag, then check the board, the note, the model page and the paper PDF on policybench.org.🤖 Generated with Claude Code