Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/src/notes/2026-09-01-claude-fable-5-1-added.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"date": "2026-09-01",
"title": "Claude Fable 5.1 added",
"paragraphs": [
"Claude Fable 5.1 was released on 2026-09-01 and joined the board the same day. Board row: {exactRate}% of answers within $1, #{rank} of {nModels}. {parsed} of {answers} answers parsed.",
"Anthropic released Claude Fable 5.1 on 2026-09-01, and PolicyBench added it to the board the same day and parsed {parsed} of the model's {answers} answers. Claude Fable 5.1 gets {exactRate}% of answers right and ranks #{rank} of {nModels}. PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
"The model's API rejects forced tool calls, which is how the board collects most models' answers, so its row answers as a JSON object and reasons at the provider default. The serving-configuration table in the paper records the transport for every row.",
"Sensitivity: with the answer tool declared under tool_choice auto, Fable 5.1 scores {autoRate}% (would rank #{autoRank}). Under the same request Claude Fable 5 scores {fable5AutoRate}%. Fable 5's board row ({fable5BoardRate}%) ran chunked under a forced tool call, which switches off its thinking, so the {boardGap}-point gap between the two board rows is mostly request shape.",
"The next board version moves every model to tool_choice auto so each provider's default reasoning engages under the recorded request shape (policybench#139)."
Expand Down
2 changes: 1 addition & 1 deletion app/src/notes/2026-09-05-gpt-6-astra-debuts-second.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"date": "2026-09-05",
"title": "GPT-6 Astra debuts second: the two rules it invented",
"paragraphs": [
"GPT-6 Astra reached the API on 2026-09-04 and joined the board on 2026-09-05 at {astraExact}% of answers within $1, #{astraRank} of {nModels}, behind GPT-5.6 Sol ({solExact}%) and ahead of Claude Fable 5.1 ({fableExact}%, #{fableRank}). Every row on this board is scored on {scoredOutputs} of its {totalOutputs} requested outputs: {excludedOutputs} outputs whose reference depends on an engine input the household facts never list are excluded for every model.",
"GPT-6 Astra reached the API on 2026-09-04 and joined the board on 2026-09-05. It gets {astraExact}% of answers right and ranks #{astraRank} of {nModels}, behind GPT-5.6 Sol ({solExact}%) and ahead of Claude Fable 5.1 ({fableExact}%, #{fableRank}). PolicyBench scores every model on {scoredOutputs} of its {totalOutputs} requested outputs and excludes the other {excludedOutputs}, since their references depend on an engine input the household facts never list. PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
"Row by row, the two OpenAI models agree far more than they differ. Both are right on {bothRight} scored outputs and both wrong on {bothWrong}. The gap is {astraOnlyMisses} outputs Astra misses and Sol gets right, against {solOnlyMisses} the other way.",
"Astra's solo misses cluster in two invented rules. In {esiRows} rows it treats a listed employer-sponsored insurance premium as a pre-tax salary reduction and subtracts it from wages before computing payroll tax, federal income tax or state income tax; the household facts state gross wages, and the reference keeps them. In {medicareRows} rows it marks a household member under 65 who is flagged disabled as Medicare-eligible; the reference grants Medicare before 65 only with 24 months of Social Security disability receipt recorded, and none of these households carry that input. The remaining {otherRows} solo misses are one-off rule errors with no shared mechanism. One further Medicare row of the same kind, on scenario_074, sits on an excluded output and is outside every count here.",
"The judge's row notes name an employer-sponsored insurance premium on {astraEsiMentions} of Astra's annotated rows and on {solEsiMentions} of Sol's. Sol netted the same premium once, on scenario_045, which cost it {solEsiRows} of its {solOnlyMisses} solo misses; the rest of Sol's solo misses share no mechanism.",
Expand Down
2 changes: 1 addition & 1 deletion app/src/notes/2026-09-22-gpt-6-sol-debuts-first.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"date": "2026-09-22",
"title": "GPT-6 Sol debuts first, Claude Opus 5.5 second",
"paragraphs": [
"GPT-6 Sol, GPT-6 Luna and Claude Opus 5.5 joined the board on 2026-09-22. GPT-6 Sol leads at {solExact}% of answers within $1, weighted by household impact, #{solRank} of {nModels} and {solLead} points above Claude Opus 5.5 ({opusExact}%, #{opusRank}). GPT-5.6 Sol, the previous leader, is #{sol56Rank} at {sol56Exact}%.",
"GPT-6 Sol, GPT-6 Luna and Claude Opus 5.5 joined the board on 2026-09-22. GPT-6 Sol gets {solExact}% of answers right, {solLead} points above Claude Opus 5.5 ({opusExact}%, #{opusRank}), and ranks #{solRank} of {nModels}. GPT-5.6 Sol, the previous leader, ranks #{sol56Rank} at {sol56Exact}%. PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
"GPT-6 Luna scores {lunaExact}% (#{lunaRank}) at ${lunaCost} a household, against ${solCost} for GPT-6 Sol and ${opusCost} for Claude Opus 5.5.",
"Claude Opus 5.5's API rejects forced tool calls, as Claude Fable 5.1's does, so its row answers as a JSON object and reasons at the provider default. Claude Opus 5's board row runs the forced tool call, which switches that model's thinking off, and scores {opus5BoardExact}%; its tool_choice auto re-run, where it reasons, scores {opus5AutoExact}%. The two Opus rows differ in serving shape as well as in model, so the {opusGap}-point gap between them is not a measure of the model change alone.",
"Every row is scored on {scoredOutputs} of its {totalOutputs} requested outputs. This release regenerated {regenerated} references and excludes {excluded} outputs from scoring for every model; the reference audit note explains both.",
Expand Down
4 changes: 2 additions & 2 deletions app/src/notes/2026-09-29-claude-sonnet-5-5-debuts-fifth.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
"date": "2026-09-29",
"title": "Claude Sonnet 5.5 debuts fifth as the references move to policyengine-us 2.15.17",
"paragraphs": [
"Claude Sonnet 5.5, Grok 4.7 and DeepSeek V4.1 Flash joined the board on 2026-09-29, bringing it to {nModels} models. Claude Sonnet 5.5 scores {sonnetExact}% of answers within $1, weighted by household impact, #{sonnetRank} of {nModels}. GPT-6 Luna, #{lunaRank}, also rounds to {lunaExact}% and sits about level with it, less than {lunaSonnetGapBelow} points ahead. Grok 4.7 scores {grokExact}% (#{grokRank}) and DeepSeek V4.1 Flash {flashExact}% (#{flashRank}). GPT-6 Sol still leads at {solExact}%, ahead of Claude Opus 5.5 ({opusExact}%) and GPT-5.6 Sol ({sol56Exact}%).",
"Claude Sonnet 5.5, Grok 4.7 and DeepSeek V4.1 Flash joined the board on 2026-09-29, bringing it to {nModels} models. Claude Sonnet 5.5 gets {sonnetExact}% of answers right and ranks #{sonnetRank} of {nModels}. GPT-6 Luna, #{lunaRank}, also rounds to {lunaExact}% and sits about level with Claude Sonnet 5.5, less than {lunaSonnetGapBelow} points ahead. Grok 4.7 scores {grokExact}% (#{grokRank}) and DeepSeek V4.1 Flash {flashExact}% (#{flashRank}). GPT-6 Sol still leads at {solExact}%, ahead of Claude Opus 5.5 ({opusExact}%) and GPT-5.6 Sol ({sol56Exact}%). PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
"Claude Sonnet 5.5 costs ${sonnetCost} a household, against ${lunaCost} for GPT-6 Luna. DeepSeek V4.1 Flash costs ${flashCost} at DeepSeek's peak list price, and Grok 4.7 costs ${grokCost}.",
"Claude Sonnet 5.5's API rejects forced tool calls, as Claude Opus 5.5's and Claude Fable 5.1's do, so its row answers as a JSON object and reasons with adaptive thinking, the provider default. Grok 4.7 answers through the forced tool call with a {grokTimeoutSeconds}-second request timeout, because its whole-household onboarding probe ran past a {grokOnboardingTimeoutSeconds}-second timeout. DeepSeek serves V4.1 Flash under the alias deepseek-flash, and all {flashAnswers} of the row's answers report that alias and one system fingerprint; none reports a version. PolicyBench labels the row from DeepSeek's September 10 release note, which put V4.1 Flash on that alias.",
"The release also moves PolicyBench's scored references from policyengine-us {previousEngine} to {engineVersion}, the newest release when PolicyBench began sweeping the references on 2026-09-29 (uploaded at {engineUploadedUtc} UTC). The newer version includes the {upstreamFixed:words} upstream fixes that PolicyBench applied as sandbox fixes for the September 22 references, and it encodes law the older version lacked. PolicyBench still builds each scored reference from the stated facts and law published before it froze the references on 2026-07-03, so it ported its {conventions:words} publication conventions to the new version. With those conventions and an adapter that keeps Maryland county tax out of state income tax, policyengine-us {checkEngine}, the newest release when PolicyBench checked PyPI on 2026-09-29 at {checkPypiReadUtc} UTC, gives the same value as {engineVersion} for all {totalOutputs} outputs.",
"The move changes {scoredChanges:words} scored references. New Jersey's child tax credit schedule for 2026 to 2028, which the state approved on June 30, raises one household's state refundable credits from ${njBefore} to ${njAfter}. Arizona raised the income limit for its broad-based categorical eligibility from {azLimitBefore}% to {azLimitAfter}% of the poverty guideline effective March, and the new limit gives an Arizona household ${azAfter} of SNAP where the reference was ${azBefore}. policyengine-us now counts child support received as school-meal income, which ends a Pennsylvania household's eligibility for reduced-price meals. A fix to the rounding in New York's Empire State child credit phase-out (policyengine-us #9425) raises a New York household's state refundable credits from ${nyBefore} to ${nyAfter}.",
"PolicyBench stops scoring {newExclusions:words} federal income tax outputs. policyengine-us {engineVersion} counts the whole of a listed state and local tax refund as income. Federal law counts the refund only to the extent the refunded tax lowered the household's federal tax in the year the household paid it, and the prompts do not say whether it did. PolicyBench also stops scoring one California household head's Medicaid eligibility, which its re-review of the household's excluded SNAP output flagged. On disability, the prompt says only that the head is disabled. The head's income, {medicaidIncomePercent}% of the poverty guideline, is above the {expansionLimitPercent}% limit for the adult expansion group, so only a disability pathway leads to Medi-Cal, California's Medicaid program. Medi-Cal's Working Disabled Program requires SSI's definition of disability, which the prompt does not state, and policyengine-us {engineVersion} tests the general disability flag instead. The same unstated fact already keeps the household's SNAP out of scoring. PolicyBench now scores every model on {scoredOutputs} of its {totalOutputs} requested outputs and excludes {excluded}. Another {withinTolerance:words} references move by less than $1, and PolicyBench re-reviewed the {rechecked} excluded outputs whose values moved; all stay excluded.",
"Together, the new references and the Medicaid exclusion raise the exact rate of every one of the {incumbents} earlier models, by {driftMin} to {driftMax} points. None of the {incumbents} matched any of the {newExclusions:words} federal outputs now excluded, and all {incumbents} had matched the Arizona household's old ${azBefore} SNAP reference, which none matches now. On the Medicaid output, {medicaidMatched} of the {incumbents} had matched the reference and {medicaidMissed} had not. Among the {incumbents}, {riserCount:words} models move up in the order: {riserOne} moves above {riserOnePassed}, {riserTwo} above {riserTwoPassed}, {riserThree} above {riserThreePassed}, {riserFour} above {riserFourPassed}, and {riserFive} above {riserFivePassed}. Every other pair keeps its order.",
"Together, the new references and the Medicaid exclusion lift every one of the {incumbents} earlier models, by {driftMin} to {driftMax} points. None of the {incumbents} matched any of the {newExclusions:words} federal outputs now excluded, and all {incumbents} had matched the Arizona household's old ${azBefore} SNAP reference, which none matches now. On the Medicaid output, {medicaidMatched} of the {incumbents} had matched the reference and {medicaidMissed} had not. Among the {incumbents}, {riserCount:words} models move up in the order: {riserOne} moves above {riserOnePassed}, {riserTwo} above {riserTwoPassed}, {riserThree} above {riserThreePassed}, {riserFour} above {riserFourPassed}, and {riserFive} above {riserFivePassed}. Every other pair keeps its order.",
"The Arizona household's income keeps it from qualifying for SNAP under the program's ordinary tests, so it qualifies only through broad-based categorical eligibility, and all {nModels} models answer $0 for it. PolicyBench's October 5 note on such households counts it among five held back by income, with {bbceIncomeHeldCount:words} in {bbceIncomeHeldStates}. That note's data list every model's answer for those five households and for the {bbceAssetHeldCount:words} held back by savings, including the three models this release adds."
],
"facts": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"The Virginia household's head is {va114Age}, and the prompt lists no Medicare enrollment or premium. policyengine-us treats everyone eligible for Medicare as enrolled and counts the standard Part B premium, ${partBPremium} for 2026, as a medical expense. The household itemizes, and Virginia's itemized deductions start from the federal ones, so the premium lowers its Virginia income tax: without it, that output is ${va114Alt} rather than the reference's ${va114Ref}. The premium lowers the household's federal income tax too, and that output's exclusion record names both unstated facts.",
"The {payrollOutputs:words} payroll outputs count an employee share of a state paid-leave or disability premium: {optionalPrograms}. In each of these states the premium is the employer's, and the employer may deduct up to a set share of it from wages but need not, so the law requires no amount of the employee. The prompts ask for mandatory employee state payroll taxes, and the references count the largest share each state's published 2026 rates let the employer deduct, without the prompts saying whether it does. Without that share, the outputs are ${mnAlt} in Minnesota, ${coAlt} in Colorado, ${maAlt} in Massachusetts, and ${nyAlt} in New York, against references of ${mnRef}, ${coRef}, ${maRef}, and ${nyRef}. PolicyBench still scores the payroll outputs of {mandatoryScored:words} households in {mandatoryStates}, where the law puts the contribution on the worker and requires the employer to withhold it.",
"No model's answer is within $1 of the reference on any of the {incomeTaxOutputs:words} income tax outputs. On the {payrollOutputs:words} payroll outputs, {payrollHits} answers match the reference within $1: {mnHits:words} for Minnesota, {coHits} for Colorado, {maHits:words} for Massachusetts, and {nyHits:words} for New York. Another {payrollAltHits} match the amount without the state share.",
"Every model's exact rate, its share of answers within $1 weighted by household impact, rises by {riseMin} to {riseMax} points. GPT-6 Sol still leads at {solExact}%, ahead of Claude Opus 5.5 ({opusExact}%) and GPT-5.6 Sol ({sol56Exact}%). Claude Sonnet 5.5 ({sonnetExact}%) passes GPT-6 Luna ({lunaExact}%) for fourth. Further down, {riserTwo} moves above {riserTwoPassed}, {riserThree} above {riserThreePassed}, and {riserFour} above {riserFourPassed}. Every other pair keeps its order.",
"Every model's share of answers right rises by {riseMin} to {riseMax} points. PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact. GPT-6 Sol still leads at {solExact}%, ahead of Claude Opus 5.5 ({opusExact}%) and GPT-5.6 Sol ({sol56Exact}%). Claude Sonnet 5.5 ({sonnetExact}%) passes GPT-6 Luna ({lunaExact}%) for fourth. Further down, {riserTwo} moves above {riserTwoPassed}, {riserThree} above {riserThreePassed}, and {riserFour} above {riserFourPassed}. Every other pair keeps its order.",
"PolicyBench's next prompt contract remains an unactivated draft (PolicyEngine/policybench#173). It states the state income tax paid for each tax unit or a named convention for estimating withholding; each person's Medicare Part B enrollment and annual net premium; and whether each person's employer withholds the employee share of state paid-leave or disability premiums. Its payroll output definition counts optional employee shares only when the employer withholds them. A later run can score these outputs after its reference builder and a fresh input sweep validate the stated conventions.",
"Two corrections in this release move no score. PolicyBench now reports that it collected model responses from {responseStart} to {responseEnd}, 2026, ending on the UTC date of the last answer; release dashboard-data-20260930 gave {previousResponseEnd}, the date of that release. The audit text for a California household head's Medicaid eligibility now explains the head's income test as the engine applies it: California's ${medicaidDisregard} monthly income disregard, which takes the place of SSI's ${ssiExclusion} general income exclusion, brings countable income under the limit. The earlier text said the engine subtracts a Medicare Part B premium from income, which it does not."
],
Expand Down
Loading
Loading