Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 17 additions & 12 deletions app/src/notes/2026-10-19-america-gov-scores-level-with-gpt-5-5.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,45 +3,50 @@
"date": "2026-10-19",
"title": "America.gov, the government's new assistant, scores level with GPT-5.5",
"paragraphs": [
"America.gov, the General Services Administration's assistant, live since September 29, gets {agovExact}% of PolicyBench answers within $1, weighted by household impact. On the board it would rank #{agovRank} of {nRanked}, level with GPT-5.5 ({gpt55Exact}%); GPT-6 Sol leads at {solExact}%. It does not join the board, which scores models answering alone, with no tools and no web access. America.gov says it uses \"Super Intelligence (SI) to find and explain information from official government sources.\"",
"Between October 14 and 18, PolicyBench sent each of its {households} households to America.gov in a new chat, with the board's own prompt in the JSON answer format, and scored every reply as it scores the board. Six prompts run past the {charLimit} characters America.gov asks a message to stay within, so each went in as two chats asking for half the outputs apiece. Before freezing a run, PolicyBench asks a board model again when its response leaves answers out, and America.gov got the same kind of retry: each refusal went back word for word in a fresh chat, up to twice. On first attempts alone it gets {agovFirstExact}% (#{agovFirstRank}).",
"America.gov's first replies included {firstRefusals:words} refusals. \"I cannot estimate or calculate federal or state tax, credit, payroll, or benefit amounts,\" it replied to the prompt for a Virginia single parent of two. Asked again, {answeredOnRetry:words} of the refusals turned into full answers and {refusedThrice:words} came back word for word the same, which leaves {unansweredRate}% of its scored answers blank; {fullModels} of the {nModels} models on the board answer every one.",
"America.gov is strongest on taxes. It gets {fedTaxRate}% of federal income tax amounts within $1, against a median of {fedTaxMedianRate}% across the board's models, and {stateTaxRate}% of state income tax amounts, against {stateTaxMedianRate}%. Benefits are weaker. On SNAP it gets {snapRate}%, below the median of {snapMedianRate}%, and it put a Michigan couple's SNAP at ${snap072} a year where the reference and {snap072Models} of the {nModels} models say $0. On Medicaid eligibility, the output that carries the most weight, it gets {medicaidRate}% against a median of {medicaidMedianRate}%, and it called both adults in a New York couple eligible where the reference and {medicaid004Models} models say neither is.",
"The current prompts define {peOutputs:words} outputs by PolicyEngine: Medicaid and CHIP eligibility \"under PolicyEngine rules,\" and the two school-meal outputs as whether \"PolicyEngine returns\" support. PolicyBench is revising that wording. At least {peRefusals:words} of America.gov's {firstRefusals:words} refusals cited PolicyEngine; no model on the board declined a household on that ground, though some models' recorded reasoning notes that they cannot run PolicyEngine. Counting only the answers each gave, America.gov is never more than {peMaxGap} points below the median model on those {peOutputs:words} outputs, and without them it would rank #{noPeRank} of {nRanked}.",
"This is one run, from one week in October, and America.gov's terms say answers generated with SI \"can be incomplete or wrong.\" The board's last repair step, asking again for each answer still missing, one at a time, would take at least {rowRepairChats} more chats and could lift America.gov to {rowRepairMaxRate}% at most. The replies, the scores and the method, including how the long prompts were split, are in the data below."
"America.gov, which says it answers questions with AI and information from government websites, scores {agovExact}% on PolicyBench. GPT-5.5, which OpenAI released in April, matches that score with AI alone and no web access, while GPT-6 Sol leads at {solExact}%, also with AI alone. America.gov would rank #{agovRank} of {nRanked}, but PolicyBench keeps its board to AI alone. PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
"The General Services Administration has run America.gov since September 29. Between October 14 and 18, PolicyBench sent each of its {households} households to America.gov in a new chat, with the board's own prompt in the JSON answer format. It scored every reply as it scores the board. Six prompts run past the {charLimit} characters America.gov asks a message to stay within, so PolicyBench split each across two chats asking for half the outputs apiece. Before publishing a run, PolicyBench asks again whenever a board model leaves answers out. America.gov got the same kind of retry: PolicyBench sent each refused prompt again, word for word, in a fresh chat, up to twice. On first attempts alone, America.gov gets {agovFirstExact}% (#{agovFirstRank}).",
"America.gov refused prompts for {firstRefusalHouseholds:words} households on the first attempt. \"I cannot estimate or calculate federal or state tax, credit, payroll, or benefit amounts,\" it replied to the prompt for a Virginia single parent of two. When PolicyBench asked again, America.gov answered {answeredOnRetry:words} of those prompts in full and repeated its refusal word for word on the other {refusedThrice:words}, which leaves {unansweredRate}% of its scored answers blank. Of the {nModels} models on the board, {fullModels} answer every one.",
"On income taxes, America.gov falls between GPT-5.5 and GPT-6 Sol: it gets {fedTaxRate}% of federal amounts within $1, against their {gpt55FedTaxRate}% and {solFedTaxRate}%, and {stateTaxRate}% of state amounts, against {gpt55StateTaxRate}% and {solStateTaxRate}%. It trails both on SNAP amounts and Medicaid eligibility. On SNAP it gets {snapRate}%, against {gpt55SnapRate}% and {solSnapRate}%, and it put a Michigan couple's SNAP at ${snap072} a year, where PolicyEngine and {snap072Models} of the {nModels} models give $0. On Medicaid eligibility, which PolicyBench weights most heavily, it gets {medicaidRate}%, against {gpt55MedicaidRate}% and {solMedicaidRate}%. It called both adults in a New York couple eligible, where PolicyEngine and {medicaid004Models} models find neither eligible.",
"The board's prompts name PolicyEngine, the open-source tax and benefit model behind the references, in {peOutputs:words} output definitions. They ask about Medicaid and CHIP eligibility \"under PolicyEngine rules,\" and ask \"whether PolicyEngine returns\" support for free and for reduced-price school meals. PolicyBench is revising that wording. At least {peRefusals:words} of America.gov's {firstRefusals:words} refusals cited PolicyEngine, and no model on the board declined a household on that ground. On the answers each gave, America.gov leads GPT-5.5 on {peLeads:words} of those {peOutputs:words} outputs and trails it by at most {peMaxGap} points on the others. If PolicyBench left all {peOutputs:words} out for every model, America.gov would rank #{noPeRank} of {nRanked}.",
"America.gov's terms say the answers it generates with AI \"can be incomplete or wrong.\" PolicyBench tested America.gov once and skipped the last retry it gives board models, in which it asks again, one at a time, for each answer still missing. If PolicyBench ran that retry, it would need at least {rowRepairChats} more chats, and America.gov could rise to {rowRepairMaxRate}% at most."
],
"facts": {
"agovExact": 87.9,
"agovRank": 14,
"nRanked": 51,
"gpt55Exact": 87.9,
"solExact": 95.2,
"households": 100,
"charLimit": 8000,
"agovFirstExact": 86.6,
"agovFirstRank": 19,
"firstRefusals": 5,
"firstRefusalHouseholds": 5,
"answeredOnRetry": 2,
"refusedThrice": 3,
"unansweredRate": 3.7,
"fullModels": 46,
"nModels": 50,
"fedTaxRate": 72.6,
"fedTaxMedianRate": 54.8,
"gpt55FedTaxRate": 70.2,
"solFedTaxRate": 90.5,
"stateTaxRate": 66.7,
"stateTaxMedianRate": 60.2,
"gpt55StateTaxRate": 65.6,
"solStateTaxRate": 72.0,
"snapRate": 81.7,
"snapMedianRate": 84.9,
"gpt55SnapRate": 84.9,
"solSnapRate": 89.2,
"snap072": 6432,
"snap072Models": 49,
"medicaidRate": 92.5,
"medicaidMedianRate": 95.4,
"gpt55MedicaidRate": 96.5,
"solMedicaidRate": 97.7,
"medicaid004Models": 49,
"rowRepairChats": 73,
"rowRepairMaxRate": 89.8,
"peRefusals": 2,
"peOutputs": 4,
"peMaxGap": "0.8",
"peLeads": 2,
"peMaxGap": "1.3",
"noPeRank": 13
},
"data": [
Expand Down
70 changes: 61 additions & 9 deletions app/tests/notes.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,24 +213,76 @@ describe("notes", () => {
createElement(NoteArticle, { note, titleLevel: "h1" }),
);
expect(markup).toContain(
"gets 87.9% of PolicyBench answers within $1, weighted by household impact.",
"America.gov, which says it answers questions with AI and information from government websites, scores 87.9% on PolicyBench.",
);
expect(markup).toContain(
"On the board it would rank #14 of 51, level with GPT-5.5 (87.9%); GPT-6 Sol leads at 95.2%.",
"PolicyBench counts a dollar amount as right when it falls within $1 of the reference PolicyEngine computes, and an eligibility answer as right when it matches. It weights each answer by household impact.",
);
expect(markup).toContain(
"GPT-5.5, which OpenAI released in April, matches that score with AI alone and no web access, while GPT-6 Sol leads at 95.2%, also with AI alone.",
);
expect(markup).toContain(
"America.gov would rank #14 of 51, but PolicyBench keeps its board to AI alone.",
);
// Rates keep one decimal, so GPT-6 Sol's state income tax rate reads 72.0%.
expect(markup).toContain(
"On income taxes, America.gov falls between GPT-5.5 and GPT-6 Sol: it gets 72.6% of federal amounts within $1, against their 70.2% and 90.5%, and 66.7% of state amounts, against 65.6% and 72.0%.",
);
expect(markup).toContain(
"It trails both on SNAP amounts and Medicaid eligibility. On SNAP it gets 81.7%, against 84.9% and 89.2%,",
);
expect(markup).toContain(
"On Medicaid eligibility, which PolicyBench weights most heavily, it gets 92.5%, against 96.5% and 97.7%. It called both adults in a New York couple eligible, where PolicyEngine and 49 models find neither eligible.",
);
expect(interpolateNoteText(note, note.paragraphs[0]).length).toBeLessThan(600);
// Whole numbers under ten are spelled out; thousands get separators.
expect(markup).toContain("America.gov's first replies included five refusals.");
expect(markup).toContain("Six prompts run past the 8,000 characters");
expect(markup).toContain("put a Michigan couple's SNAP at $6,432 a year");
expect(markup).toContain(
"At least two of America.gov's five refusals cited PolicyEngine; no model on the board declined a household on that ground, though some models' recorded reasoning notes that they cannot run PolicyEngine.",
"America.gov refused prompts for five households on the first attempt.",
);
expect(markup).toContain(
"Six prompts run past the 8,000 characters America.gov asks a message to stay within, so PolicyBench split each across two chats asking for half the outputs apiece.",
);
expect(markup).toContain(
"When PolicyBench asked again, America.gov answered two of those prompts in full and repeated its refusal word for word on the other three, which leaves 3.7% of its scored answers blank. Of the 50 models on the board, 46 answer every one.",
);
expect(markup).toContain(
"and it put a Michigan couple's SNAP at $6,432 a year, where PolicyEngine and 49 of the 50 models give $0.",
);
expect(markup).toContain(
"The board's prompts name PolicyEngine, the open-source tax and benefit model behind the references, in four output definitions.",
);
// React writes the note's straight double quotes as ".
expect(markup).toContain(
"They ask about Medicaid and CHIP eligibility "under PolicyEngine rules," and ask "whether PolicyEngine returns" support for free and for reduced-price school meals.",
);
expect(markup).toContain(
"PolicyBench is revising that wording. At least two of America.gov's five refusals cited PolicyEngine, and no model on the board declined a household on that ground.",
);
expect(markup).toContain("it replied to the prompt for a Virginia single parent of two.");
expect(markup).toContain("Between October 14 and 18, PolicyBench sent");
expect(markup).toContain(
"never more than 0.8 points below the median model on those four outputs, and without them it would rank #13 of 51.",
"The General Services Administration has run America.gov since September 29.",
);
expect(markup).toContain(
"Between October 14 and 18, PolicyBench sent each of its 100 households to America.gov in a new chat, with the board's own prompt in the JSON answer format. It scored every reply as it scores the board.",
);
expect(markup).toContain(
"Before publishing a run, PolicyBench asks again whenever a board model leaves answers out. America.gov got the same kind of retry: PolicyBench sent each refused prompt again, word for word, in a fresh chat, up to twice. On first attempts alone, America.gov gets 86.6% (#19).",
);
expect(markup).toContain(
"On the answers each gave, America.gov leads GPT-5.5 on two of those four outputs and trails it by at most 1.3 points on the others.",
);
expect(markup).toContain(
"If PolicyBench left all four out for every model, America.gov would rank #13 of 51.",
);
expect(markup).toContain(
"America.gov's terms say the answers it generates with AI "can be incomplete or wrong."",
);
expect(markup).toContain(
"PolicyBench tested America.gov once and skipped the last retry it gives board models, in which it asks again, one at a time, for each answer still missing.",
);
expect(markup).toContain(
"If PolicyBench ran that retry, it would need at least 73 more chats, and America.gov could rise to 89.8% at most.",
);
expect(markup).toContain("would take at least 73 more chats and could lift America.gov to 89.8% at most.");
expect(markup).toContain(
'href="https://github.com/PolicyEngine/policybench/blob/main/notes/data/america_gov_20261019.csv"',
);
Expand Down
Loading
Loading