[MaxGhenis]

The scorecard side of the pre-publish comparables gate Max asked for on 2026-10-21 ("how do we ensure we always do a full review of comparables to add to scorecard before publishing an analysis - pe-wide"), and of his follow-up the same day ("even if we've already published something, we should do a fresh comp review before big pushes like newsletters"). Before an analysis goes on policyengine.org, or is promoted again, every comparable external estimate is found, recorded here, compared with the analysis's own figures, and every gap over the threshold is explained with a cited cause. The contract is docs/COMPARABLES.md.

The case that prompted it: the £100,000 childcare dashboard published £0.43bn for the 30 hours in 2027-28 on 12 October. CenTax had published £0.80bn in September (HMRC data), and the dashboard listed CenTax as a benchmark. The cause, a Microcosm hours draw, had been filed as microcosm#1126 on 7 October. Nothing required the gap to be explained. Its record, the first one here, fails the gate on seven rows without the annotation that cites microcosm#1126.

What it adds

• data/comparables/producers.json: the one producer list, by country.
  • Max's UK and US lists, kept and pinned by a test.
  • Additions, each with a why: HMRC, DWP, DfE and the Scottish Fiscal Commission (UK); CPSP and CRFB (US).
  • open_search and academic on both lists.
  • Each producer has search hints, where to look, applies_to scopes, claim-source aliases, and an added date (never in the future), so a later addition never invalidates an old search.
• data/comparables/policy.json: gap_threshold 0.25 (a parameter below 1, queued for Max as d1813), promotion_window_days 7 (d1818) and the re-harvest window (183 days).
• scorecard_db/comparables.py: the gate, and the CLI scaffold, check, index, pins, record, due, report and fresh. A directory sources/analysis-<country>-<slug>/ holds analysis.json, search_log.json, manifest.jsonl, claims_staged.jsonl and annotations.json. The validator checks, mechanically where it can:
  • Identity. Every estimate and claim names its metric, unit, period, time basis, conditions.basis (static, post-behavioural, forecast or outturn) and conditions.program; a fiscal year agrees with the period.
  • Searches. Each is dated on a real day, no later than today and no earlier than the producer list; the first is prepublish (on or before publication) or backfill (on or after it), and later ones are reharvest or prepromotion, after publication. Each records the results files it compared (results_pins), and the latest search's pins are the estimates' current ones.
  • Producers. Every listed producer is logged as found, none_found, found_not_staged or not_applicable (only when its scope excludes the analysis). A listed producer's claims are found under its own entry; source is never open_search, academic or policyengine.
  • Values. value parses back from value_raw under a registered normalisation whose scale matches the printed one (m, bn, %), parentheses, minus words and Unicode minus included, for chart readings too; the quote prints the whole number with its sign; NaN is refused; a claim is never re-signed.
  • Comparisons. A compared claim has the estimate's full identity and baseline, is the producer's own and not cumulative; across worlds it needs a reason, a basis and constructed status. An uncompared claim near an estimate names it as nearest_estimate with a concept_mismatch or pe_gap annotation, and one that is the same quantity must be compared.
  • Ranges. Printed ranges are paired bounds in a range_group, judged as an interval.
  • The gap. signs.py orients the external value; a change metric's convention must be one-sided and not inverted. A gap over the threshold, a sign disagreement or a published zero needs an annotation with explains_gap: true, a cause and a basis (a GitHub issue or pull, pinned engine metadata, a document or diagnostic URL with a path, or a file in this repository; placeholders refused). pending, with a tracking issue, only for what a search after publication found.
  • Files. Primaries are hashed in the manifest and every manifest line is cited; JSON with a repeated key fails.
• Added while backfilling and in review round 2:
  • conditions.basis_note and program_note hold free text; a claim may have basis unstated, and conditions.prices (cash by default, real, or unstated for a claim) is part of a figure's identity, so DWP's real-terms table is never compared with a cash estimate;
  • lower_bound/upper_bound for one-sided figures ("over £1 billion"), never compared, like cumulative;
  • a dash between two numbers reads as a range in quotes ("£3-3.4bn");
  • a household calculator records its search with calculator and no estimates;
  • a hashed primary counts as used when a claim cites it, an annotation basis links it or a search lists it as checked;
  • a convention naming the other side of the ledger ("positive = increase in federal revenue" for a spending metric) does not orient;
  • record cites the latest search (or --search-date/--search-kind), so a revised analysis can cite its new search; report runs while gaps are unexplained;
  • analysis.json may list deployments (published in the index) so newsletters can hold a delisted dashboard's own URL to its record; the childcare record lists its Vercel deployment;
  • registry.kind: tracker records a bill scored on the state legislative tracker, and due lists such records.
• data/comparables/index.json: the published index, built from the directories, listing only records that pass. It carries each record's searches with their pins, current pins, comparisons (gap, nearest-estimate gap, sign disagreement, range-group gap) and the entry rules (which tags make an entry an analysis, which may declare no estimate) so app-v2 and newsletters can check they apply the same ones. The scorecard is public, so the fetch needs no token.
• scorecard_db/ingest_analysis_comparables.py (build step analysis_comparables): claims held out, with conditions.analysis and conditions.country; a compared claim's result is the analysis's own pinned figure with the record's status; a claim beside its nearest estimate gets that figure, labelled with the mismatch; a superseded estimate adds a history result. The feeds (populations.json, lanes.json and their app copies) are regenerated, and the pinned export counts rise by exactly the record's 22 rows.
• First record: sources/analysis-uk-childcare-100k-threshold/, a backfill searched on 2026-10-21 across every UK producer, pinned to PolicyEngine/uk-childcare-100k-threshold@f25f0db.
  • CenTax Table 4.2 (central, with the high- and low-cost scenarios as ranges) and Table B.1's 2028-29 row, static and net; the Centre for British Progress's Tables 2 and 3. Primaries match Add CenTax's costing of removing the £100,000 childcare limit #207's hashes.
  • Static rows are compared across worlds with the 30-hours static estimate (gaps −25% to −50%, explained by microcosm#1126; the cause cites the rerun in uk-childcare-100k-threshold#8). Net rows sit beside the dashboard's net cost as a concept mismatch. CBP's net gain disagrees in sign with the dashboard's net cost, explained by the bunching annotation.
  • Logged as found but not staged: DfE's FOI estimate and the Conservatives' £700m.
  • The dashboard was taken down on 21 October (app-v2#1379); the record's README says so.
• docs/COMPARABLES.md, with pointers in docs/ARCHITECTURE.md and README.md, and app labels for the new producer slugs.

Invariants and their tests (tests/test_comparables.py, 148 tests)

• Completeness: every committed record passes; the committed index equals what the records build, twice; it is strict JSON.
• The gate, for any values of either sign (Hypothesis): a compared claim fails exactly when its gap exceeds the threshold or its sign disagrees, unless explained; an explained gap always passes.
• No evasion: 22 relabellings of a like-for-like claim with a 200% gap (null metric, uncompared with a reason, basis or programme relabelled, period moved, chart reading, NaN, unsigned or two-sided convention, cumulative, mismatch labels, placeholder bases, pending before publication, same-day or future re-harvest, open_search, scale word) each get a validator error.
• Search integrity, ranges, nearest estimates, pins, superseded history: one regression test per counterexample in the review.
• Gap identities (Hypothesis): zero for equal values; invariant to rescaling; opposite convention is negation; magnitudes ignore sign; a sign disagreement is over every allowed threshold; monotone in the threshold; a published zero.
• Harvest rule 1 (Hypothesis): every printed form parses back at its stated scale, and a perturbed value fails.
• The database: every analysis claim is held out; the claims ingested are exactly the records' claims the database can hold, each with its result and status.
• Differential: the app-v2 block and the index cite the same search; the index's comparisons equal evaluate; tests/fixtures/comparables_fresh_cases.json (96 cases) is regenerated from fresh here and checked by PolicyEngine/newsletters' own fresh.

Checks run

• Full pytest tests/ at 6c8eb0b (Python 3.12 for the Modal test, as CI): 2,451 passed, 15 skipped, then the three failures it found (the export pins, the app copies, and the Modal test under a non-3.12 interpreter) fixed and rerun: 240 passed in those files and the build chain.
• Two from-scratch builds agree on the content hash (452f64fb…); the tree is clean after the build.
• ruff format --check: clean.

Companion PRs (merge order)

1. This PR. app-v2 and newsletters read its index from main.
2. Require a comparables record on research entries after the cutover policyengine-app-v2#1380: the registry block and CI gate.
3. Refuse to send an edition whose analyses lack a fresh comparables search newsletters#29: the freshness check before a send.
4. Add the pre-publish comparables review skill and wire it into publishing policyengine-skills#100: the procedure (policyengine-comparables-review).

A weekly re-harvest job runs the search again for every analysis published in the last 183 days, using comparables due and report from this PR, once this merges.

Overlaps

• Add CenTax's costing of removing the £100,000 childcare limit #207 also edits models.py, build_db.py and sourceLabels.ts. The Centre for British Progress label is left to Add CenTax's costing of removing the £100,000 childcare limit #207. Its CenTax and CBP producer-level sources stay the canonical claims; this record re-stages them for the analysis. After Add CenTax's costing of removing the £100,000 childcare limit #207 merges, the childcare record's across_worlds basis moves from Add CenTax's costing of removing the £100,000 childcare limit #207's test file at 661c14d to main.

axiom: n/a: scorecard data and tooling; no PolicyEngine country-model change.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 22, 2026 5:48pm UTC