[MaxGhenis]

WIP: waits on PolicyEngine/policyengine-scorecard#208, whose index this gate reads.

The registry side of the pre-publish comparables gate Max asked for on 2026-10-21 ("how do we ensure we always do a full review of comparables to add to scorecard before publishing an analysis - pe-wide").

What it does

A research entry is an apps.json app with displayWithResearch or the policy tag, or any posts.json post (a post with no country tag is filed under us, as the site does). Every research entry needs a comparables block naming the record of its comparables search in policyengine-scorecard, unless it is on website/src/data/comparablesGrandfathered.json, the list of the 256 entries and 11 proxied pages live before the gate. The date an entry carries does not exempt it.

"comparables": {
  "status": "reviewed",
  "scorecardSource": "analysis-uk-some-dashboard",
  "searchDate": "2026-11-02",
  "searchKind": "prepublish",
  "producersSearched": ["academic", "centax", "..."],
  "found": 4,
  "searchLog": "https://github.com/PolicyEngine/policyengine-scorecard/blob/main/sources/analysis-uk-some-dashboard/search_log.json"
}

The scorecard prints the block: python -m scorecard_db.comparables record <source>. found counts the comparables staged as claims; the search log also lists any seen but not staged (a primary that cannot be read).

• The search is dated on or before the entry's date and no more than 30 days before it, and is not a backfill (prepublish, or reharvest/prepromotion for an analysis republished after a revision).
• no_estimate is only for a post that is plainly not an analysis: tagged org, event, giving, conferences, year-in-review or april-fools, and none of the analysis tags. An app never declares it.
• The grandfather list names only entries on the site, dated before the cutover (or undated apps). An entry taken down leaves it, so it comes back only with a record: restoring the childcare dashboard with its old date and no block fails. Entries may join the list only before the cutover; after it, the gate workflow refuses any addition relative to the base branch.
• On the day a change lists an entry. For an entry a change adds, re-dates or re-blocks, the gate workflow (which reads the base branch's registries) requires that the cited search is no more than 30 days old that day, whatever date the entry carries; that it compared the results files the analysis reads now (results_pins equal the record's current_pins); and that no gap it found is still pending. So a backdated draft, or a revised dashboard returning with a gap under investigation, is refused. Unchanged entries are not re-judged, so the archive does not expire.
• Featured research cards. A card a change adds to the homepage banner must link a research entry whose search is fresh that day under the scorecard's promotion window (the scorecard's fresh, tested against its shared cases), grandfathered entries included.
• Proxied pages. The tests read the evaluated next.config.ts rewrites and redirects (website/src/data/analysisRoutes.ts), so a route in any syntax, wildcard or single-quoted, is seen. Every page proxied to another deployment is a research app's (or served from one's deployment), on the grandfather list's routes, or on its new infrastructure list of pages that publish no analysis (API and model docs, slides, the Belgium demo).
• Route map for newsletters. website/src/data/analysisRoutes.json maps every path, redirect and deployment host that leads to a research app, built by scripts/build-analysis-routes.ts; a test fails while it is stale. PolicyEngine/newsletters reads it.
• The Comparables gate workflow runs on every pull request (no path filter, so it can be a required check), on pushes to main, and weekly. It downloads the index from a URL pinned in the workflow itself (a test checks comparables.ts agrees), fails rather than skips without an index (COMPARABLES_GATE=1), checks the index's schema and that its published entry rules equal comparables.ts's, validates every entry and resolves every block (the record exists with the same kind, slug and country, and a search of that date and kind has the same producers and count), and checks the grandfather list has not grown after the cutover. A failed weekly run opens or updates an issue.
• CODEOWNERS asks for Max's review on the gate's files, the lists, the route map and the tests that enforce them.
• Fails closed. A base commit the checkout cannot read fails the gate job.
• docs/engineering/skills/repository-guidance.md and the posts README state the rule. postSlug is now exported from postTransformers.ts and shared.

No entry carries a block yet. The first scorecard record is the £100,000 childcare dashboard (a backfill), which left apps.json on 21 October (#1379); it needs a new search against its revised results to come back.

Choices to check

• Cutover: 2026-10-29 (pinned no later than that by a test). Until then entries may join the grandfather list, so Budget-day (28 October) work is not blocked by a brand-new gate; from the 29th the list only shrinks. Queued for Max to confirm.
• Cross-repo resolution: the published index, read from the scorecard's main branch. The scorecard is public, so a raw fetch needs no token. A scorecard-side check cannot block a merge here; a vendored copy would go stale; pinning a scorecard commit would let a record on an unmerged branch pass. The index lists only records that passed the scorecard's own gate, so a match is the proof, provided the scorecard's main is protected.
• Search freshness: 30 days (MAX_SEARCH_AGE_DAYS, pinned at 30 or fewer).
• Not yet required by the rulesets (Max's call, queued): making Tests and Comparables gate required checks here, turning on code-owner review, and protecting the scorecard's main with a pull request and its pytest · ruff format check. Required checks also close the [skip ci] hole.
• GiveCalc. The GiveCalc app and post would share the record name analysis-us-givecalc. Both predate the gate; a test refuses any such clash for an entry that needs a record.

Proposed: "How we compare" on dashboards and posts (copy for Max)

Design only, nothing rendered yet. The scorecard index carries a comparisons list per analysis: each producer's figure as printed, with its period, our figure, the gap, the status and the cause.

• Where. On a dashboard, one line under the headline estimate and a section beside the methodology; on a research post, the same section at the end. Dashboards are zones, so the component belongs in @policyengine/ui-kit (<HowWeCompare source="analysis-uk-…" />), reading the index at build time so the figures are pinned to a scorecard commit.
• The line (proposed copy): "How we compare: CenTax estimates £0.80bn for 2027-28 (static); we estimate £0.43bn. Why they differ ↓"
• The section (proposed copy):
  • Heading: "How we compare"
  • Intro: "Other organisations have estimated the same things. We list every estimate we found and explain any difference larger than 25%."
  • A table: Source · Their estimate · Ours · Difference · Why they differ, the last column holding the annotation's cause with its link.
  • Footer: "We searched 19 producers on 21 October 2026 · Search log"
  • When the search staged nothing and saw nothing it could not stage: "We searched 19 producers on 21 October 2026 and found no other estimate of these figures." When it saw an estimate it could not stage, the section names it instead.

Max decides the copy, and whether it ships at all, before anything renders.

Checks

• bunx vitest run src/__tests__/content/: 41 passed; the live file (5 tests) passes with COMPARABLES_GATE=1 against the scorecard branch's index and main's registries as the base, and throws without an index path.
• tsc --noEmit, ESLint (--max-warnings 0) and Prettier: clean.

Properties tested (fast-check):

• An entry off the list with a well-formed block, whose search the index lists, always resolves, for listing dates 900 days either side of the cutover.
• Any one of 11 disagreements with the index fails resolution, the search log included.
• Any one of 21 malformations of a block fails validation.
• An entry off the list needs a block and a real listing date, whatever date it carries; one on the list never needs a block, but a block it carries must be well formed.
• Exactly the posts that are plainly not analyses may declare no estimate.
• researchEntries over generated raw apps and posts (timestamped dates, underscores and capitals in file names, zero to several country tags, with and without policy).
• A block matches either of two same-day searches; a record filed under another country fails.
• The grandfather list may grow only before the cutover.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-calculator-next	Ready	Preview	Oct 22, 2026 5:54pm UTC
policyengine-website	Ready	Preview	Oct 22, 2026 5:54pm UTC