Skip to content

Add the pre-publish comparables review skill and wire it into publishing - #100

Draft
MaxGhenis wants to merge 6 commits into
mainfrom
feat/comparables-review
Draft

MaxGhenis wants to merge 6 commits into
mainfrom
feat/comparables-review

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

The operating procedure for PolicyEngine's pre-publish comparables gate: before an analysis is listed on policyengine.org, or featured in a newsletter, social campaign, webinar or press push, every external estimate of what it estimates is found, recorded in PolicyEngine/policyengine-scorecard, compared with the analysis's own figures, and every gap over the threshold is explained.

What it adds

  • skills/policyengine-comparables-review/: the review, step by step.
    • Scaffold a record, pin every figure the page publishes (with its identity: basis, programme, fiscal year), search every producer on the scorecard's data/comparables/producers.json, stage what is found, compare like with like or set a claim beside its nearest estimate with a stated mismatch, and explain every gap over the threshold with a cited cause.
    • The fields a record needs, with the childcare record as the worked example, and which harvest rules apply.
    • The evasions never to use (leaving a figure out, a null metric, a chart reading for a printed figure, choosing a period to avoid a match, relabelling a miss as a mismatch) and a reviewer checklist for what the validator cannot see (every published figure pinned, pe_missing true, bases support causes, none_found spot-checked, hashes recomputed).
    • US guidance: sign phrasings that orient, fiscal versus calendar years, budget-window totals, state fiscal notes.
    • The household-calculator path (searched, no fixed estimate), "Before a push" (a pending gap blocks it, newsletters and the homepage banner alike), the after-publication re-harvest, and bill-tracker scores (reviewed before dispatch, recorded as registry.kind: tracker).
    • What app-v2 checks on the day a change lists an entry (search age that day, current results, no gap pending), the proxied-page rule, deployments, record's latest-search default, report running while gaps are unexplained, and that window totals are staged, not compared.
    • Commands as CI runs them (uv run, CI's test dependencies), including committing every feed the build rewrites.
  • references/search-brief.md: a self-contained brief for running the search as a background job, for any agent with web search, a writable workspace and gh push access (PolicyEngine's internal Subfleet lanes need -s workspace-write). It covers all four search kinds, pushes a draft PR even when a gap is unexplained, and writes a pending cause that cites its tracking issue.
  • /comparables-review: the Claude command.
  • Wiring: /deploy-dashboard runs the review before anything is public, wires the zone through appZoneRoutes.ts and uses a registry template app-v2 accepts; /new-tool keeps the repository private and undeployed until the review merges; /create-dashboard's next steps, report-logger's draft research posts, the policyengine-writing checklist, policyengine-tools, policyengine-content, /generate-content and the content orchestrator (freshness before a push), /reform-pipeline (both routes reviewed, the tracker route before dispatch), policyengine-prior-scores and presets/scorekeepers.yaml (the scorecard list is the one this review must cover), docs/when-to-use.md and the skills README.
  • Bundles: complete, analysis-tools, dashboard-builder and app-development carry the skill and command. Version 4.0.2 → 4.1.0.
  • dashboard/src/data/manifest.json is regenerated so the guide's command check sees /comparables-review. scripts/build_manifest.py orders some lists differently on each run, so most of that file's diff is reordering; main's copy was already out of date.

Companion PRs and merge order

  1. Record every comparable external estimate before an analysis is published policyengine-scorecard#208: the producer list, validator, threshold, index, first record and the commands this skill runs.
  2. Require a comparables record on research entries after the cutover policyengine-app-v2#1380: the registry block and CI gate.
  3. Refuse to send an edition whose analyses lack a fresh comparables search newsletters#29: the freshness check before a send.
  4. This PR, last, so every check it describes is on main when it lands.

Checks

  • uv run pytest --ignore=tests/test_skill_examples.py: 178 passed.
  • scripts/build_claude_wrapper.py: builds; the skill, references/search-brief.md and commands/comparables-review.md are present.
  • Description 945 characters.

After merge, policyengine-app-v2's ai-agents stats need refreshing (24 skills, 36 agents, 21 commands, 8 bundles): PolicyEngine/policyengine-app-v2#1397.

🤖 Generated with Claude Code

policyengine-comparables-review is the operating procedure for the gate
policyengine-app-v2 CI enforces: search every producer on the scorecard's
list, stage what is found under the harvest rules, compare with the
analysis's own figures, and explain every gap over the threshold. Ships a
Subfleet research brief and /comparables-review, and makes the review a
required step in /deploy-dashboard, /new-tool, /create-dashboard,
report-logger drafts and the writing checklist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 5 commits October 21, 2026 11:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-21: "even if we've already published something, we should do a
fresh comp review before big pushes like newsletters". Adds the Before a push
section and the prepromotion search kind to the comparables review, and makes
the freshness check a required first step of /generate-content and the
content orchestrator, and of the writing checklist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The background job gets a writable workspace and gh push access, and the
brief covers every search kind, pushes a draft PR even with an unexplained
gap, and writes a pending cause that cites its issue. The skill lists the
record's fields, the evasions never to use, a reviewer checklist, US sign
conventions and fiscal years, the calculator path, and the commands as CI
runs them. A pending gap blocks a push. Dashboards are reviewed before they
are public. The brief is renamed search-brief.md, and the catalog manifest
is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
basis_note and program_note, the price base, unstated basis or prices,
percent figures, range dashes, one-sided bounds, and hashing the documents
annotations cite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/new-tool and /deploy-dashboard review before anything is public, wire the
zone through appZoneRoutes.ts, and use a registry template app-v2 accepts.
The tracker route is reviewed too, as a tracker record. The skill states the
listing-day, banner and proxied-page rules, the deployments field, record's
latest-search default, report's tolerance of unexplained gaps, the Python
3.12 suite, and that window totals are not compared yet. The brief repins
after superseding estimates. The catalog manifest is rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant