Skip to content

Show agreement with TAXSIM release by release on the validation dashboard - #1418

Draft
MaxGhenis wants to merge 7 commits into
mainfrom
alignment-history
Draft

Show agreement with TAXSIM release by release on the validation dashboard#1418
MaxGhenis wants to merge 7 commits into
mainfrom
alignment-history

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Why

At the BLS Consumer Expenditure seminar on 2026-10-08, BLS asked whether we share how each release changes results. We said the validation dashboard would show alignment with TAXSIM by version. This PR adds that view, backfills it, and keeps it current.

What it shows

A new Agreement by release section on /us/taxsim/dashboard, under the headline metrics. It follows the page's year, state and tolerance controls:

  • a line chart of federal and state agreement for each released policyengine-taxsim / policyengine-us pair, with a crosshair tooltip (keyboard: arrow keys);
  • a table, newest first, with the change from the previous pair;
  • the releases that could not be scored, each with its reason.

Version labels match policyengine-taxsim --version and the provenance keys in #1412 / #1414 (policyengineTaxsimVersion, policyengineUsVersion, policyengineCoreVersion, taxsimBinaryBuild, taxsimBinarySha256).

How a row is measured

Details are in docs/alignment-history.md.

  • TAXSIM side, fixed: the TAXSIM rows of the published full-data release the dashboard links to (full-ecps-comparison-run-35953332152). They come from build cd2026090910, except GA and MD in 2024–2025, which use build cd2026081819. Every row is scored against the same outputs, so a change between rows comes from the emulator or the model.
  • PolicyEngine side: the pair installed in a fresh venv as PyPI stood just after its release (uv pip install --exclude-newer), with the refresh's options. Options a release lacks are recorded.
  • Agreement: refresh_dashboard.match_flags(), the same function as the summaries. The file stores counts only, overall and per state. The dashboard computes the rates.

Which pairs

  • Each emulator release, with the policyengine-us that was newest when it came out.
  • One policyengine-us release per day (the last one PyPI serves), with the emulator release that was newest then.
  • Pairs PyPI can't install are recorded as not-installable with the reason, never scored with a substitute. Of 112 emulator releases, 83 can't be installed: PyPI deleted the release, its policyengine-us counterpart, or both.
  • The planner reads each missing policyengine-us version's Publish job. A failed, skipped or cancelled publish means nobody could have installed that version. For example, taxsim 2.31.3 pairs with policyengine-us 2.6.17, because 2.6.18 to 2.6.20 never published.

Wiring

.github/workflows/alignment-history.yml runs:

  • after CI on main (which publishes each release);
  • daily, for policyengine-us releases;
  • on demand: release for one emulator release, backfill for all of them.

Its jobs are plan → reference → one measure job per pair → record. Record commits the file to main, after waiting until no versioning job is running, because that job pushes without pulling.

ci.yml and the release process are unchanged.

Invariants (all tested)

  • Every count is in [0, households]; every rate is in [0, 100%].
  • State households sum to the total, and state counts sum to each total count.
  • No state count exceeds that state's households.
  • Every measured year covers the reference's full population.
  • score() doesn't depend on household order.
  • Differential:
    • score() and refresh_dashboard.summarize() tally identically (Hypothesis).
    • Re-scoring the published release reproduces every committed summary_{year}.json exactly (run in the reference job; verified locally for 2021–2025).
    • The dashboard's JS rate derivation reproduces those summaries from the recorded counts (fast-check suite).
    • On PRs, the history worker reproduces the refresh's PolicyEngine outputs household by household.
  • Timing: a release's publish window always contains its true time, and when the planner says it is certain which policyengine-us was newest, it is right for every consistent history (Hypothesis).
  • Merge: idempotent, order-independent, and it refuses smoke rows and changed references.

Status

Draft. The backfill is running from a scratch branch. Numbers will be added here only from computed rows.

axiom: n/a: dashboard and validation tooling; no policy logic.

🤖 Generated with Claude Code

@MaxGhenis @claude
Add release-by-release agreement with TAXSIM to the validation dashboard
d1033ee 
Scores each released policyengine-taxsim / policyengine-us pair against the
TAXSIM side of the published full-data release, with the dashboard's own
households, tolerances and match_flags(). Pairs PyPI can no longer install are
recorded with the reason rather than scored with a substitute.

- scripts/alignment_history.py: plan, reference, measure, merge
- scripts/alignment_worker.py: runs one batch inside the pair's own venv
- .github/workflows/alignment-history.yml: after CI on main, daily, on demand
- dashboard: Agreement by release section (chart, table, unscored releases)
- property, differential and invariant tests (Hypothesis, fast-check)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policyengine-taxsim Ready Ready Preview Oct 10, 2026 12:50am UTC

Request Review

@MaxGhenis @claude
Test the Agreement by release view; pluralize the unscored count
ee41488 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Prefer the build stamp refresh provenance records; list the view in t…
3523fb9 
…he README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis mentioned this pull request Oct 9, 2026
Draft
@MaxGhenis @claude
Run one worker per batch and year; don't record infrastructure failures
573c2f3 
The first full run (37981903700) showed a worker that ran all five years in
turn passing 6 GiB. One process per batch and year matches the dashboard
refresh, whose workers peaked at 3.2 GiB.

A worker stopped for memory or time, or one that crashes, now fails the job
and writes no row, so the pair is measured again later. Only a year the
release itself cannot compute is recorded as failed. Failed and
install-failed pairs are planned again while in scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Check for a running release right before pushing rows; split row and …
94984ce 
…output artifacts

- record: the versioning check now runs immediately before each push, inside
  the retry loop, so the window in which a push could land between a release's
  checkout and its push is a second or two, not the length of the merge.
- measure: row.json is its own small artifact, so record no longer downloads
  every pair's per-household outputs.
- chart: label the first and last day when the releases span under two months.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Address review: evidence-based pairing, release-safe recording, stric…
e49ec85 
…ter checks

Pairing
- A release is available from its first uploaded file, which is what
  uv --exclude-newer sees; the environment is resolved once both are complete.
- A version-bump commit is only a lower bound on publication. A release PyPI
  lacks is placed from its publish job: published in that job's window, or
  never published. Evidence is read newest candidate first, and only as far
  as needed (96 lookups for the full backfill plan).
- All modes use the same rule, so a release deleted after publishing is no
  longer replaced by an older one. A pair that can't be settled is reported
  as undetermined and not recorded, so better evidence can settle it later.
- A release published within the last day but missing from PyPI's index is
  left undetermined rather than called deleted.

Recording
- ci.yml: the versioning job fast-forwards over commits that touch only
  alignment_history.json, so a row recorded during CI can't fail that run's
  release. The record job still refuses to push while a versioning job runs,
  and now treats a GitHub API error as running.
- merge replaces a recorded row only with one that measured more years, adds
  new triggers to a kept row, and drops a not-installable row once its
  releases are re-paired. It copies its inputs.
- Only planned, full-population rows for years the reference has are valid.

Measuring
- MemoryError and OSError crash the worker (retried, never recorded as the
  release's failure). Every worker's process group is cleared on exit.
- measure verifies the source and reference hashes before scoring.
- reference checks every TAXSIM input and the tax year against the source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis @claude
Rewrite the method doc to the final rules; keep the reference selecto…
8aaa947 
…r; thin date labels

- docs/alignment-history.md: pairing evidence, the two kinds of failure, how
  recording stays clear of releases, and measured figures (34 minutes and
  4,480 MiB per worker in run 38006256962).
- dashboard: the TAXSIM reference selector stays when the chosen reference has
  nothing scored; month labels thin to the chart's width.
- tests: crashed workers leave no children; the total-memory requeue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 8aaa947d Deployed Oct 10, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MaxGhenis