Skip to content

Re-pin the US Chronicle feed so three dimension values carry one label each - #1345

Draft
anth-volk wants to merge 8 commits into
mainfrom
us-feed-agi-age-labels
Draft

anth-volk wants to merge 8 commits into
mainfrom
us-feed-agi-age-labels

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1343

Summary

The pinned US Chronicle feed labels three dimension values two ways, so the compiled US targets show one value id under two names. This PR re-pins the feed to a Chronicle commit that gives each value the one wording #1343 chose, and changes nothing else.

Dimension = value Before After Targets that change
us:statutes/26/62#adjusted_gross_income = under_1 "No adjusted gross income and deficit" (week-47 filing-season package); "No adjusted gross income" (seven other IRS SOI packages) "No adjusted gross income" 2
us:statutes/26/62#adjusted_gross_income = all "All returns" (Historic Table 2, Table 1.1); "All returns, total" (Tables 1.2, 1.4, 2.1, 4.3) "All returns, total" 47
age = 85_plus "Age 85 and over" (national Census PEP); "Age 85 plus" (state Census PEP, 51 targets) "Age 85 plus" 1

Update 2026-10-27: merged with main and moved onto #1189's feed commit

#1189 merged first and moved main's pin from 51aa40f to cf9ece2 (51aa40f plus the digit-text guard match and the BEA B4923C package; tag microcosm-us-feed-454-v1). This branch had pinned 9b91e8b (51aa40f plus the three label commits), so the two pins each lacked the other's changes, and seven files derived from the pin conflicted. This update follows the plan this description set out for that order:

  • Chronicle: a new branch, feed/us-cf9ece2-agi-age-labels, starts at cf9ece2 and carries the three label commits, cherry-picked in order. They applied without conflicts, and each patch is identical to its original. Its last commit, a9026cc, is tagged microcosm-us-feed-1343-v1.
  • Merge 11ba36758: merges main (7d45c0c33) with a merge commit. Each of the seven conflicts is resolved to a9026cc, not to either side's text: the scope file takes main's 640 pairs with the new source_commit, the feed is rebuilt, the parity files are regenerated, and the pins move (the commit message lists each file).
  • 529944125: reruns compare_feeds.py with main's cf9ece2 feed as the baseline and updates the changelog fragment.
  • 7fe24b972: fixes a test that Produce ESI employer premiums scaled to BEA B4923C from restored ASEC policyholder fields (#454) #1189 added and that the new pin broke in CI (run 40527755862, engine-free).
    • test_the_committed_feed_pin_banks_the_anchor read ESI employer premium contributions as a first-class, validated resource component #454's re-pin report and required its feed digest to equal the pin.
    • It now follows the committed reports of later re-pins that kept every row, which so far is only this PR's compare.json. Each report must start where the previous feed ended, keep every key, change only labels and record_set_spec_hash, and leave the three B4923C record sets alone. The last report must end at the pinned digest.
  • 8f1973735: a docstring in test_us_fiscal_targets.py names the feed this PR replaces (built at cf9ece2) instead of only 51aa40f. Docstring only.
    • Before the fix, 1 of the class's 3 tests failed; after it, all 148 tests in the file pass.
  • Same changes as before: the comparison with main's feed finds exactly the row and target changes the label commits made on 51aa40f: 816 rows and 50 targets, labels only.
  • The old branch feed/us-51aa40f-agi-age-labels (9b91e8b) is no longer used.

The Chronicle commit

a9026cc8f80307af0c4527fdb3779fab32ffad9f on feed/us-cf9ece2-agi-age-labels, tag microcosm-us-feed-1343-v1: main's current pin cf9ece2 plus three label-only commits, not Chronicle main.

  1. 2431ba8 (from edb7cf1): the week-47 YAML change of Give each dimension value one Chronicle-declared label, keep the package's own wording beside it (consumer_fact.v5) chronicle#377 (6d74e42), and nothing else from that PR (not its build-bundle check).
  2. d1d28dc (from 59d9d06): soi-historic-table-2 and soi-table-1-1 relabel their all rows.
    • Table 1.1's row had no expected_row_header, so its row-header check compared the label itself with the IRS text in cell A10. The row now states expected_row_header: All returns, so the check still reads the same cell for the same text.
    • Matching test changes: two tests that pin Table 1.1's record_set_spec_hash, the checked-in consumer fact sample built from Table 1.1's first rows, and that sample's pinned digest. cf9ece2 changes none of these, and they pass unchanged at the new base.
  3. a9026cc (from 9b91e8b): census-pep-2024-national-age-sex relabels its 85_plus row.

Every package's checks of the publisher's text are unchanged: the week-47 column header, Historic Table 2's AGI_STUB 0, Table 1.1's row header, and Census PEP's row header 85 and single-year age sequence. soi-historic-table-2-state-broad-2022 keeps "All returns" for state = all (chronicle#267). That is the state dimension, which no other feed package labels, and its 2,397 targets carry one label.

Checks at a9026cc:

  • chronicle validate-package: valid, no warnings, for each changed package at each scope year (week 47 at 2024, Historic Table 2 at 2022, Table 1.1 at 2022 and 2023, national PEP at 2024), run one at a time.
  • chronicle build-suite: valid for the same five runs, with lineage coverage 1.0 and 0 acceptance errors. The only warnings come from running without Axiom: no_concept_alignments (week 47, national PEP) and concept_alignment_validation_skipped (Historic Table 2, Table 1.1).
  • Chronicle tests covering these packages: 373 passed in 18 files, with a peak of 745 MB:
    • 264 in the 10 files of the earlier run (test_chronicle_soi_targets, _source_cells, _source_package, _suite, _harness, _consumer_contract, _core, _facts_only, test_issue_265_non_uk_labels, test_etl_soi);
    • 109 in 8 more files that name these packages (test_chronicle_artifacts, _mirror, _database, _pe_source_plan, _soi_state_agi_2023, test_policyengine_chronicle_imports, _schema, test_us_poverty_target_coverage).
  • tests/test_chronicle_bundle.py, which builds the merged bundle, was not run locally; see "Unverified".

What changes in this PR (against main)

  • us/chronicle_feed.json: source_commit, build text, facts_sha256 and scope_sha256. fact_row_count stays 40,181.
  • us/chronicle_feed_scope.json: only source_commit; the 640 pairs are main's.
  • target_parity_feed_families.json and target_parity_manifest.json, regenerated with tools/build_us_target_parity_manifest.py:
    • the feed name and feed_sha256 change;
    • the feed-family inventory gains dimension_value_label_variants (only age_20 and age_21 on this feed; see "Review" below);
    • still 32 compiled families, 53 reviewed exclusions and 82 feed families.
  • tools/build_us_target_parity_manifest.py: DEFAULT_FEED_NAME, and the listing of label variants.
  • Tests:
    • _CHRONICLE_COMMIT and _FACTS_SHA256 in test_us_chronicle_feed.py, plus its test that the committed inventory lists only the reviewed label variants;
    • in test_us_fiscal_targets.py, the national_state surface version in test_pinned_feed_national_state_surface_restores_the_fences (the version hashes the hierarchy labels), and test_pinned_feed_targets_give_each_reviewed_value_one_label;
    • a unit test of the label-variant listing in test_release_target_parity.py;
    • test_us_esi_premiums.py: the B4923C anchor check follows the label-only re-pin's report to the pin (see the update above).
  • docs/us-chronicle-feed-repin.md: a new section above ESI employer premium contributions as a first-class, validated resource component #454's, and the pin table. changelog.d/1343-us-feed-one-label-per-value.fixed.md.
  • experiments/us-feed-agi-age-labels/compare_feeds.py and its report compare.json.

Digests

Before (main) After
Chronicle commit cf9ece2e7be11158c21fdc4dd38b44d92db552f6 a9026cc8f80307af0c4527fdb3779fab32ffad9f
Feed file consumer_facts_us_cf9ece2.jsonl, 40,181 rows, 169,055,585 bytes consumer_facts_us_a9026cc.jsonl, 40,181 rows, 169,056,411 bytes
facts_sha256 054e5684005a7b143dcb84eabac634135ce9b2b33bb57b540602e474a4ba7c57 4d89e4a498ffd73a407821d9ce2c45837072901e0a29acf3206160d1fafd7c14
scope_sha256 f4a218fc3be534f3e447f25bf1ec4064d9f3db6b72a55874ed5db4823c922239 04460f39e5c2b79011724ce33cac4ccf3b0f42812bc7179b6b532773483fe04e
Registry version (all 33,250 targets) dde00e855af8 b8d571e91c22
national_state surface (6,102 targets) 3fad14f85468 3aa2a0830e7d

Reproducibility: tools/build_us_chronicle_feed.py --skip-artifact at cf9ece2, with main's scope file, reproduced main's 054e5684… byte for byte, so the comparison below starts from main's pinned feed.

Only labels change

uv run python experiments/us-feed-agi-age-labels/compare_feeds.py <cf9ece2 feed> <a9026cc feed> compare.json, compiled at 2024 as #940 did (packaged CD crosswalk, age targets, Medicaid substitutions):

Feed rows, joined on aggregate_fact_key:

  • The same 40,181 keys are in both feeds, including the three B4923C facts. 816 rows change, and no field other than the ones below changes on any row.

    Field Rows Record sets
    dimension_value_labels[…adjusted_gross_income][under_1] 2 week-47 eitc_by_agi.count and .amount
    dimension_value_labels[…adjusted_gross_income][all] 63 ty2022.historic_table_2.us (55), ty2022.table_1_1 (4), ty2023.table_1_1 (4)
    dimension_value_labels[age][85_plus] 1 census_pep.cy2024.national_resident_population_age
    layout.groupby_value_label 66 the same rows
    layout.record_set_spec_hash (covers row labels) 816 every row of those six record sets: 605 + 80 + 80 + 19 + 16 + 16
  • Label variants in the feed. Before: the three values, plus age = age_20 and age_21 ("Age 20" against "Female 20 years", and the same for 21, from Census population projections and ACS B01001). After: only age_20 and age_21, which no compiled target selects.

Compiled targets, compared field by field on TargetSpec.to_dict():

  • Both feeds give 33,250 targets, with the same names in the same order and the same values. 50 targets change, in exactly two fields each:

    • hierarchy.dimensions[…].value_label;
    • metadata["ledger_layout_groupby_value_label"].
    Old label New label Targets
    "No adjusted gross income and deficit" "No adjusted gross income" 2
    "All returns" "All returns, total" 47
    "Age 85 and over" "Age 85 plus" 1
  • Label variants in the compiled targets. Before: those three values. After: none, so no dimension value in the compiled targets has two labels.

  • The registry and surface versions move as listed above. The surface keeps its 6,102 targets.

compare.json lists every changed feed label, every record-set hash change and all 100 changed target fields. Against the 21 October comparison (51aa40f to 9b91e8b), only the feed names, digests and row counts in the report differ.

Tests

2026-10-27, with MICROCOSM_US_CHRONICLE_FACTS and the default feed path set to the a9026cc feed (at 529944125):

  • three of the four test files this PR changes (test_release_target_parity.py, test_us_chronicle_feed.py, test_us_fiscal_targets.py): 239 passed, 0 skipped;
  • the pinned-feed tests in test_us_acs_local_release_tool, _acs_local_sparse_cd, _fiscal_refresh_builder (including Produce ESI employer premiums scaled to BEA B4923C from restored ASEC policyholder fields (#454) #1189's B4923C anchor checks), _release_gate_preflight, _soi_congressional_district_vintage and _soi_state_agi_bands: 19 passed, 0 skipped;
  • ruff check . passes, and tools/ci_test_plan.py verify reports ok.

Without the feed, as in CI (at 7fe24b972):

  • test_us_esi_premiums.py, the fourth test file this PR changes (in 7fe24b972), which needs no feed: 148 passed;
  • the other US test files that read the pin, its scope or the parity files (test_us_medicare_irmaa_holdout, _fiscal_refresh_builder, _release_gate_preflight, _soi_congressional_district_vintage, _soi_state_agi_bands, _spec_bundle, _acs_local_release_tool, _acs_local_sparse_cd): 637 passed, 15 skipped. The skipped tests are the ones that need the feed.

CI: all ten jobs pass on 7fe24b972 (run 40532463928): lint, wheels, select-countries, engine-free, engine-us and engine-uk on Python 3.13 and 3.14, and integration-uk. Run 40527755862 on 529944125 failed in engine-free on the B4923C anchor test, which 7fe24b972 fixes.

Red, then green (2026-10-21, on the 51aa40f base):

  • Red 5769515: tests only. test_pinned_feed_targets_give_each_reviewed_value_one_label compiles the pinned feed as the release does, and requires every target that selects each value to carry the chosen label. On the 51aa40f feed all three cases fail: 2 targets read "No adjusted gross income and deficit", 47 read "All returns", and 1 reads "Age 85 and over".
  • Green 09b950e: the re-pin to 9b91e8b.

Like the other pinned-feed tests, that test skips where the feed file is absent, as in CI. The committed-inventory test below runs in CI without the feed.

Interplay with #1192

#1192 also re-pins the feed, from the pre-#940 base c5e5bf8 with the W-2 package (f98acf4), and conflicts with main. If this PR merges first, #1192 should:

  • base its W-2 commit on a9026cc;
  • rebuild the feed and compare it with 4d89e4a4…;
  • regenerate the parity files. The committed inventory test fails if its feed brings back a second label for under_1, all or 85_plus;
  • explain any difference beyond its own rows.

Unverified

  • Chronicle CI on the feed branch. The push started Chronicle's CI on a9026cc (run 40523279822), because Chronicle's ci.yml runs on every push.
    • It failed only in test_build_bundle_writes_merged_consumer_contract (1 failed, 1,209 passed). That test pins the merged bundle's row counts to an older bundle, and it failed the same way at cf9ece2 (run 38173127570), with the same counts: 340,969 against the pinned 339,948.
    • The count assertion fails first, so the test's later check of the cross-package label warnings never runs. CI has therefore not checked those warnings at a9026cc.
    • The test peaks near 15 GB, so it was not run locally.
  • Chronicle main (272c987, 25 October) still has all three old labels in these packages. Give each dimension value one Chronicle-declared label, keep the package's own wording beside it (consumer_fact.v5) chronicle#377, still open, sets the same three wordings in its label register (chronicle/dimension_values/canonical_labels.json at ab38667). Until it merges, a re-pin onto main (Read restamped Chronicle facts at their data year before aging #1030) must carry these labels or lose them.

Review (2026-10-21, of the 51aa40f-based version)

No bug found in this PR or in the Chronicle branch feed/us-51aa40f-agi-age-labels:

  • An independent rebuild with tools/build_us_chronicle_feed.py reproduced that version's facts_sha256 e477e04a… and scope_sha256 8fedefb7… byte for byte.
  • compare_feeds.py confirmed that only labels change: 50 national targets change only value_label and ledger_layout_groupby_value_label.
  • Regenerated parity files matched. Table 1.1's new expected_row_header checks the same IRS cell, A10.
  • Consumers:
    • loss weights use the label only for congressional-district rows;
    • the target-frame cache rebuilds once, because the registry version is in its key;
    • the dashboard matches targets by name and value_id;
    • .py doesn't read these fields.

Added by the review: a check CI runs without the feed file. The pinned-feed label test skips in CI.

  • The parity generator records, in target_parity_feed_families.json, every dimension value the feed labels more than one way. On this feed that's only age_20 and age_21.
  • A test reads that committed file, so it runs in CI. It fails if under_1, all or 85_plus gain a second label again, or if any other value gains one without review.
  • Why it can't be skipped: a re-pin must regenerate that file, because CI already requires it to restate the pinned feed's digest.
Commit Content Result
388699c9 Tests only 2 fail (KeyError, AttributeError)
2880b263 The generator change and the regenerated file 247 pass, 0 skipped, with the feed; without it, the 2 new tests pass and the feed tests skip; ruff passes

🤖 Generated with Claude Code

anth-volk and others added 8 commits October 22, 2026 00:00
…l each (#1343)

Tests only. On the pinned feed (Chronicle 51aa40f, facts 965a29ac...), the
compiled 2024 targets label three values two ways, and this test fails for
each:

- us:statutes/26/62#adjusted_gross_income = under_1: 2 targets (the week-47
  filing-season EITC rows) read "No adjusted gross income and deficit";
- us:statutes/26/62#adjusted_gross_income = all: 47 targets (Historic
  Table 2 and Table 1.1) read "All returns";
- age = 85_plus: 1 target (national Census PEP) reads "Age 85 and over".

Like the other pinned-feed tests, it skips where the feed file is absent,
as in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l each (#1343)

The pin moves from Chronicle 51aa40f to 9b91e8b: 51aa40f plus three
label-only commits (branch feed/us-51aa40f-agi-age-labels), not Chronicle
main.

- AGI under_1: the week-47 package's "No adjusted gross income and deficit"
  becomes "No adjusted gross income" (chronicle#377's 6d74e42, without its
  build-bundle check).
- AGI all: Historic Table 2 and Table 1.1's "All returns" becomes "All
  returns, total". Table 1.1 now states its IRS row header explicitly, so
  its row-header check still reads "All returns" in cell A10.
- age 85_plus: national Census PEP's "Age 85 and over" becomes "Age 85 plus".

Measured (experiments/us-feed-agi-age-labels/compare.json): the feed keeps
its 40,178 aggregate_fact_keys and every value, key and constraint; 816
rows in six record sets change only in label fields (66 rows) and
layout.record_set_spec_hash. Compiled at 2024 as the release does, the
33,250 targets keep their names, order and values; 50 change only in the
hierarchy value_label and metadata ledger_layout_groupby_value_label (2
under_1, 47 all, 1 85_plus). Registry dde00e855af8 -> b8d571e91c22;
national_state 6,102 targets, 3fad14f85468 -> 3aa2a0830e7d.

Pin: facts 965a29ac... -> e477e04a..., scope f0901020... -> 8fedefb7...
(only source_commit), 40,178 rows. Parity resources regenerated: only the
feed name (consumer_facts_us_9b91e8b.jsonl) and digest change, 32 compiled /
52 reviewed. Test pins, the re-pin doc and a changelog fragment updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#1343)

Tests only. The label test added for #1343 compiles the pinned feed, so it
skips in CI, and a later re-pin built on a Chronicle commit without the
three label commits would bring the old labels back unnoticed. A re-pin
must regenerate target_parity_feed_families.json from its feed (CI checks
that the inventory restates the pinned digest), so the inventory is where
CI can see the labels:

- test_feed_inventory_lists_only_the_reviewed_label_variants requires the
  committed inventory to list, as dimension_value_label_variants, exactly
  the two values the feed still labels two ways (age_20 and age_21), and
  none of adjusted gross income under_1 or all or age 85_plus;
- TestFeedLabelVariants checks the generator's listing on synthetic rows,
  reading both dimension_value_labels and layout.groupby_value_label.

Both fail now: the inventory has no dimension_value_label_variants
(KeyError) and the generator has no _dimension_value_label_variants
(AttributeError).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/build_us_target_parity_manifest.py now writes, into
target_parity_feed_families.json, dimension_value_label_variants: every
(dimension, value id) the feed labels more than one way, with its labels,
read from dimension_value_labels and layout.groupby_value_label as the
ledger compiler reads them. Regenerated from consumer_facts_us_9b91e8b.jsonl
(sha256 e477e04a...), it lists only age = age_20 and age_21; the manifest
and the family counts are unchanged.

A re-pin must regenerate the inventory from its feed, because CI checks that
the inventory restates the pinned digest. So a re-pin built on a Chronicle
commit without the three #1343 label commits now lists those values again
and fails test_feed_inventory_lists_only_the_reviewed_label_variants in CI,
where the compiled-target label test skips for want of the feed. The re-pin
doc says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onicle a9026cc

main's microcosm#1189 moved the US feed pin from 51aa40f to cf9ece2 (51aa40f
plus the digit-text guard match and the BEA B4923C package), while this branch
pinned 9b91e8b (51aa40f plus the three #1343 label commits). The seven files
derived from the pin conflicted, and neither side's text is right: the feed
has to come from a Chronicle commit that holds both changes.

That commit is a9026cc (Chronicle branch feed/us-cf9ece2-agi-age-labels, tag
microcosm-us-feed-1343-v1): cf9ece2 with the three label commits
cherry-picked, without conflicts. Each conflict is resolved to it:

- chronicle_feed_scope.json: main's 640 pairs, source_commit a9026cc.
- chronicle_feed.json: source_commit, build text, facts_sha256 4d89e4a4...
  and scope_sha256 04460f39...; fact_row_count stays 40,181.
- build_us_target_parity_manifest.py: DEFAULT_FEED_NAME
  consumer_facts_us_a9026cc.jsonl, keeping this branch's label-variant
  listing.
- target_parity_manifest.json and target_parity_feed_families.json:
  regenerated from the a9026cc feed. Only the feed name and digest change
  against main, plus this branch's dimension_value_label_variants (age_20
  and age_21 only); still 32 compiled families, 53 reviewed exclusions and
  82 feed families.
- test_us_chronicle_feed.py: _CHRONICLE_COMMIT and _FACTS_SHA256.
- docs/us-chronicle-feed-repin.md: the #1343 section rewritten for the
  cf9ece2 base, above main's #454 section, and the pin table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-run experiments/us-feed-agi-age-labels/compare_feeds.py with main's
pinned feed as the baseline: consumer_facts_us_cf9ece2.jsonl (rebuilt at
cf9ece2 with main's scope file; it reproduced 054e5684... byte for byte)
against consumer_facts_us_a9026cc.jsonl (4d89e4a4...).

The changes are the ones the label commits made on the 51aa40f feed: the
same 40,181 keys, 816 rows changed only in labels and record_set_spec_hash
(66 label rows), and of the 33,250 compiled targets, 50 change only their
value label (2 under_1, 47 all, 1 85_plus). The registry version goes from
dde00e855af8 to b8d571e91c22 and the national_state surface (6,102
targets) from 3fad14f85468 to 3aa2a0830e7d. Only the report's feed names,
digests and row counts change.

The changelog fragment names the new pin and its base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_the_committed_feed_pin_banks_the_anchor (#1189) read the #454 re-pin
report and required its new feed digest to equal the pin. After the merge
the pin is a9026cc's feed, so the check failed in CI (run 40527755862,
engine-free) although the feed still carries the three B4923C facts.

The check now follows a list of committed reports of later re-pins that
kept every row, starting from the #454 feed: each must start where the
previous feed ended, keep every key and the row count, change only label
fields and record_set_spec_hash, and leave the three B4923C record sets
alone; the last must end at the pinned digest. The only such report is
#1343's compare.json (cf9ece2 feed to a9026cc feed). A later re-pin that
adds or changes rows still fails here until it brings its own check.

Before: 1 failed, 2 passed in TestAnchorFeedCheck. After: the whole file,
148 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1343)

The docstring still called the old feed 51aa40f; since main's pin moved, the
feed this PR replaces was built at Chronicle cf9ece2. Docstring only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

US targets label three dimension values two ways: re-pin the Chronicle feed with one label each

1 participant