[MaxGhenis]

At a fixed budget of source households, which mix of CPS ASEC years, ACS households and CPS location clones gives the national compact file and the CD and county local file the best held-out accuracy? We calibrated 30 arms for each product and scored all 60 calibrations on held-out targets. The memo, docs/us-support-mix-bakeoff.md, gives the design, verdict and caveats, and experiments/us-support-mix-bakeoff-20260930/ holds the receipts.

Merging adds the results and the tooling and changes no build default. A PR that adopts a mix changes how we build the dataset, so it needs its own from-scratch build and certification receipt under the US release build rule. Max has chosen one US pool that includes ACS rows, and the national file takes the national mix below from that pool (d2283). The local mix awaits his call (d2311).

Verdict

We rank arms by the release's own objective on held-out targets: the loss-weighted capped relative error over every held-out CPS-native target (income components, tax items, program receipt) at a level.

For the national compact file, we recommend the newest three CPS years plus ACS to 300,000 source households. With 2022–2024 that comes to 484,080 physical rows, 37% more than the three-year CPS file. Against the CPS-only file, as means over three ACS draws:

• held-out state CPS-native error falls from 0.119 to 0.113, and district error from 0.353 to 0.291;
• state income components improve (0.250 to 0.224), state tax items worsen (0.103 to 0.113), and state program receipt does not measurably change (0.208 against 0.198, draw SD 0.014);
• every ACS-native level improves (district 0.236 to 0.188, county 0.307 to 0.259).

Larger ACS fills keep lowering district error (0.266 at 600k, 0.244 at 1.2M), edge state tax items up and do not help state program receipt. The 600k 50/50 fill scores best at state level (0.111) with 2.1 times the rows, but we stop at 300k to hold down build and serving cost.

For the CD and county local file, we recommend the newest three CPS years plus ACS households to 1.2M source households, with no CPS location clones. That comes to 1,384,080 physical rows, 13% fewer than September's 1,588,854-row ACS local file. At 1.2M:

• held-out district CPS-native error ties the clones (0.169 against 0.170) with 55% of their rows, and state error falls below theirs (0.112 against 0.114);
• ACS-native error runs 44% below the clones' in districts and 31% below in counties;
• the clones and the CPS-only file beat the recommendation on state program receipt (0.218 and 0.220 against 0.256) and state tax items (0.096 and 0.101 against 0.117).

Calibration worsens held-out Census PEP state age cells in every arm: across the 60 calibrations, their mean capped error rises from 0.066 at the starting weights to 0.168. We report the age column, rank no arm on it, and leave finding the cause to a follow-up. We report SPM poverty too, and no arm calibrates to it or gates on it.

We ran no five-year arm and no arm on the 2023–2025 CPS years. Income years 2020 and 2021 pass the response-quality check but lack Route A enrichment, and we pinned the 2025 ASEC on main (#1044) after we built the Route A export these CPS rows come from. The memo's follow-ups cover both.

What this adds

• tools/bakeoff_us_support_mix.py runs the bake-off through compile, shard, truth, materialize, diffcheck, partcheck, run-grid and report. arm solves a single arm, spm-flags writes the SPM flag receipt and grid lists the arms. Each run step writes a receipt with wall time and peak RSS, and each arm receipt records digests of the inputs its arm reads.
• packages/microcosm-build/src/microcosm/build/us_runtime/support_mix.py holds pure primitives: arm accounting, the nested ACS order, clone allocation, starting weights, sparse target-matrix assembly, a mirror of the pending US holdout split and distinct-household ESS. test_us_spine_blindness.py lists it among the runtime modules outside the population-treatment registry.
• test_us_support_mix.py adds 22 property, differential and fixture tests, and test_bakeoff_us_support_mix_tool.py adds 24 tests of the SPM flag, the staleness guards, partcheck, the grid and the report.
• docs/us-support-mix-bakeoff.md, the memo, includes a CPS year response-quality check that asec_quality.py computes.
• experiments/us-support-mix-bakeoff-20260930/ holds the report tables and the compile, diffcheck, partcheck, SPM-flag, ACS-truth, ASEC-quality and per-part materialization receipts. The per-arm receipts and saved weights stayed in the run's work directory, which no longer exists, so a rerun takes every step from compile.

We run the engine once per household and reuse its output across arms:

• compile groups the 32,842 registry targets into 419 geography-free concepts;
• materialize computes those concepts on household shards with the release tool's own _materialize_target_frame;
• each arm assembles concept × state or district mask as a sparse matrix and calls the production solve._optimize with the release's settings.

Invariants

Property tests in test_us_support_mix.py:

• an arm's CPS, ACS and clone source households sum to its budget, and the primitives refuse an infeasible arm instead of truncating it;
• ACS selections nest across budgets and don't depend on input order, and replicates take disjoint blocks of one order;
• clone copies sum to the requested count and differ by at most one across households;
• starting weights stay positive and sum to the requested mass, each CPS year gets an equal share, CPS and ACS split in proportion to source households, and a row's copies sum back to its weight;
• distinct-household ESS never exceeds the row ESS or the number of distinct units;
• the holdout role depends only on the group key: district children share their state parent's key, and vintage tokens never change a role.

Differential tests and checks:

• the sparse target matrix equals a naive dense construction, including cells that carry several targets;
• hash_uniform reproduces draws from the port branch's own implementation;
• diffcheck compared production per-target materialization with concept × mask on 4,000 real households from each source, and across 349 targets the maximum scaled difference came to 0.0;
• partcheck re-materialized 200 households from each of the 8 CPS parts and 8 of the 62 ACS parts under the final concept specs, and across every column the largest scaled difference from the stored values came to 5.8e-8, float32 rounding.

Driver rules in test_bakeoff_us_support_mix_tool.py:

• the tool flags a household for SPM exclusion if and only if one of its SPM units has no member the pinned engine counts as an adult;
• the tool refuses to flag from a person table without household-role columns instead of falling back to age;
• an arm's receipt records digests of the inputs its arm reads and of no others;
• a concept digest moves with the parts' contents and with what their columns mean, the compiled concept specs and the source's measure list;
• materialize refuses a stored concept part whose receipt names another source h5, other concept specs or another null-fill summary, and rebuilds it only under --replace-stale;
• arm --rescore re-scores saved weights only when the rows digest and the inputs match;
• moving the grid to newer CPS years keeps the ACS-only arms' labels;
• replicate tables cover only groups with more than one draw;
• report reads only its own grid's receipts, refuses missing ones unless run with --allow-missing, and refuses stale ones unless run with --allow-stale, which leaves them out;
• partcheck fails a column when either side holds a nonfinite value or the scaled difference exceeds the tolerance, refuses an unknown or empty part selection, and refuses a negative or nonfinite tolerance or a sample size under 1.

Review record

• Round 1 found three High and six Medium findings, and we fixed all of them: collapsed multi-target cells, dropped unsupported holdout targets, overlapping concept parts, national loss weights, group-quarters handling and CPS key checks.
• Round 2 found two High and seven Medium findings, and we fixed them or stated them in the memo. It found the national gain resting on seed 0 alone, so we moved the verdict to three-draw means and the loss-weighted headline. We limited a "more CPS years" claim to the national product and added the counter-readings and the age-cell finding. We also made re-scoring require --rescore and keep the original solve cost, computed the SPM flag in code and left group quarters out of SPM rates.
• Round 3 found no Highs, judged the PR merge-ready and raised four Mediums, which we fixed:
  • the one-year against three-year readings, which we re-read under the noise rule;
  • a false claim about 2020–21 item allocation;
  • staleness bugs that would have re-solved the ACS-only arms and reused stale CPS concept parts in a 2023–2025 rerun;
  • a rescore path that could score weights against changed inputs.
• Round 4 found no Highs, judged the PR merge-ready and raised two rerun-path gaps and six Lows, which we fixed. report now reads only current receipts of its own grid, and concept parts record the specs and null-fill summary they came from; round 5 found receipt reuse still skipping that check, which 4d45da8 closes. Three CPS concept parts predated the last compile, so after that round we added partcheck.
• Round 5 (GPT-6.1 Sol) reviewed 8dc5e7d and 8472fba, judged the PR merge-ready and raised two Medium and three Low findings for reruns, which 4d45da8 addresses (below).
• Round 6 reviews the whole PR at fb91442.

Changes since round 5

• e0b80ea merges main at 1d174fc.

• 4d45da8 fixes round 5's four tool findings, each with a test:

  • receipt digests now cover the concept specs and the measure list, so a specs-only recompile makes old receipts stale;
  • partcheck fails nonfinite columns;
  • partcheck refuses unknown or empty part selections and a bad tolerance or sample size;
  • report --allow-stale alone writes a report and leaves stale receipts out of the missing count.

  It discloses the fifth finding in the memo and the README: the materialize/ receipts predate the provenance digests the work copies carried, and on 2026-09-30 their numerical fields matched the work copies'. The run's work directory, which held the per-arm receipts and saved weights, no longer exists, so the same commit dates the memo's byte-for-byte report reproduction to 2026-09-30 and says a rerun takes every step. We also rewrote the cost notes to name the arms.csv fields that came from the grid log, and cut the memo's review history.

• fb91442 rewrites the memo's be-verb sentences with active verbs. No number moves.

🤖 Generated with Claude Code