From 1eea236ecb6ff83cd252895e5ea5db3ef6867ce4 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Fri, 6 Nov 2026 16:12:13 -0500 Subject: [PATCH 1/5] Add Microcosm's regime-gated QRF to the MOOP imputation experiment Fit Microcosm's RegimeGatedQRF (microcosm-fit at PolicyEngine/microcosm 7080ba8), the model that imputes PolicyEngine's microdata, on the same five household-grouped folds: once with the ASEC SPM unit weights, as Microcosm weights its fits by default, and once without weights, like the other methods. The note and tables report both fits beside microimpute's QRF and ZI-QRF. run.py records the commit each Microcosm package came from, hashes Microcosm's source into the cache key and caps Microcosm's threads at --n-jobs. Microcosm installs from git and needs Python 3.13, so the experiment imports it only inside the Microcosm method and its tests skip without it. Co-Authored-By: Claude Opus 5.5 --- changelog.d/moop-spm-microcosm-qrf.changed.md | 1 + experiments/moop_spm/NOTE.md | 40 +-- experiments/moop_spm/NOTE.template.md | 12 +- experiments/moop_spm/README.md | 56 +++- experiments/moop_spm/TABLE.md | 82 +++--- experiments/moop_spm/methods.py | 87 +++++- experiments/moop_spm/render.py | 30 +- experiments/moop_spm/results/results.json | 261 +++++++++++++++++- experiments/moop_spm/run.py | 102 ++++++- tests/test_moop_spm_experiment.py | 92 +++++- 10 files changed, 663 insertions(+), 100 deletions(-) create mode 100644 changelog.d/moop-spm-microcosm-qrf.changed.md diff --git a/changelog.d/moop-spm-microcosm-qrf.changed.md b/changelog.d/moop-spm-microcosm-qrf.changed.md new file mode 100644 index 00000000..212af4de --- /dev/null +++ b/changelog.d/moop-spm-microcosm-qrf.changed.md @@ -0,0 +1 @@ +Add Microcosm's regime-gated QRF, fit with and without the ASEC unit weights, to the MOOP imputation experiment (`experiments/moop_spm`). diff --git a/experiments/moop_spm/NOTE.md b/experiments/moop_spm/NOTE.md index f9309ce5..0da5a114 100644 --- a/experiments/moop_spm/NOTE.md +++ b/experiments/moop_spm/NOTE.md @@ -6,40 +6,40 @@ The American Community Survey (ACS) does not ask about medical out-of-pocket spe We use the 134,729 people, 55,401 SPM units and 53,344 households in the 2026 CPS ASEC public-use file, which covers calendar-year 2025. We impute each unit's SPM medical expenses (`SPM_MEDXPNS`), its members' MOOP plus their Medicare premiums. Census subtracts them from SPM resources dollar for dollar, so we swap each unit's reported value for an imputed one and compare the new resources with an unchanged threshold. With reported MOOP and the ASEC person weight, we compute an SPM rate of 13.11% with a 90% margin of error of 0.31 points. Census published 13.11% and 0.31 in [P60-290](https://www.census.gov/library/publications/2026/demo/p60-290.html). Subtracting MOOP raises the rate by 2.25 points (SE 0.08), and Census reports 2.28. -We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to 5 folds and impute each fold from models we fit on the rest, without survey weights. We average 10 draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. +We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to 5 folds and impute each fold from models we fit on the rest, all without survey weights except one fit in method (c). We average 10 draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests from PolicyEngine's microimpute package, alone (QRF) and inside its zero-inflated wrapper (ZI-QRF). We give each unit the value at a random quantile of its predicted distribution. In PolicyEngine's microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest grown on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results -| | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:| -| SPM rate, all people (%) | 13.11 | 13.42 | 13.46 | 13.50 | 13.43 | 13.04 | 13.07 | -| Change, all people (pp, SE) | | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | -| Change, 65 and older (pp) | | +0.98 | +1.15 | +0.75 | +0.60 | -0.10 | -0.13 | -| Change, with a disability (pp) | | +1.20 | +1.40 | +0.59 | +0.41 | -0.21 | -0.22 | -| Change, Medicare (pp) | | +1.20 | +1.23 | +0.85 | +0.59 | -0.08 | -0.11 | -| Change, resources before MOOP 1.0 to 1.25 x threshold (pp) | | +14.69 | +14.37 | +4.88 | +4.19 | +0.55 | +0.61 | -| Mean MOOP, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | -| Units with zero MOOP (%) | 6.4 | 0.5 | 4.3 | 6.5 | 6.2 | 6.2 | 6.1 | -| Median MOOP | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | -| 99th percentile MOOP | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | -| Total MOOP ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | -| People newly poor / newly not poor (%) | | 1.37 / 1.07 | 1.40 / 1.06 | 1.74 / 1.36 | 1.63 / 1.31 | 1.33 / 1.40 | 1.37 / 1.41 | +| | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| SPM rate, all people (%) | 13.11 | 13.42 | 13.46 | 13.50 | 13.43 | 13.04 | 13.07 | 13.18 | 13.11 | +| Change, all people (pp, SE) | | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | +0.06 (0.07) | +0.00 (0.06) | +| Change, 65 and older (pp) | | +0.98 | +1.15 | +0.75 | +0.60 | -0.10 | -0.13 | -0.04 | -0.02 | +| Change, with a disability (pp) | | +1.20 | +1.40 | +0.59 | +0.41 | -0.21 | -0.22 | -0.17 | -0.23 | +| Change, Medicare (pp) | | +1.20 | +1.23 | +0.85 | +0.59 | -0.08 | -0.11 | -0.04 | -0.03 | +| Change, resources before MOOP 1.0 to 1.25 x threshold (pp) | | +14.69 | +14.37 | +4.88 | +4.19 | +0.55 | +0.61 | +0.42 | -0.13 | +| Mean MOOP, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | $2,434 | $2,389 | +| Units with zero MOOP (%) | 6.4 | 0.5 | 4.3 | 6.5 | 6.2 | 6.2 | 6.1 | 6.4 | 6.3 | +| Median MOOP | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | $3,602 | $3,517 | +| 99th percentile MOOP | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | $33,749 | $32,872 | +| Total MOOP ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | 850 | 842 | +| People newly poor / newly not poor (%) | | 1.37 / 1.07 | 1.40 / 1.06 | 1.74 / 1.36 | 1.63 / 1.31 | 1.33 / 1.40 | 1.37 / 1.41 | 1.47 / 1.41 | 1.41 / 1.42 | We compute rates and changes over people and MOOP statistics over SPM units, and show standard errors in parentheses. -We estimate that logit plus cell means raises the SPM rate by 0.30 percentage points (SE 0.08), or 13% of MOOP's own 2.25-point effect. The OLS variant adds 0.34 points, logit draws with donors 0.39 and PMM 0.32. QRF changes the rate by -0.07 points (SE 0.07) and ZI-QRF by -0.05 (SE 0.07), though their draws fall 3.6% and 3.7% short of the reported MOOP total. For people 65 and older, the rate changes by +0.98 points under logit plus cell means, +0.60 under PMM and -0.10 under QRF. For people with a disability, it changes by +1.20, +0.41 and -0.21. +We estimate that logit plus cell means raises the SPM rate by 0.30 percentage points (SE 0.08), or 13% of MOOP's own 2.25-point effect. The OLS variant adds 0.34 points, logit draws with donors 0.39 and PMM 0.32. All four QRF fits leave the rate within 0.1 point of its level with reported MOOP. Microcosm's QRF changes it by +0.06 points (SE 0.07) with the unit weights and by +0.00 (SE 0.06) without them, and its weighted draws fall 1.0% short of the reported MOOP total. The microimpute QRF and ZI-QRF change the rate by -0.07 (SE 0.07) and -0.05 (SE 0.07), and their draws fall 3.6% and 3.7% short. For people 65 and older, the rate changes by +0.98 points under logit plus cell means, +0.60 under PMM and -0.04 under Microcosm's weighted QRF. For people with a disability, it changes by +1.20, +0.41 and -0.17. -Logit draws with donors track the reported distribution, with 6.5% of units at zero against 6.4% and a 99th percentile of $33,147 against $33,640, yet raise the rate by 0.39 points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend $2,385 on average, and units at 2 to 3 times spend $6,086. In that first band above the line, cell means assign $3,189, donor draws $2,949, PMM $2,799 and QRF $2,352. The band's 24.9% poverty rate rises by 14.69, 4.88, 4.19 and 0.55 points under those four methods. +Logit draws with donors track the reported distribution, with 6.5% of units at zero against 6.4% and a 99th percentile of $33,147 against $33,640, yet raise the rate by 0.39 points. Microcosm's weighted QRF also tracks it, with 6.4% at zero and a 99th percentile of $33,749, and moves the rate by +0.06 points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend $2,385 on average, and units at 2 to 3 times spend $6,086. In that first band above the line, cell means assign $3,189, donor draws $2,949, PMM $2,799 and Microcosm's weighted QRF $2,434. The band's 24.9% poverty rate rises by 14.69, 4.88, 4.19 and 0.42 points under those four methods. Under the 0.5 cutoff, 0.5% of units get no MOOP, against 6.4% in the file. Drawing participation instead lowers the national change from +0.30 to +0.10 points with cell means, and from +0.59 to +0.39 with donor draws (Table 3 in `TABLE.md`). When we draw participation and assign cell means, the rate still rises 11.50 points for people at 1.0 to 1.25 times the threshold and falls 4.80 points at 1.25 to 1.5. Cell means also cut the 99th percentile from $33,640 to $15,585. Under the same cutoff, coverage-group means alone, without the age, income and size cells, raise the national rate by 2.07 points, 92% of MOOP's own effect. -Every method moves people across the line in both directions. QRF moves 1.33% of people into poverty and 1.40% out of it, and logit plus cell means moves 1.37% in and 1.07% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. +Every method moves people across the line in both directions. Microcosm's weighted QRF moves 1.47% of people into poverty and 1.41% out of it, and logit plus cell means moves 1.37% in and 1.07% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. -Census imputes MOOP for 43% of the people in the file and the Medicare premium for 13%, so part of our benchmark comes from its imputation. In the 44% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by 0.26 to 0.52 points. QRF and ZI-QRF change it by +0.01 and +0.07 points (SE 0.09 and 0.10). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. +Census imputes MOOP for 43% of the people in the file and the Medicare premium for 13%, so part of our benchmark comes from its imputation. In the 44% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by 0.26 to 0.52 points. The four QRF fits change it by +0.01 to +0.14 points (SE 0.09 to 0.10). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. For each method's change by age, disability, coverage type and resources relative to the threshold, see Table 1 in `TABLE.md`; for the code, see `README.md`. diff --git a/experiments/moop_spm/NOTE.template.md b/experiments/moop_spm/NOTE.template.md index 6abab906..5a795b25 100644 --- a/experiments/moop_spm/NOTE.template.md +++ b/experiments/moop_spm/NOTE.template.md @@ -6,11 +6,11 @@ The American Community Survey (ACS) does not ask about medical out-of-pocket spe We use the {n_persons} people, {n_units} SPM units and {n_households} households in the 2026 CPS ASEC public-use file, which covers calendar-year 2025. We impute each unit's SPM medical expenses (`SPM_MEDXPNS`), its members' MOOP plus their Medicare premiums. Census subtracts them from SPM resources dollar for dollar, so we swap each unit's reported value for an imputed one and compare the new resources with an unchanged threshold. With reported MOOP and the ASEC person weight, we compute an SPM rate of {rate2_all_reported}% with a 90% margin of error of {reported_moe90} points. Census published {published_rate}% and {published_moe90} in [P60-290](https://www.census.gov/library/publications/2026/demo/p60-290.html). Subtracting MOOP raises the rate by {moop_effect} points (SE {moop_effect_se}), and Census reports {published_moop_effect}. -We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to {folds} folds and impute each fold from models we fit on the rest, without survey weights. We average {draws} draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. +We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to {folds} folds and impute each fold from models we fit on the rest, all without survey weights except one fit in method (c). We average {draws} draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests from PolicyEngine's microimpute package, alone (QRF) and inside its zero-inflated wrapper (ZI-QRF). We give each unit the value at a random quantile of its predicted distribution. In PolicyEngine's microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest grown on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results @@ -19,14 +19,14 @@ We limit every method to predictors an ACS-style file carries: members' ages, se We compute rates and changes over people and MOOP statistics over SPM units, and show standard errors in parentheses. -We estimate that logit plus cell means raises the SPM rate by {pp_all_logit_cell_mean} percentage points (SE {se_all_logit_cell_mean}), or {share_of_effect_logit_cell_mean}% of MOOP's own {moop_effect}-point effect. The OLS variant adds {pp_all_logit_reg_mean} points, logit draws with donors {pp_all_logit_draw_hotdeck} and PMM {pp_all_pmm}. QRF changes the rate by {diff_all_qrf} points (SE {se_all_qrf}) and ZI-QRF by {diff_all_zi_qrf} (SE {se_all_zi_qrf}), though their draws fall {agg_gap_qrf}% and {agg_gap_zi_qrf}% short of the reported MOOP total. For people 65 and older, the rate changes by {diff_senior_logit_cell_mean} points under logit plus cell means, {diff_senior_pmm} under PMM and {diff_senior_qrf} under QRF. For people with a disability, it changes by {diff_disabled_logit_cell_mean}, {diff_disabled_pmm} and {diff_disabled_qrf}. +We estimate that logit plus cell means raises the SPM rate by {pp_all_logit_cell_mean} percentage points (SE {se_all_logit_cell_mean}), or {share_of_effect_logit_cell_mean}% of MOOP's own {moop_effect}-point effect. The OLS variant adds {pp_all_logit_reg_mean} points, logit draws with donors {pp_all_logit_draw_hotdeck} and PMM {pp_all_pmm}. All four QRF fits leave the rate within 0.1 point of its level with reported MOOP. Microcosm's QRF changes it by {diff_all_microcosm_qrf} points (SE {se_all_microcosm_qrf}) with the unit weights and by {diff_all_microcosm_qrf_unweighted} (SE {se_all_microcosm_qrf_unweighted}) without them, and its weighted draws fall {agg_gap_microcosm_qrf}% short of the reported MOOP total. The microimpute QRF and ZI-QRF change the rate by {diff_all_qrf} (SE {se_all_qrf}) and {diff_all_zi_qrf} (SE {se_all_zi_qrf}), and their draws fall {agg_gap_qrf}% and {agg_gap_zi_qrf}% short. For people 65 and older, the rate changes by {diff_senior_logit_cell_mean} points under logit plus cell means, {diff_senior_pmm} under PMM and {diff_senior_microcosm_qrf} under Microcosm's weighted QRF. For people with a disability, it changes by {diff_disabled_logit_cell_mean}, {diff_disabled_pmm} and {diff_disabled_microcosm_qrf}. -Logit draws with donors track the reported distribution, with {zero_logit_draw_hotdeck}% of units at zero against {zero_reported}% and a 99th percentile of {p99_logit_draw_hotdeck} against {p99_reported}, yet raise the rate by {pp_all_logit_draw_hotdeck} points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend {bandmean1_reported} on average, and units at 2 to 3 times spend {bandmean4_reported}. In that first band above the line, cell means assign {bandmean1_logit_cell_mean}, donor draws {bandmean1_logit_draw_hotdeck}, PMM {bandmean1_pmm} and QRF {bandmean1_qrf}. The band's {rate_band1_reported}% poverty rate rises by {pp_band1_logit_cell_mean}, {pp_band1_logit_draw_hotdeck}, {pp_band1_pmm} and {pp_band1_qrf} points under those four methods. +Logit draws with donors track the reported distribution, with {zero_logit_draw_hotdeck}% of units at zero against {zero_reported}% and a 99th percentile of {p99_logit_draw_hotdeck} against {p99_reported}, yet raise the rate by {pp_all_logit_draw_hotdeck} points. Microcosm's weighted QRF also tracks it, with {zero_microcosm_qrf}% at zero and a 99th percentile of {p99_microcosm_qrf}, and moves the rate by {diff_all_microcosm_qrf} points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend {bandmean1_reported} on average, and units at 2 to 3 times spend {bandmean4_reported}. In that first band above the line, cell means assign {bandmean1_logit_cell_mean}, donor draws {bandmean1_logit_draw_hotdeck}, PMM {bandmean1_pmm} and Microcosm's weighted QRF {bandmean1_microcosm_qrf}. The band's {rate_band1_reported}% poverty rate rises by {pp_band1_logit_cell_mean}, {pp_band1_logit_draw_hotdeck}, {pp_band1_pmm} and {pp_band1_microcosm_qrf} points under those four methods. Under the 0.5 cutoff, {zero_logit_cell_mean}% of units get no MOOP, against {zero_reported}% in the file. Drawing participation instead lowers the national change from {diff_all_logit_cell_mean} to {diff_all_logit_draw_cell_mean} points with cell means, and from {diff_all_logit_threshold_hotdeck} to {diff_all_logit_draw_hotdeck} with donor draws (Table 3 in `TABLE.md`). When we draw participation and assign cell means, the rate still rises {pp_band1_logit_draw_cell_mean} points for people at 1.0 to 1.25 times the threshold and falls {pp_band2_logit_draw_cell_mean} points at 1.25 to 1.5. Cell means also cut the 99th percentile from {p99_reported} to {p99_logit_cell_mean}. Under the same cutoff, coverage-group means alone, without the age, income and size cells, raise the national rate by {pp_all_logit_group_mean} points, {share_of_effect_logit_group_mean}% of MOOP's own effect. -Every method moves people across the line in both directions. QRF moves {newly_poor_qrf}% of people into poverty and {newly_nonpoor_qrf}% out of it, and logit plus cell means moves {newly_poor_logit_cell_mean}% in and {newly_nonpoor_logit_cell_mean}% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. +Every method moves people across the line in both directions. Microcosm's weighted QRF moves {newly_poor_microcosm_qrf}% of people into poverty and {newly_nonpoor_microcosm_qrf}% out of it, and logit plus cell means moves {newly_poor_logit_cell_mean}% in and {newly_nonpoor_logit_cell_mean}% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. -Census imputes MOOP for {share_persons_moop_allocated}% of the people in the file and the Medicare premium for {share_persons_premium_allocated}%, so part of our benchmark comes from its imputation. In the {share_units_no_allocation}% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by {subset_pp_low_ab_pmm} to {subset_pp_high_ab_pmm} points. QRF and ZI-QRF change it by {subset_diff_qrf} and {subset_diff_zi_qrf} points (SE {subset_se_qrf} and {subset_se_zi_qrf}). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. +Census imputes MOOP for {share_persons_moop_allocated}% of the people in the file and the Medicare premium for {share_persons_premium_allocated}%, so part of our benchmark comes from its imputation. In the {share_units_no_allocation}% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by {subset_pp_low_ab_pmm} to {subset_pp_high_ab_pmm} points. The four QRF fits change it by {subset_diff_low_forest} to {subset_diff_high_forest} points (SE {subset_se_low_forest} to {subset_se_high_forest}). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. For each method's change by age, disability, coverage type and resources relative to the threshold, see Table 1 in `TABLE.md`; for the code, see `README.md`. diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index 2ddac855..43c584ab 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -60,8 +60,8 @@ carries: ## Methods -We fit every method on donor units, without survey weights, and impute -held-out recipient units (`methods.py`). +We fit every method on donor units and impute held-out recipient units +(`methods.py`). Only the weighted Microcosm fit uses survey weights. | Name | Participation | Amount | |---|---|---| @@ -71,6 +71,27 @@ held-out recipient units (`methods.py`). | `pmm` | implicit | predictive mean matching on log(1 + MOOP) within five coverage groups, random one of 5 nearest donors | | (c) `qrf` | implicit | microimpute `QRF`, one random quantile per unit | | (c) `zi_qrf` | microimpute gate | microimpute `ZeroInflatedImputer` around `QRF` | +| (c) `microcosm_qrf` | Microcosm gate, a gradient-boosted classifier | Microcosm `RegimeGatedQRF` forest grown on donor spenders, one random quantile per unit; gate and forest use the SPM unit weights | +| (c) `microcosm_qrf_unweighted` | same, without weights | same, without weights | + +Microcosm (`PolicyEngine/microcosm`) builds PolicyEngine's microdata, +and its `microcosm-fit` package imputes with `RegimeGatedQRF`. We fit it +with its defaults (100 trees, every leaf sample kept, no cross-fitting +of the gate), once with each donor unit's SPM weight (`SPM_WEIGHT / 100`) and +once without weights. The weighted fit passes the weights to the gate as +sample weights and grows the forest on spenders resampled in proportion +to them. Both fits share a seed, so their draws use the same random +numbers. Beyond the weights, Microcosm's model differs from microimpute's +`QRF` in three ways: + +- Its gate draws whether a unit spends. microimpute's `QRF` has no + gate, and `ZeroInflatedImputer` adds one of its own. +- It keeps every training sample in each leaf, where microimpute keeps + quantile-forest's default of at most one. +- It reads each unit's draw at that unit's own quantile, interpolating + a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute + rounds the quantile to the nearest of 101 points from 1/102 to + 101/102. We form cells from the unit's coverage group (Medicare, employer, direct purchase, public, uninsured), the head's age band, cash income relative @@ -106,7 +127,19 @@ cells). ## Running -Install microimpute, then run from the repository root: +Install microimpute with its `dev` extra, and Microcosm's `graph`, +`frame` and `fit` packages at the commit we used. Microcosm needs Python +3.13 or later and installs from git: + +```bash +uv pip install -e ".[dev]" +uv pip install \ + "microcosm-graph @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-graph" \ + "microcosm-frame @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-frame" \ + "microcosm-fit @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-fit" +``` + +Then run from the repository root: ```bash python -m experiments.moop_spm.run --data-dir ~/data/cps_asec_2026 @@ -115,15 +148,20 @@ python -m experiments.moop_spm.render The first command downloads the 139 MB Census zip, checks its SHA-256, caches each fold's imputations in `results/cache/`, which git ignores, -and writes `results/results.json`. +and writes `results/results.json`. `--n-jobs` (default 4) caps the +threads of microimpute's `QRF` and of each Microcosm fit; a test checks +that Microcosm's draws do not depend on it. `run.py` refits every fold when any of these change: - the settings or the experiment's source files; -- any microimpute source file; +- any source file of microimpute or of Microcosm's `fit`, `frame` and + `graph` packages; - the version of Python, numpy, pandas, scipy, scikit-learn, - quantile-forest or microimpute; + quantile-forest, microimpute or Microcosm, or the commit Microcosm came + from; - the model inputs or their row keys. -`run.py` also records each method's run time in `results.json`. The -tests in `tests/test_moop_spm_experiment.py` check the experiment's -invariants on synthetic data. +`run.py` also records each method's run time, and the commit each +Microcosm package came from, in `results.json`. The tests in +`tests/test_moop_spm_experiment.py` check the experiment's invariants on +synthetic data; the Microcosm tests skip on a machine without Microcosm. diff --git a/experiments/moop_spm/TABLE.md b/experiments/moop_spm/TABLE.md index ce282bfd..64a8a208 100644 --- a/experiments/moop_spm/TABLE.md +++ b/experiments/moop_spm/TABLE.md @@ -4,52 +4,52 @@ Source: CPS ASEC 2026 public-use file (calendar-year 2025 income), 134,729 peopl ## Table 1. SPM poverty rate with reported MOOP (%), and the change under each imputation (percentage points, SE) -| Group | People (n) | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| All people | 134,729 | 13.1 | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | -| **Age** | | | | | | | | | -| Under 18 | 31,444 | 13.4 | +0.17 (0.13) | +0.26 (0.14) | +0.42 (0.11) | +0.41 (0.11) | +0.00 (0.11) | +0.06 (0.11) | -| 18 to 64 | 77,083 | 12.3 | +0.13 (0.09) | +0.11 (0.09) | +0.26 (0.07) | +0.19 (0.07) | -0.08 (0.07) | -0.06 (0.07) | -| 65 and older | 26,202 | 15.4 | +0.98 (0.17) | +1.15 (0.16) | +0.75 (0.13) | +0.60 (0.15) | -0.10 (0.14) | -0.13 (0.13) | -| **Disability** | | | | | | | | | -| With a disability | 14,477 | 22.7 | +1.20 (0.22) | +1.40 (0.21) | +0.59 (0.18) | +0.41 (0.20) | -0.21 (0.18) | -0.22 (0.18) | -| No disability | 94,282 | 11.7 | +0.21 (0.09) | +0.20 (0.09) | +0.35 (0.07) | +0.28 (0.07) | -0.07 (0.07) | -0.04 (0.07) | -| Not asked (mostly under 15) | 25,970 | 13.2 | +0.15 (0.14) | +0.28 (0.15) | +0.42 (0.11) | +0.41 (0.12) | +0.02 (0.12) | +0.05 (0.12) | -| **Current health insurance** | | | | | | | | | -| Medicare | 27,328 | 16.6 | +1.20 (0.17) | +1.23 (0.16) | +0.85 (0.14) | +0.59 (0.15) | -0.08 (0.15) | -0.11 (0.13) | -| Employer | 66,520 | 3.9 | -0.21 (0.08) | -0.12 (0.08) | +0.14 (0.07) | +0.23 (0.07) | -0.02 (0.07) | +0.02 (0.08) | -| Direct purchase | 8,341 | 18.3 | +0.06 (0.39) | +2.28 (0.48) | +0.20 (0.32) | +0.71 (0.36) | -0.22 (0.36) | -0.04 (0.36) | -| Medicaid or other means-tested | 18,134 | 32.1 | +1.18 (0.23) | +0.32 (0.22) | +0.99 (0.18) | +0.33 (0.20) | +0.01 (0.17) | -0.03 (0.18) | -| TRICARE, VA or CHAMPVA | 3,174 | 7.2 | -0.17 (0.21) | -0.59 (0.36) | -0.29 (0.29) | -0.15 (0.29) | -0.01 (0.26) | -0.09 (0.30) | -| Uninsured | 11,232 | 25.8 | +0.07 (0.27) | -0.22 (0.28) | +0.06 (0.23) | -0.05 (0.22) | -0.33 (0.23) | -0.28 (0.24) | -| **Resources before MOOP / threshold** | | | | | | | | | -| Under 1.0 | 13,803 | 100.0 | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | -| 1.0 to 1.25 | 8,441 | 24.9 | +14.69 (1.01) | +14.37 (1.05) | +4.88 (0.74) | +4.19 (0.75) | +0.55 (0.79) | +0.61 (0.81) | -| 1.25 to 1.5 | 9,233 | 5.8 | -4.75 (0.50) | -3.92 (0.49) | +0.50 (0.49) | +0.23 (0.51) | -0.76 (0.48) | -0.57 (0.50) | -| 1.5 to 2.0 | 17,495 | 1.6 | -1.56 (0.15) | -1.54 (0.15) | +0.19 (0.17) | +0.17 (0.18) | -0.29 (0.17) | -0.26 (0.16) | -| 2.0 to 3.0 | 30,010 | 0.3 | -0.32 (0.07) | -0.32 (0.07) | +0.05 (0.07) | +0.03 (0.08) | -0.10 (0.07) | -0.06 (0.07) | -| 3.0 or more | 55,747 | 0.1 | -0.05 (0.02) | -0.05 (0.02) | +0.03 (0.02) | +0.02 (0.02) | +0.02 (0.02) | +0.00 (0.02) | +| Group | People (n) | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| All people | 134,729 | 13.1 | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | +0.06 (0.07) | +0.00 (0.06) | +| **Age** | | | | | | | | | | | +| Under 18 | 31,444 | 13.4 | +0.17 (0.13) | +0.26 (0.14) | +0.42 (0.11) | +0.41 (0.11) | +0.00 (0.11) | +0.06 (0.11) | +0.14 (0.10) | +0.02 (0.10) | +| 18 to 64 | 77,083 | 12.3 | +0.13 (0.09) | +0.11 (0.09) | +0.26 (0.07) | +0.19 (0.07) | -0.08 (0.07) | -0.06 (0.07) | +0.07 (0.07) | -0.01 (0.07) | +| 65 and older | 26,202 | 15.4 | +0.98 (0.17) | +1.15 (0.16) | +0.75 (0.13) | +0.60 (0.15) | -0.10 (0.14) | -0.13 (0.13) | -0.04 (0.13) | -0.02 (0.13) | +| **Disability** | | | | | | | | | | | +| With a disability | 14,477 | 22.7 | +1.20 (0.22) | +1.40 (0.21) | +0.59 (0.18) | +0.41 (0.20) | -0.21 (0.18) | -0.22 (0.18) | -0.17 (0.18) | -0.23 (0.17) | +| No disability | 94,282 | 11.7 | +0.21 (0.09) | +0.20 (0.09) | +0.35 (0.07) | +0.28 (0.07) | -0.07 (0.07) | -0.04 (0.07) | +0.09 (0.07) | +0.02 (0.07) | +| Not asked (mostly under 15) | 25,970 | 13.2 | +0.15 (0.14) | +0.28 (0.15) | +0.42 (0.11) | +0.41 (0.12) | +0.02 (0.12) | +0.05 (0.12) | +0.13 (0.11) | +0.03 (0.12) | +| **Current health insurance** | | | | | | | | | | | +| Medicare | 27,328 | 16.6 | +1.20 (0.17) | +1.23 (0.16) | +0.85 (0.14) | +0.59 (0.15) | -0.08 (0.15) | -0.11 (0.13) | -0.04 (0.14) | -0.03 (0.14) | +| Employer | 66,520 | 3.9 | -0.21 (0.08) | -0.12 (0.08) | +0.14 (0.07) | +0.23 (0.07) | -0.02 (0.07) | +0.02 (0.08) | +0.06 (0.07) | +0.00 (0.07) | +| Direct purchase | 8,341 | 18.3 | +0.06 (0.39) | +2.28 (0.48) | +0.20 (0.32) | +0.71 (0.36) | -0.22 (0.36) | -0.04 (0.36) | +0.63 (0.36) | +0.21 (0.35) | +| Medicaid or other means-tested | 18,134 | 32.1 | +1.18 (0.23) | +0.32 (0.22) | +0.99 (0.18) | +0.33 (0.20) | +0.01 (0.17) | -0.03 (0.18) | +0.12 (0.17) | +0.04 (0.18) | +| TRICARE, VA or CHAMPVA | 3,174 | 7.2 | -0.17 (0.21) | -0.59 (0.36) | -0.29 (0.29) | -0.15 (0.29) | -0.01 (0.26) | -0.09 (0.30) | -0.53 (0.35) | -0.17 (0.30) | +| Uninsured | 11,232 | 25.8 | +0.07 (0.27) | -0.22 (0.28) | +0.06 (0.23) | -0.05 (0.22) | -0.33 (0.23) | -0.28 (0.24) | -0.05 (0.23) | -0.17 (0.25) | +| **Resources before MOOP / threshold** | | | | | | | | | | | +| Under 1.0 | 13,803 | 100.0 | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +| 1.0 to 1.25 | 8,441 | 24.9 | +14.69 (1.01) | +14.37 (1.05) | +4.88 (0.74) | +4.19 (0.75) | +0.55 (0.79) | +0.61 (0.81) | +0.42 (0.75) | -0.13 (0.77) | +| 1.25 to 1.5 | 9,233 | 5.8 | -4.75 (0.50) | -3.92 (0.49) | +0.50 (0.49) | +0.23 (0.51) | -0.76 (0.48) | -0.57 (0.50) | +0.03 (0.49) | -0.23 (0.49) | +| 1.5 to 2.0 | 17,495 | 1.6 | -1.56 (0.15) | -1.54 (0.15) | +0.19 (0.17) | +0.17 (0.18) | -0.29 (0.17) | -0.26 (0.16) | +0.12 (0.17) | +0.03 (0.16) | +| 2.0 to 3.0 | 30,010 | 0.3 | -0.32 (0.07) | -0.32 (0.07) | +0.05 (0.07) | +0.03 (0.08) | -0.10 (0.07) | -0.06 (0.07) | +0.04 (0.08) | -0.01 (0.08) | +| 3.0 or more | 55,747 | 0.1 | -0.05 (0.02) | -0.05 (0.02) | +0.03 (0.02) | +0.02 (0.02) | +0.02 (0.02) | +0.00 (0.02) | +0.03 (0.02) | +0.04 (0.02) | Every method imputes MOOP of zero or more, so people whose unit's resources before MOOP fall below its threshold stay poor, and the 'Under 1.0' row cannot change. ## Table 2. MOOP distribution across SPM units (unit weights) and gross flows of people (person weights) -| Statistic | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:| -| Mean | $6,011 | $6,156 | $6,133 | $5,984 | $5,922 | $5,797 | $5,791 | -| Median | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | -| Share with zero MOOP | 6.4% | 0.5% | 4.3% | 6.5% | 6.2% | 6.2% | 6.1% | -| 75th percentile | $8,160 | $8,625 | $8,983 | $8,154 | $8,091 | $7,949 | $7,891 | -| 90th percentile | $14,300 | $11,436 | $11,507 | $14,276 | $14,158 | $13,820 | $13,765 | -| 99th percentile | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | -| Total ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | -| Mean, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | -| Mean, resources before MOOP 1.25 to 1.5 x threshold | $3,381 | $3,679 | $3,667 | $3,446 | $3,452 | $3,282 | $3,272 | -| Mean, resources before MOOP 1.5 to 2.0 x threshold | $4,510 | $4,788 | $4,612 | $4,546 | $4,527 | $4,335 | $4,389 | -| Mean, resources before MOOP 2.0 to 3.0 x threshold | $6,086 | $6,116 | $5,872 | $5,953 | $5,783 | $5,809 | $5,843 | -| People newly poor (%) | 0.0 | 1.37 | 1.40 | 1.74 | 1.63 | 1.33 | 1.37 | -| People newly not poor (%) | 0.0 | 1.07 | 1.06 | 1.36 | 1.31 | 1.40 | 1.41 | -| Change in rate among people in units where Census imputes no MOOP or Medicare premium (pp) | 0.00 | +0.36 (0.11) | +0.26 (0.11) | +0.52 (0.09) | +0.48 (0.10) | +0.01 (0.09) | +0.07 (0.10) | +| Statistic | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Mean | $6,011 | $6,156 | $6,133 | $5,984 | $5,922 | $5,797 | $5,791 | $5,949 | $5,891 | +| Median | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | $3,602 | $3,517 | +| Share with zero MOOP | 6.4% | 0.5% | 4.3% | 6.5% | 6.2% | 6.2% | 6.1% | 6.4% | 6.3% | +| 75th percentile | $8,160 | $8,625 | $8,983 | $8,154 | $8,091 | $7,949 | $7,891 | $8,040 | $7,982 | +| 90th percentile | $14,300 | $11,436 | $11,507 | $14,276 | $14,158 | $13,820 | $13,765 | $14,285 | $14,139 | +| 99th percentile | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | $33,749 | $32,872 | +| Total ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | 850 | 842 | +| Mean, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | $2,434 | $2,389 | +| Mean, resources before MOOP 1.25 to 1.5 x threshold | $3,381 | $3,679 | $3,667 | $3,446 | $3,452 | $3,282 | $3,272 | $3,432 | $3,344 | +| Mean, resources before MOOP 1.5 to 2.0 x threshold | $4,510 | $4,788 | $4,612 | $4,546 | $4,527 | $4,335 | $4,389 | $4,473 | $4,427 | +| Mean, resources before MOOP 2.0 to 3.0 x threshold | $6,086 | $6,116 | $5,872 | $5,953 | $5,783 | $5,809 | $5,843 | $6,071 | $5,929 | +| People newly poor (%) | 0.0 | 1.37 | 1.40 | 1.74 | 1.63 | 1.33 | 1.37 | 1.47 | 1.41 | +| People newly not poor (%) | 0.0 | 1.07 | 1.06 | 1.36 | 1.31 | 1.40 | 1.41 | 1.41 | 1.42 | +| Change in rate among people in units where Census imputes no MOOP or Medicare premium (pp) | 0.00 | +0.36 (0.11) | +0.26 (0.11) | +0.52 (0.09) | +0.48 (0.10) | +0.01 (0.09) | +0.07 (0.10) | +0.14 (0.09) | +0.05 (0.09) | ## Table 3. How the participation and amount rules of the logit approach move the rate (all people) diff --git a/experiments/moop_spm/methods.py b/experiments/moop_spm/methods.py index 90af7811..cba52e78 100644 --- a/experiments/moop_spm/methods.py +++ b/experiments/moop_spm/methods.py @@ -21,6 +21,12 @@ - ``qrf``: microimpute's ``QRF`` fit on all donors, one random quantile draw per recipient. - ``zi_qrf``: microimpute's ``ZeroInflatedImputer`` around ``QRF``. +- ``microcosm_qrf``: Microcosm's ``RegimeGatedQRF``, the model that + imputes PolicyEngine's microdata, fit with the donors' unit weights. + On Microcosm's own data frames, a fit uses the frame's weights unless + told not to. +- ``microcosm_qrf_unweighted``: the same model fit without weights, like + every other method here. Decomposition variants cross the participation rule (threshold or draw) with the amount rule (cell mean or donor draw), and @@ -31,7 +37,9 @@ from __future__ import annotations -from collections.abc import Callable, Sequence +import os +from collections.abc import Callable, Iterator, Sequence +from contextlib import contextmanager from dataclasses import dataclass import numpy as np @@ -170,10 +178,15 @@ def participation_probability(model: Pipeline, X: pd.DataFrame) -> np.ndarray: @dataclass class FoldData: - """Donor and recipient data for one cross-fitting fold.""" + """Donor and recipient data for one cross-fitting fold. + + ``weight_donor`` holds the donors' SPM unit weights; only the weighted + Microcosm fit reads them. + """ X_donor: pd.DataFrame moop_donor: np.ndarray + weight_donor: np.ndarray cells_donor: pd.DataFrame group_donor: np.ndarray X_recipient: pd.DataFrame @@ -353,6 +366,70 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: return draws +@contextmanager +def microcosm_threads(n_jobs: int) -> Iterator[None]: + """Cap Microcosm's threads at ``n_jobs``. + + Microcosm reads its forest-fitting and drawing widths from environment + variables that its ``qrf`` module names; when they are unset, it fits + and draws on every core. Its gate, a scikit-learn gradient-boosted + classifier, runs on OpenMP threads, which ``threadpoolctl`` caps. The + previous settings come back on exit. + """ + from microcosm.fit import qrf as microcosm_qrf_module + from threadpoolctl import threadpool_limits + + names = ( + microcosm_qrf_module._N_JOBS_ENV, + microcosm_qrf_module._PREDICT_WORKERS_ENV, + ) + saved = {name: os.environ.get(name) for name in names} + os.environ.update({name: str(n_jobs) for name in names}) + try: + with threadpool_limits(limits=n_jobs, user_api="openmp"): + yield + finally: + for name, value in saved.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + + +def microcosm_qrf( + fold: FoldData, + n_draws: int, + seed: int, + weighted: bool, + n_jobs: int = 4, +) -> np.ndarray: + """Microcosm's ``RegimeGatedQRF`` on all donors. + + A gradient-boosted classifier gates zero against positive MOOP, and a + quantile regression forest grown on donor spenders gives each gated + spender the amount at a random quantile of its predicted distribution. + With ``weighted``, the gate takes the donors' unit weights as sample + weights and the forest grows on spenders resampled in proportion to + them; without it, both ignore the weights. Each draw calls ``predict`` + again on the one fitted model, which advances the model's own random + stream. + """ + from microcosm.fit import NO_WEIGHTS, RegimeGatedQRF + + train = fold.X_donor.copy() + train["moop"] = fold.moop_donor + predictors = list(fold.X_donor.columns) + weights = fold.weight_donor if weighted else NO_WEIGHTS + draws = np.empty((n_draws, len(fold.X_recipient))) + with microcosm_threads(n_jobs): + fitted = RegimeGatedQRF(seed=seed).fit( + train, predictors, ["moop"], weights=weights + ) + for r in range(n_draws): + draws[r] = fitted.predict(fold.X_recipient)["moop"].to_numpy() + return draws + + MAIN_METHODS = [ "logit_cell_mean", "logit_reg_mean", @@ -360,6 +437,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", ] DECOMPOSITION_METHODS = [ "logit_threshold_hotdeck", @@ -381,6 +460,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm": "PMM within coverage groups", "qrf": "(c) QRF (microimpute)", "zi_qrf": "(c) Zero-inflated QRF (microimpute)", + "microcosm_qrf": "(c) Regime-gated QRF (Microcosm), unit weights", + "microcosm_qrf_unweighted": "(c) Regime-gated QRF (Microcosm), unweighted", "logit_threshold_hotdeck": "Logit, p >= 0.5, donor amount in cell", "logit_draw_cell_mean": "Logit draw, cell mean", "logit_expected_value": "p x cell mean for every unit", @@ -391,6 +472,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", "logit_threshold_hotdeck", "logit_draw_cell_mean", } diff --git a/experiments/moop_spm/render.py b/experiments/moop_spm/render.py index 04831c1b..bfc5020a 100644 --- a/experiments/moop_spm/render.py +++ b/experiments/moop_spm/render.py @@ -24,6 +24,8 @@ "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", ] SHORT = { "reported": "Reported", @@ -31,8 +33,10 @@ "logit_reg_mean": "(a) Logit + OLS mean", "logit_draw_hotdeck": "(b) Logit draw + donor", "pmm": "PMM", - "qrf": "(c) QRF", - "zi_qrf": "(c) ZI-QRF", + "qrf": "(c) microimpute QRF", + "zi_qrf": "(c) microimpute ZI-QRF", + "microcosm_qrf": "(c) Microcosm QRF, weighted", + "microcosm_qrf_unweighted": "(c) Microcosm QRF, unweighted", } DECOMPOSITION = [ "logit_cell_mean", @@ -339,6 +343,28 @@ def placeholder_values(results: dict) -> dict[str, str]: raise ValueError("A method (a), (b) or PMM no longer raises the subset rate.") values["subset_pp_low_ab_pmm"] = f"{100 * min(subset_ab_pmm):.2f}" values["subset_pp_high_ab_pmm"] = f"{100 * max(subset_ab_pmm):.2f}" + # The note says all four QRF fits leave the national rate within 0.1 + # point and fall short of the reported MOOP total. + forests = ["qrf", "zi_qrf", "microcosm_qrf", "microcosm_qrf_unweighted"] + if max(abs(methods[m]["diff"][0]) for m in forests) >= 0.001: + raise ValueError("A QRF fit moves the national rate by 0.1 point or more.") + totals = [methods[m]["distribution"]["aggregate_billions"] for m in forests] + if max(totals) >= reported_total: + raise ValueError("A QRF fit no longer falls short of the reported total.") + # It says four methods raise the rate of the first band above the line. + band1 = keyed["pre_moop_ratio::1.0 to 1.25"] + cited = ["logit_cell_mean", "logit_draw_hotdeck", "pmm", "microcosm_qrf"] + if min(methods[m]["diff"][band1] for m in cited) <= 0: + raise ValueError("A method the note cites no longer raises the band's rate.") + subset_forests = [methods[m]["no_allocation_subset"] for m in forests] + values["subset_diff_low_forest"] = signed(min(s["diff"] for s in subset_forests)) + values["subset_diff_high_forest"] = signed(max(s["diff"] for s in subset_forests)) + values["subset_se_low_forest"] = ( + f"{100 * min(s['diff_se'] for s in subset_forests):.2f}" + ) + values["subset_se_high_forest"] = ( + f"{100 * max(s['diff_se'] for s in subset_forests):.2f}" + ) return values diff --git a/experiments/moop_spm/results/results.json b/experiments/moop_spm/results/results.json index 0015c2c2..f2f0d9d0 100644 --- a/experiments/moop_spm/results/results.json +++ b/experiments/moop_spm/results/results.json @@ -1356,6 +1356,236 @@ }, "n_draws": 10 }, + "microcosm_qrf": { + "rate": [ + 0.13176597711983554, + 0.13526968715039864, + 0.12350002246414134, + 0.15340594010547154, + 0.2248051223160781, + 0.11747645511385332, + 0.13305603599806434, + 0.16518960209642802, + 0.039906728623815765, + 0.18932364377891436, + 0.3223734578052057, + 0.06627867410088427, + 0.25705105851270915, + 1.0, + 0.25319871622296414, + 0.05838029863988854, + 0.016825135077092105, + 0.0036201119433489437, + 0.000798666807527483 + ], + "rate_se": [ + 0.0017722283598928515, + 0.003213551830044547, + 0.0019635171995840786, + 0.003145575335591901, + 0.004570227562425885, + 0.0017890591770403663, + 0.0034203544079126503, + 0.003258369600304733, + 0.0012402801655167996, + 0.007340907278622133, + 0.006704386028836305, + 0.008397985771672481, + 0.0069289837720845505, + 0.0, + 0.0064046245896981525, + 0.002575172671985775, + 0.0008808686908466718, + 0.00036387923913772964, + 0.00012341934682562108 + ], + "diff": [ + 0.0006492289734812418, + 0.001356539466790374, + 0.0007271289585814439, + -0.00037309587107262066, + -0.001697528458331904, + 0.0008522919105011101, + 0.0012577162752293591, + -0.0003615060214874899, + 0.0006332811849191072, + 0.006294465433623164, + 0.0012330146010093856, + -0.005308201685261957, + -0.0004650027951850211, + 0.0, + 0.004150763059049073, + 0.0003451013307301451, + 0.0012113353767259135, + 0.0004145096552255717, + 0.00029358445461573086 + ], + "diff_se": [ + 0.0006592651961442288, + 0.0010223477206047426, + 0.0007442494544711315, + 0.0013388474853659532, + 0.0018367416554727917, + 0.0006962035984257459, + 0.0011316013507247148, + 0.001399497208018087, + 0.0007312309884633065, + 0.0036334331101260772, + 0.001715176458800662, + 0.0034876761358560294, + 0.002348819525701735, + 0.0, + 0.007528622170789294, + 0.004931367011484733, + 0.0016992785907958133, + 0.0007953055936745014, + 0.00021537566922358558 + ], + "newly_poor_share": 0.0147271338655482, + "newly_nonpoor_share": 0.014077904892067034, + "overall_draw_sd": 0.00046779452762348413, + "distribution": { + "mean": 5948.790858172932, + "share_zero": 0.06368620791710222, + "aggregate_billions": 850.0712999217154, + "near_line_mean": 3671.0686281868802, + "median": 3601.6532323149245, + "p75": 8040.322086535487, + "p90": 14284.542837359739, + "p95": 19501.498348512174, + "p99": 33748.81173189083, + "band_means": [ + 1453.6003032561512, + 2434.0878138343346, + 3431.6513280482445, + 4473.238904220103, + 6070.872709541648, + 8641.984509555059 + ], + "mean_draw_sd": 37.017322597518294 + }, + "no_allocation_subset": { + "rate": 0.1202665144927313, + "diff": 0.001378939779788993, + "diff_se": 0.000939223230646948 + }, + "n_draws": 10 + }, + "microcosm_qrf_unweighted": { + "rate": [ + 0.13106692164163733, + 0.13415258046530917, + 0.12266759186609895, + 0.1535806780580723, + 0.22420695950786557, + 0.11683665432179541, + 0.13205388625435582, + 0.165254917347693, + 0.0393048547945922, + 0.18510263194184223, + 0.32149653668439665, + 0.06989252912718669, + 0.2557872081718448, + 1.0, + 0.24774896156061657, + 0.05572065656492517, + 0.015926979409537096, + 0.0030581291317636405, + 0.0009537681944727924 + ], + "rate_se": [ + 0.0017695074323104496, + 0.003213453284334693, + 0.0019564415761225773, + 0.0031537878382792957, + 0.004652394622314318, + 0.0017813163206770075, + 0.003432591884034214, + 0.0032725193303407593, + 0.0012324462825733747, + 0.007340458726984088, + 0.0067616772408449904, + 0.008757042095872686, + 0.006905335606267363, + 0.0, + 0.006152360771847005, + 0.0022250301228301096, + 0.0007600690732998304, + 0.00030587397587153307, + 0.0001374811138293 + ], + "diff": [ + -4.982650471696548e-05, + 0.0002394327817009101, + -0.00010530163946094462, + -0.00019835791847186424, + -0.0022956912665444418, + 0.0002124911184431949, + 0.00025556653152083997, + -0.0002961907702225264, + 3.140735569553965e-05, + 0.0020734535965510337, + 0.0003560934802003257, + -0.001694346658959539, + -0.0017288531360493686, + 0.0, + -0.0012989916032984938, + -0.0023145407442332275, + 0.00031317970917090453, + -0.00014747315635973155, + 0.00044868584156104025 + ], + "diff_se": [ + 0.0006483601963995772, + 0.0010427416199333315, + 0.0007169516960344145, + 0.0013375249925742245, + 0.001736854270344326, + 0.0006897482765244226, + 0.0011607888236397864, + 0.0013996269222166927, + 0.0007072886785143316, + 0.003482219843123647, + 0.0017802852979193254, + 0.0029699212851560337, + 0.0024659140628588844, + 0.0, + 0.00773088623665069, + 0.004935477370410093, + 0.0015855809293758682, + 0.000754633029987932, + 0.00021749338188776913 + ], + "newly_poor_share": 0.014148643352674075, + "newly_nonpoor_share": 0.014198469857391266, + "overall_draw_sd": 0.0004589807548556485, + "distribution": { + "mean": 5891.13011170698, + "share_zero": 0.0626552384039673, + "aggregate_billions": 841.8316850367131, + "near_line_mean": 3614.174309373083, + "median": 3516.8740708910273, + "p75": 7981.77211733802, + "p90": 14139.202845924421, + "p95": 19092.281833557445, + "p99": 32871.98515295489, + "band_means": [ + 1423.7467982065887, + 2389.264924735185, + 3343.7693548217867, + 4427.242685978768, + 5928.645050553944, + 8617.923648416621 + ], + "mean_draw_sd": 65.8135035294827 + }, + "no_allocation_subset": { + "rate": 0.11937030819262767, + "diff": 0.00048273347968535996, + "diff_se": 0.000879829562190696 + }, + "n_draws": 10 + }, "no_moop": { "rate": [ 0.10859685755998726, @@ -1649,11 +1879,16 @@ ], "source_sha256": { "cps_asec.py": "647857a0fa249c44", - "methods.py": "8ce97d8ddd38df23", + "methods.py": "8001218c33952fca", "evaluate.py": "09daf780b35188ad", - "run.py": "324dad8ce5a851e0" + "run.py": "71be1ff2744734c3" }, "microimpute_sha256": "e14c884a85e90486", + "microcosm_sha256": { + "microcosm.fit": "797ad4f288a8c2e5", + "microcosm.frame": "a5630b096c4ba177", + "microcosm.graph": "ee26504154cb597e" + }, "environment": { "python": "3.13.13", "numpy": "2.5.4", @@ -1661,17 +1896,22 @@ "scipy": "1.18.1", "scikit_learn": "1.8.0", "quantile_forest": "1.4.2", - "microimpute": "3.1.4" + "microimpute": "3.2.0", + "microcosm_fit": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_frame": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" }, "input_sha256": "5a129f5d767fff81", - "tag": "a82da82512c894e6", + "tag": "880f00af5a1da5d8", "n_jobs": 4 }, "timings_seconds": { - "logit_family": 38.13895297050476, - "pmm": 0.5446643829345703, - "qrf": 68.79297137260437, - "zi_qrf": 185.8473162651062 + "logit_family": 28.513712406158447, + "pmm": 1.0840468406677246, + "qrf": 117.33195805549622, + "zi_qrf": 305.8486762046814, + "microcosm_qrf": 142.04153990745544, + "microcosm_qrf_unweighted": 188.22922801971436 }, "environment": { "python": "3.13.13", @@ -1680,6 +1920,9 @@ "scipy": "1.18.1", "scikit_learn": "1.8.0", "quantile_forest": "1.4.2", - "microimpute": "3.1.4" + "microimpute": "3.2.0", + "microcosm_fit": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_frame": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" } } diff --git a/experiments/moop_spm/run.py b/experiments/moop_spm/run.py index d910e96b..3ca56915 100644 --- a/experiments/moop_spm/run.py +++ b/experiments/moop_spm/run.py @@ -12,9 +12,11 @@ The first run downloads the Census public-use zip (139 MB) into ``--data-dir`` and checks its SHA-256. Per-fold imputations are cached in ``/cache`` so an interrupted run resumes. The cache key covers the -settings, the experiment's source files, every microimpute source file, -the versions of Python and the numerical packages, and the model inputs -with their row keys, so a change to any of them refits every fold. +settings, the experiment's source files, every source file of microimpute +and of Microcosm's ``fit``, ``frame`` and ``graph`` packages, the versions +of Python and the numerical packages, the commit Microcosm came from, and +the model inputs with their row keys, so a change to any of them refits +every fold. """ from __future__ import annotations @@ -26,7 +28,7 @@ import platform import time from collections.abc import Callable -from importlib.metadata import version +from importlib.metadata import PackageNotFoundError, distribution, version from pathlib import Path import numpy as np @@ -54,6 +56,7 @@ FoldData, cell_keys, logit_family, + microcosm_qrf, pmm, qrf, zi_qrf, @@ -70,6 +73,14 @@ "quantile_forest": "quantile-forest", "microimpute": "microimpute", } +# Microcosm's distributions and the packages they install. They install +# from git, and their version numbers do not change from commit to +# commit, so we record the commit and hash the source. +MICROCOSM_PACKAGES = { + "microcosm_fit": ("microcosm-fit", "microcosm.fit"), + "microcosm_frame": ("microcosm-frame", "microcosm.frame"), + "microcosm_graph": ("microcosm-graph", "microcosm.graph"), +} PUBLISHED = json.loads((HERE / "published_2025.json").read_text()) @@ -85,9 +96,35 @@ def config_hash(config: dict) -> str: return hashlib.sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()[:16] -def environment() -> dict[str, str]: - """Return the Python version and the versions in ``PACKAGES``.""" - versions = {name: version(dist) for name, dist in PACKAGES.items()} +def installed_source(dist: str) -> str | None: + """Return a distribution's version and source, or None if it is absent. + + For an install from git, the source is the commit that pip or uv + recorded in the distribution's ``direct_url.json`` (PEP 610). + """ + try: + found = distribution(dist) + except PackageNotFoundError: + return None + text = found.read_text("direct_url.json") + if not text: + return found.version + direct = json.loads(text) + source = direct.get("vcs_info", {}).get("commit_id") or direct.get("url") + return f"{found.version} @ {source}" + + +def environment() -> dict[str, str | None]: + """Return the versions of Python, ``PACKAGES`` and Microcosm. + + Microcosm's entries carry the commit it came from, and they are None + when it is not installed. + """ + versions: dict[str, str | None] = { + name: version(dist) for name, dist in PACKAGES.items() + } + for name, (dist, _) in MICROCOSM_PACKAGES.items(): + versions[name] = installed_source(dist) return {"python": platform.python_version(), **versions} @@ -104,6 +141,31 @@ def package_sha256(package: str) -> str: return digest.hexdigest()[:16] +def installed_package_sha256(package: str) -> str | None: + """Return ``package_sha256``, or None when the package is not installed. + + An installed package that fails to import raises as usual. + """ + try: + return package_sha256(package) + except ModuleNotFoundError as error: + parts = package.split(".") + parents = {".".join(parts[:i]) for i in range(1, len(parts) + 1)} + if error.name in parents: + return None + raise + + +def require_microcosm() -> None: + """Fail before any fitting when Microcosm's ``fit`` package is absent.""" + if installed_source("microcosm-fit") is None: + raise SystemExit( + "The Microcosm methods need microcosm-fit, which installs from " + "git (see README.md). Pass --skip-forests to run without the " + "forest methods." + ) + + def frame_sha256(*frames: pd.DataFrame) -> str: """Hash frames' column names, row keys and values.""" digest = hashlib.sha256() @@ -137,6 +199,10 @@ def build_config( "cell_levels": [list(level) for level in CELL_LEVELS], "source_sha256": {name: sha256_of(HERE / name)[:16] for name in SOURCES}, "microimpute_sha256": package_sha256("microimpute"), + "microcosm_sha256": { + module: installed_package_sha256(module) + for _, module in MICROCOSM_PACKAGES.values() + }, "environment": environment(), "input_sha256": frame_sha256(units, X, cells, fold_frame), } @@ -194,6 +260,7 @@ def run_fold( data = FoldData( X_donor=X.loc[donor], moop_donor=units["moop"].to_numpy()[donor], + weight_donor=units["unit_weight"].to_numpy()[donor], cells_donor=cells.loc[donor], group_donor=units["group"].to_numpy()[donor], X_recipient=X.loc[recipient], @@ -220,6 +287,22 @@ def run_fold( start = time.time() out["zi_qrf"] = zi_qrf(data, args.draws, seed=args.seed + fold) timings["zi_qrf"] = time.time() - start + + # Both Microcosm fits share a seed, so their draws use the same + # uniforms and differ only through the weights. + for method, weighted in ( + ("microcosm_qrf", True), + ("microcosm_qrf_unweighted", False), + ): + start = time.time() + out[method] = microcosm_qrf( + data, + args.draws, + seed=args.seed + fold, + weighted=weighted, + n_jobs=args.n_jobs, + ) + timings[method] = time.time() - start out["_timings"] = np.array([list(timings.values())]) out["_timing_names"] = np.array(list(timings.keys())) return out @@ -236,7 +319,8 @@ def main() -> None: parser.add_argument( "--skip-forests", action="store_true", - help="Skip QRF and ZI-QRF (for quick checks).", + help="Skip the forest methods, microimpute's and Microcosm's " + "(for quick checks).", ) parser.add_argument( "--max-households", @@ -245,6 +329,8 @@ def main() -> None: help="Subsample households (for quick checks).", ) args = parser.parse_args() + if not args.skip_forests: + require_microcosm() output = args.output cache = output / "cache" diff --git a/tests/test_moop_spm_experiment.py b/tests/test_moop_spm_experiment.py index 4063c4d4..12f82e58 100644 --- a/tests/test_moop_spm_experiment.py +++ b/tests/test_moop_spm_experiment.py @@ -15,12 +15,17 @@ - The logistic-participation variants follow their stated rules. - Census's MOOP and Medicare premium imputations both count as allocated. - Extracted CSVs that differ from the pinned zip are extracted again. -- The fold cache misses whenever the environment, microimpute's source - or the model inputs change. +- The fold cache misses whenever the environment, microimpute's or + Microcosm's source or the model inputs change. +- Microcosm's draws are zero or lie within the range of the spenders + that carry weight, and they do not depend on the number of threads. + These tests skip when Microcosm is not installed; it installs from git. """ import argparse +import dataclasses import os +import re import zipfile import numpy as np @@ -48,6 +53,8 @@ assign_cells, fit_participation, logit_family, + microcosm_qrf, + microcosm_threads, nearest_donors, participation_probability, ) @@ -57,6 +64,8 @@ collect_folds, config_hash, frame_sha256, + installed_package_sha256, + installed_source, ) SETTINGS = settings( @@ -280,6 +289,7 @@ def features(n): return FoldData( X_donor=X_donor, moop_donor=moop, + weight_donor=rng.uniform(100, 3_000, n_donor), cells_donor=cells_donor, group_donor=cells_donor["group"].to_numpy(), X_recipient=features(n_recipient), @@ -315,6 +325,68 @@ def test_logit_family_rules(seed): assert np.all((drawn == 0) | np.isclose(drawn, means[i])) +@settings(max_examples=3, deadline=None) +@given(seed=st.integers(0, 2**31), cap_quantile=st.floats(0.3, 0.9)) +def test_microcosm_draws_stay_within_weighted_spenders(seed, cap_quantile): + pytest.importorskip("microcosm.fit") + fold = _fold(seed) + n_recipients = len(fold.X_recipient) + spent = fold.moop_donor[fold.moop_donor > 0] + + unweighted = microcosm_qrf(fold, 3, seed=seed, weighted=False, n_jobs=1) + assert unweighted.shape == (3, n_recipients) + assert np.isfinite(unweighted).all() + # A draw is zero or lies between the smallest and largest spender. + drawn = unweighted[unweighted != 0] + assert (drawn >= spent.min()).all() + assert (drawn <= spent.max()).all() + assert not np.array_equal(unweighted[0], unweighted[1]) + # The thread count changes no draw. + again = microcosm_qrf(fold, 3, seed=seed, weighted=False, n_jobs=2) + assert np.array_equal(again, unweighted) + + # Donors without weight shape no draw: with zero weight on every + # spender above the cap, no weighted draw exceeds it. + cap = float(np.quantile(spent, cap_quantile)) + capped = dataclasses.replace( + fold, + weight_donor=np.where(fold.moop_donor > cap, 0.0, fold.weight_donor), + ) + weighted = microcosm_qrf(capped, 3, seed=seed, weighted=True, n_jobs=1) + assert weighted.shape == (3, n_recipients) + assert (weighted >= 0).all() + assert weighted.max() <= cap + + +def test_microcosm_threads_sets_and_restores_widths(monkeypatch): + qrf_module = pytest.importorskip("microcosm.fit.qrf") + fit_name, draw_name = qrf_module._N_JOBS_ENV, qrf_module._PREDICT_WORKERS_ENV + monkeypatch.delenv(fit_name, raising=False) + monkeypatch.setenv(draw_name, "7") + with microcosm_threads(3): + # Microcosm's own resolvers read the cap. + assert qrf_module._fit_n_jobs() == 3 + assert qrf_module._predict_workers() == 3 + assert fit_name not in os.environ + assert os.environ[draw_name] == "7" + + +def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monkeypatch): + assert installed_source("no-such-distribution-moop") is None + assert installed_package_sha256("no_such_package_moop.sub") is None + # A package that is present but fails to import still raises. + broken = tmp_path / "broken_package_moop" + broken.mkdir() + (broken / "__init__.py").write_text("import no_such_dependency_moop\n") + monkeypatch.syspath_prepend(str(tmp_path)) + with pytest.raises(ModuleNotFoundError): + installed_package_sha256("broken_package_moop") + source = installed_source("microcosm-fit") + if source is not None: + # A git install records its commit. + assert re.fullmatch(r"\S+ @ ([0-9a-f]{40}|\S+)", source) + + @settings(max_examples=10, deadline=None) @given(seed=st.integers(0, 2**31)) def test_evaluate_invariants(seed): @@ -426,7 +498,7 @@ def fit(fold: int) -> dict[str, np.ndarray]: return fit -def test_cache_misses_when_environment_or_microimpute_changes(tmp_path, monkeypatch): +def test_cache_misses_when_environment_or_model_source_changes(tmp_path, monkeypatch): args, units, X, cells, folds = _cache_inputs() base_env = run_module.environment() monkeypatch.setattr(run_module, "environment", lambda: base_env) @@ -454,7 +526,21 @@ def test_cache_misses_when_environment_or_microimpute_changes(tmp_path, monkeypa assert calls == [0, 1, 2] assert (fresh["qrf"] == 99.0).all() + # Microcosm from another commit refits. + moved = {**base_env, "microcosm_fit": "0.1.0 @ another-commit"} + monkeypatch.setattr(run_module, "environment", lambda: moved) + assert config_hash(build_config(args, units, X, cells, folds)) != tag + monkeypatch.setattr(run_module, "environment", lambda: base_env) + assert config_hash(build_config(args, units, X, cells, folds)) == tag + # Edited Microcosm source refits, and so does edited microimpute source. + original = run_module.installed_package_sha256 + monkeypatch.setattr( + run_module, "installed_package_sha256", lambda package: "edited" + ) + assert config_hash(build_config(args, units, X, cells, folds)) != tag + monkeypatch.setattr(run_module, "installed_package_sha256", original) + assert config_hash(build_config(args, units, X, cells, folds)) == tag monkeypatch.setattr(run_module, "package_sha256", lambda package: "edited") assert config_hash(build_config(args, units, X, cells, folds)) != tag From e8030a59e026a9a618aec8ee6dee3139a6ea6fa6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 7 Nov 2026 11:02:57 -0500 Subject: [PATCH 2/5] Answer PR 223's review: fix README claims, test the gate's weights - README: describe microimpute's draw grid as its code works (it rounds each quantile q to k/100 and reads the forest's (k + 1)/102 quantile), and limit Microcosm's full leaf retention to targets with more than 32 distinct values; it caps leaves at 4,096 samples for 2 to 32. - methods.py: Microcosm's fit asks for design weights by default; a DataFrame fit must name its weights or pass NO_WEIGHTS. - run.py: installed_source reads the version through metadata.get and raises when the metadata names none. - tests: spy on the gate's fit to check each donor's weight in donor order, and add a versionless-metadata case. - results.json: a full rerun under cache tag 541e509d20712573 changed only the two source hashes, the tag and six timings. Co-Authored-By: Claude Opus 5.5 --- experiments/moop_spm/README.md | 15 ++++--- experiments/moop_spm/methods.py | 5 ++- experiments/moop_spm/results/results.json | 18 ++++---- experiments/moop_spm/run.py | 13 ++++-- tests/test_moop_spm_experiment.py | 50 +++++++++++++++++++++++ 5 files changed, 81 insertions(+), 20 deletions(-) diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index 43c584ab..79268106 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -76,8 +76,8 @@ We fit every method on donor units and impute held-out recipient units Microcosm (`PolicyEngine/microcosm`) builds PolicyEngine's microdata, and its `microcosm-fit` package imputes with `RegimeGatedQRF`. We fit it -with its defaults (100 trees, every leaf sample kept, no cross-fitting -of the gate), once with each donor unit's SPM weight (`SPM_WEIGHT / 100`) and +with its defaults (100 trees, `max_samples_leaf=None` and no +cross-fitting of the gate), once with each donor unit's SPM weight (`SPM_WEIGHT / 100`) and once without weights. The weighted fit passes the weights to the gate as sample weights and grows the forest on spenders resampled in proportion to them. Both fits share a seed, so their draws use the same random @@ -86,12 +86,15 @@ numbers. Beyond the weights, Microcosm's model differs from microimpute's - Its gate draws whether a unit spends. microimpute's `QRF` has no gate, and `ZeroInflatedImputer` adds one of its own. -- It keeps every training sample in each leaf, where microimpute keeps - quantile-forest's default of at most one. +- For a target with more than 32 distinct values, such as MOOP, it + keeps every training sample in each leaf; it caps leaves at 4,096 + samples only when a target takes 2 to 32 distinct values. microimpute + keeps quantile-forest's default of at most one sample per leaf. - It reads each unit's draw at that unit's own quantile, interpolating a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute - rounds the quantile to the nearest of 101 points from 1/102 to - 101/102. + rounds each unit's quantile q to the nearest hundredth, k/100, and + reads the forest's (k + 1)/102 quantile, so its draws come from 101 + points that run from 1/102 to 101/102 (q = 0.01 reads 2/102). We form cells from the unit's coverage group (Medicare, employer, direct purchase, public, uninsured), the head's age band, cash income relative diff --git a/experiments/moop_spm/methods.py b/experiments/moop_spm/methods.py index cba52e78..ae9b6629 100644 --- a/experiments/moop_spm/methods.py +++ b/experiments/moop_spm/methods.py @@ -23,8 +23,9 @@ - ``zi_qrf``: microimpute's ``ZeroInflatedImputer`` around ``QRF``. - ``microcosm_qrf``: Microcosm's ``RegimeGatedQRF``, the model that imputes PolicyEngine's microdata, fit with the donors' unit weights. - On Microcosm's own data frames, a fit uses the frame's weights unless - told not to. + Microcosm's ``fit`` asks for design weights by default and raises when a + frame carries another kind; a DataFrame fit must name its weights or + pass ``NO_WEIGHTS``. - ``microcosm_qrf_unweighted``: the same model fit without weights, like every other method here. diff --git a/experiments/moop_spm/results/results.json b/experiments/moop_spm/results/results.json index f2f0d9d0..1e574c90 100644 --- a/experiments/moop_spm/results/results.json +++ b/experiments/moop_spm/results/results.json @@ -1879,9 +1879,9 @@ ], "source_sha256": { "cps_asec.py": "647857a0fa249c44", - "methods.py": "8001218c33952fca", + "methods.py": "9fbe9b3acb046eb6", "evaluate.py": "09daf780b35188ad", - "run.py": "71be1ff2744734c3" + "run.py": "2a9465d844ddcb83" }, "microimpute_sha256": "e14c884a85e90486", "microcosm_sha256": { @@ -1902,16 +1902,16 @@ "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" }, "input_sha256": "5a129f5d767fff81", - "tag": "880f00af5a1da5d8", + "tag": "541e509d20712573", "n_jobs": 4 }, "timings_seconds": { - "logit_family": 28.513712406158447, - "pmm": 1.0840468406677246, - "qrf": 117.33195805549622, - "zi_qrf": 305.8486762046814, - "microcosm_qrf": 142.04153990745544, - "microcosm_qrf_unweighted": 188.22922801971436 + "logit_family": 76.79685592651367, + "pmm": 2.1582870483398438, + "qrf": 310.7086248397827, + "zi_qrf": 693.6227560043335, + "microcosm_qrf": 346.5654013156891, + "microcosm_qrf_unweighted": 369.52848386764526 }, "environment": { "python": "3.13.13", diff --git a/experiments/moop_spm/run.py b/experiments/moop_spm/run.py index 3ca56915..89258353 100644 --- a/experiments/moop_spm/run.py +++ b/experiments/moop_spm/run.py @@ -100,18 +100,25 @@ def installed_source(dist: str) -> str | None: """Return a distribution's version and source, or None if it is absent. For an install from git, the source is the commit that pip or uv - recorded in the distribution's ``direct_url.json`` (PEP 610). + recorded in the distribution's ``direct_url.json`` (PEP 610). Metadata + that names no version raises, so a damaged install never reads as an + absent one. """ try: found = distribution(dist) except PackageNotFoundError: return None + # ``metadata.get`` reads a missing field as None without the warning, + # and later KeyError, that ``Distribution.version`` gives. + found_version = found.metadata.get("Version") + if not found_version: + raise RuntimeError(f"{dist}'s distribution metadata names no version.") text = found.read_text("direct_url.json") if not text: - return found.version + return found_version direct = json.loads(text) source = direct.get("vcs_info", {}).get("commit_id") or direct.get("url") - return f"{found.version} @ {source}" + return f"{found_version} @ {source}" def environment() -> dict[str, str | None]: diff --git a/tests/test_moop_spm_experiment.py b/tests/test_moop_spm_experiment.py index 12f82e58..403cdd7a 100644 --- a/tests/test_moop_spm_experiment.py +++ b/tests/test_moop_spm_experiment.py @@ -19,11 +19,16 @@ Microcosm's source or the model inputs change. - Microcosm's draws are zero or lie within the range of the spenders that carry weight, and they do not depend on the number of threads. +- Microcosm's gate takes each donor's unit weight, in donor order, and + takes no weights when the fit is unweighted. +- Distribution metadata that names no version raises rather than reading + as an absent package. These tests skip when Microcosm is not installed; it installs from git. """ import argparse import dataclasses +import importlib import os import re import zipfile @@ -371,6 +376,42 @@ def test_microcosm_threads_sets_and_restores_widths(monkeypatch): assert os.environ[draw_name] == "7" +def test_microcosm_gate_takes_donor_weights_in_order(monkeypatch): + pytest.importorskip("microcosm.fit") + from sklearn.ensemble import HistGradientBoostingClassifier + + recorded = [] + original = HistGradientBoostingClassifier.fit + + def spy(self, X, y, sample_weight=None, **kwargs): + recorded.append( + None if sample_weight is None else np.array(sample_weight, dtype=float) + ) + return original(self, X, y, sample_weight=sample_weight, **kwargs) + + monkeypatch.setattr(HistGradientBoostingClassifier, "fit", spy) + fold = _fold(7) + microcosm_qrf(fold, 1, seed=7, weighted=True, n_jobs=1) + # One in-sample gate, fitted on every donor with that donor's weight. + assert len(recorded) == 1 + weights = recorded[0] + assert weights is not None + assert weights.shape == fold.weight_donor.shape + assert np.allclose( + weights / weights.sum(), fold.weight_donor / fold.weight_donor.sum() + ) + # A reversed donor order would fail the same check. + assert not np.allclose( + weights / weights.sum(), + fold.weight_donor[::-1] / fold.weight_donor.sum(), + ) + + recorded.clear() + microcosm_qrf(fold, 1, seed=7, weighted=False, n_jobs=1) + assert len(recorded) == 1 + assert recorded[0] is None + + def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monkeypatch): assert installed_source("no-such-distribution-moop") is None assert installed_package_sha256("no_such_package_moop.sub") is None @@ -378,9 +419,18 @@ def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monke broken = tmp_path / "broken_package_moop" broken.mkdir() (broken / "__init__.py").write_text("import no_such_dependency_moop\n") + # So does a distribution whose metadata names no version. + versionless = tmp_path / "versionless_moop-0.0.dist-info" + versionless.mkdir() + (versionless / "METADATA").write_text( + "Metadata-Version: 2.1\nName: versionless-moop\n" + ) monkeypatch.syspath_prepend(str(tmp_path)) + importlib.invalidate_caches() with pytest.raises(ModuleNotFoundError): installed_package_sha256("broken_package_moop") + with pytest.raises(RuntimeError, match="names no version"): + installed_source("versionless-moop") source = installed_source("microcosm-fit") if source is not None: # A git install records its commit. From 959ab7dbe3da92f7df8de774064dca0352ce0bfa Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 7 Nov 2026 17:44:02 -0500 Subject: [PATCH 3/5] Answer PR 223's second review: leaf rule, weight forms, row checks - README: Microcosm caps a forest's leaves at 4,096 samples only when the donors that forest trains on hold 2 to 32 distinct values. MOOP's forest trains on the donor spenders, whose amounts take 7,830 to 7,880 distinct values in each fold. - methods.py: a caller fitting a DataFrame must pass a weight column's name, a weight vector or NO_WEIGHTS. - run.py: environment() raises when a Microcosm package imports without distribution metadata, and a new test covers that without Microcosm. - tests: the gate spy checks each fit's features and labels against the donors' rows as well as the weights, and the module docstring's skip note moves out of the bullet list. - Reword participial passives in the README, the note template, the changelog fragment and the docstrings. - A full rerun refit every fold under cache tag 6ca09058c0101fce. In results.json only the two source hashes, the tag and six timings changed; TABLE.md renders unchanged, and NOTE.md changes only in the reworded template phrase. Co-Authored-By: Claude Opus 5.5 --- changelog.d/moop-spm-microcosm-qrf.changed.md | 2 +- experiments/moop_spm/NOTE.md | 2 +- experiments/moop_spm/NOTE.template.md | 2 +- experiments/moop_spm/README.md | 25 ++++---- experiments/moop_spm/methods.py | 28 ++++----- experiments/moop_spm/results/results.json | 18 +++--- experiments/moop_spm/run.py | 30 ++++++--- tests/test_moop_spm_experiment.py | 61 +++++++++++++++---- 8 files changed, 110 insertions(+), 58 deletions(-) diff --git a/changelog.d/moop-spm-microcosm-qrf.changed.md b/changelog.d/moop-spm-microcosm-qrf.changed.md index 212af4de..64cf995b 100644 --- a/changelog.d/moop-spm-microcosm-qrf.changed.md +++ b/changelog.d/moop-spm-microcosm-qrf.changed.md @@ -1 +1 @@ -Add Microcosm's regime-gated QRF, fit with and without the ASEC unit weights, to the MOOP imputation experiment (`experiments/moop_spm`). +Add Microcosm's regime-gated QRF to the MOOP imputation experiment (`experiments/moop_spm`), which fits it with and without the ASEC unit weights. diff --git a/experiments/moop_spm/NOTE.md b/experiments/moop_spm/NOTE.md index 0da5a114..05aa8cc4 100644 --- a/experiments/moop_spm/NOTE.md +++ b/experiments/moop_spm/NOTE.md @@ -10,7 +10,7 @@ We limit every method to predictors an ACS-style file carries: members' ages, se - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest grown on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest it grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results diff --git a/experiments/moop_spm/NOTE.template.md b/experiments/moop_spm/NOTE.template.md index 5a795b25..69051c5b 100644 --- a/experiments/moop_spm/NOTE.template.md +++ b/experiments/moop_spm/NOTE.template.md @@ -10,7 +10,7 @@ We limit every method to predictors an ACS-style file carries: members' ages, se - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest grown on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest it grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index 79268106..450b3576 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -71,25 +71,28 @@ We fit every method on donor units and impute held-out recipient units | `pmm` | implicit | predictive mean matching on log(1 + MOOP) within five coverage groups, random one of 5 nearest donors | | (c) `qrf` | implicit | microimpute `QRF`, one random quantile per unit | | (c) `zi_qrf` | microimpute gate | microimpute `ZeroInflatedImputer` around `QRF` | -| (c) `microcosm_qrf` | Microcosm gate, a gradient-boosted classifier | Microcosm `RegimeGatedQRF` forest grown on donor spenders, one random quantile per unit; gate and forest use the SPM unit weights | +| (c) `microcosm_qrf` | Microcosm gate, a gradient-boosted classifier | Microcosm `RegimeGatedQRF` forest on donor spenders, one random quantile per unit; gate and forest use the SPM unit weights | | (c) `microcosm_qrf_unweighted` | same, without weights | same, without weights | Microcosm (`PolicyEngine/microcosm`) builds PolicyEngine's microdata, and its `microcosm-fit` package imputes with `RegimeGatedQRF`. We fit it with its defaults (100 trees, `max_samples_leaf=None` and no -cross-fitting of the gate), once with each donor unit's SPM weight (`SPM_WEIGHT / 100`) and -once without weights. The weighted fit passes the weights to the gate as -sample weights and grows the forest on spenders resampled in proportion -to them. Both fits share a seed, so their draws use the same random -numbers. Beyond the weights, Microcosm's model differs from microimpute's -`QRF` in three ways: +cross-fitting of the gate), once with each donor unit's SPM weight +(`SPM_WEIGHT / 100`) and once without weights. With the weights, +Microcosm passes them to the gate as sample weights and resamples +spenders in proportion to them before it grows the forest. Both fits +share a seed, so their draws use the same random numbers. Beyond the +weights, Microcosm's model differs from microimpute's `QRF` in three +ways: - Its gate draws whether a unit spends. microimpute's `QRF` has no gate, and `ZeroInflatedImputer` adds one of its own. -- For a target with more than 32 distinct values, such as MOOP, it - keeps every training sample in each leaf; it caps leaves at 4,096 - samples only when a target takes 2 to 32 distinct values. microimpute - keeps quantile-forest's default of at most one sample per leaf. +- It keeps every training sample in each leaf of its MOOP forest, where + microimpute keeps quantile-forest's default of at most one sample per + leaf. Microcosm caps a forest's leaves at 4,096 samples only when the + donors that forest trains on hold 2 to 32 distinct values. Its MOOP + forest trains on the donor spenders, whose amounts take 7,830 to 7,880 + distinct values in each fold. - It reads each unit's draw at that unit's own quantile, interpolating a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute rounds each unit's quantile q to the nearest hundredth, k/100, and diff --git a/experiments/moop_spm/methods.py b/experiments/moop_spm/methods.py index ae9b6629..966482c3 100644 --- a/experiments/moop_spm/methods.py +++ b/experiments/moop_spm/methods.py @@ -22,11 +22,11 @@ draw per recipient. - ``zi_qrf``: microimpute's ``ZeroInflatedImputer`` around ``QRF``. - ``microcosm_qrf``: Microcosm's ``RegimeGatedQRF``, the model that - imputes PolicyEngine's microdata, fit with the donors' unit weights. + imputes PolicyEngine's microdata, with the donors' unit weights. Microcosm's ``fit`` asks for design weights by default and raises when a - frame carries another kind; a DataFrame fit must name its weights or - pass ``NO_WEIGHTS``. -- ``microcosm_qrf_unweighted``: the same model fit without weights, like + frame carries another kind; a caller fitting a DataFrame must pass a + weight column's name, a weight vector or ``NO_WEIGHTS``. +- ``microcosm_qrf_unweighted``: the same model without weights, like every other method here. Decomposition variants cross the participation rule (threshold or @@ -372,8 +372,8 @@ def microcosm_threads(n_jobs: int) -> Iterator[None]: """Cap Microcosm's threads at ``n_jobs``. Microcosm reads its forest-fitting and drawing widths from environment - variables that its ``qrf`` module names; when they are unset, it fits - and draws on every core. Its gate, a scikit-learn gradient-boosted + variables that its ``qrf`` module names, and without them it fits and + draws on every core. Its gate, a scikit-learn gradient-boosted classifier, runs on OpenMP threads, which ``threadpoolctl`` caps. The previous settings come back on exit. """ @@ -406,14 +406,14 @@ def microcosm_qrf( ) -> np.ndarray: """Microcosm's ``RegimeGatedQRF`` on all donors. - A gradient-boosted classifier gates zero against positive MOOP, and a - quantile regression forest grown on donor spenders gives each gated - spender the amount at a random quantile of its predicted distribution. - With ``weighted``, the gate takes the donors' unit weights as sample - weights and the forest grows on spenders resampled in proportion to - them; without it, both ignore the weights. Each draw calls ``predict`` - again on the one fitted model, which advances the model's own random - stream. + A gradient-boosted classifier draws whether each recipient spends, and + a quantile regression forest that Microcosm grows on donor spenders + gives each spender the amount at a random quantile of its predicted + distribution. With ``weighted``, the gate takes the donors' unit + weights as sample weights, and Microcosm resamples spenders in + proportion to them before it grows the forest; without it, neither the + gate nor the forest uses the weights. Each draw calls ``predict`` again + on the one fitted model, which advances the model's own random stream. """ from microcosm.fit import NO_WEIGHTS, RegimeGatedQRF diff --git a/experiments/moop_spm/results/results.json b/experiments/moop_spm/results/results.json index 1e574c90..e687f82e 100644 --- a/experiments/moop_spm/results/results.json +++ b/experiments/moop_spm/results/results.json @@ -1879,9 +1879,9 @@ ], "source_sha256": { "cps_asec.py": "647857a0fa249c44", - "methods.py": "9fbe9b3acb046eb6", + "methods.py": "4a36486e83ad2f88", "evaluate.py": "09daf780b35188ad", - "run.py": "2a9465d844ddcb83" + "run.py": "87dbb0975d32e8de" }, "microimpute_sha256": "e14c884a85e90486", "microcosm_sha256": { @@ -1902,16 +1902,16 @@ "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" }, "input_sha256": "5a129f5d767fff81", - "tag": "541e509d20712573", + "tag": "6ca09058c0101fce", "n_jobs": 4 }, "timings_seconds": { - "logit_family": 76.79685592651367, - "pmm": 2.1582870483398438, - "qrf": 310.7086248397827, - "zi_qrf": 693.6227560043335, - "microcosm_qrf": 346.5654013156891, - "microcosm_qrf_unweighted": 369.52848386764526 + "logit_family": 39.928866147994995, + "pmm": 1.5424842834472656, + "qrf": 172.0643048286438, + "zi_qrf": 475.4314589500427, + "microcosm_qrf": 228.54575967788696, + "microcosm_qrf_unweighted": 284.83655405044556 }, "environment": { "python": "3.13.13", diff --git a/experiments/moop_spm/run.py b/experiments/moop_spm/run.py index 89258353..e45a274f 100644 --- a/experiments/moop_spm/run.py +++ b/experiments/moop_spm/run.py @@ -99,10 +99,9 @@ def config_hash(config: dict) -> str: def installed_source(dist: str) -> str | None: """Return a distribution's version and source, or None if it is absent. - For an install from git, the source is the commit that pip or uv - recorded in the distribution's ``direct_url.json`` (PEP 610). Metadata - that names no version raises, so a damaged install never reads as an - absent one. + For an install from git, it reports the commit that pip or uv recorded + in the distribution's ``direct_url.json`` (PEP 610). It raises, rather + than returning None, when the metadata names no version. """ try: found = distribution(dist) @@ -124,14 +123,24 @@ def installed_source(dist: str) -> str | None: def environment() -> dict[str, str | None]: """Return the versions of Python, ``PACKAGES`` and Microcosm. - Microcosm's entries carry the commit it came from, and they are None - when it is not installed. + Microcosm's entries carry the commit each package came from, or None + for an absent package. It raises when a Microcosm package imports + without distribution metadata, since the run would then hash the + package's source into the cache key without recording where it came + from. """ versions: dict[str, str | None] = { name: version(dist) for name, dist in PACKAGES.items() } - for name, (dist, _) in MICROCOSM_PACKAGES.items(): - versions[name] = installed_source(dist) + for name, (dist, module) in MICROCOSM_PACKAGES.items(): + source = installed_source(dist) + if source is None and installed_package_sha256(module) is not None: + raise RuntimeError( + f"{module} imports, but no {dist} distribution metadata " + "records its version or source; reinstall it from git " + "(see README.md)." + ) + versions[name] = source return {"python": platform.python_version(), **versions} @@ -149,9 +158,10 @@ def package_sha256(package: str) -> str: def installed_package_sha256(package: str) -> str | None: - """Return ``package_sha256``, or None when the package is not installed. + """Return ``package_sha256``, or None when Python finds no such package. - An installed package that fails to import raises as usual. + When the package exists but fails to import, the import error + propagates. """ try: return package_sha256(package) diff --git a/tests/test_moop_spm_experiment.py b/tests/test_moop_spm_experiment.py index 403cdd7a..4627e1f2 100644 --- a/tests/test_moop_spm_experiment.py +++ b/tests/test_moop_spm_experiment.py @@ -17,13 +17,17 @@ - Extracted CSVs that differ from the pinned zip are extracted again. - The fold cache misses whenever the environment, microimpute's or Microcosm's source or the model inputs change. +- ``installed_source`` raises, rather than reading a package as absent, + when its metadata names no version, and ``environment`` raises when a + Microcosm package imports without distribution metadata. - Microcosm's draws are zero or lie within the range of the spenders that carry weight, and they do not depend on the number of threads. -- Microcosm's gate takes each donor's unit weight, in donor order, and - takes no weights when the fit is unweighted. -- Distribution metadata that names no version raises rather than reading - as an absent package. - These tests skip when Microcosm is not installed; it installs from git. +- Microcosm fits its gate on every donor's row: the donor's predictors + and whether it spends, with its unit weight in the weighted fit and no + weights in the unweighted fit. + +The tests of Microcosm's draws, threads and gate skip on a machine +without Microcosm, which installs from git. """ import argparse @@ -384,17 +388,23 @@ def test_microcosm_gate_takes_donor_weights_in_order(monkeypatch): original = HistGradientBoostingClassifier.fit def spy(self, X, y, sample_weight=None, **kwargs): - recorded.append( - None if sample_weight is None else np.array(sample_weight, dtype=float) - ) + weights = sample_weight + if weights is not None: + weights = np.array(weights, dtype=float) + recorded.append((np.array(X, dtype=float), np.array(y), weights)) return original(self, X, y, sample_weight=sample_weight, **kwargs) monkeypatch.setattr(HistGradientBoostingClassifier, "fit", spy) fold = _fold(7) + donors = fold.X_donor.to_numpy(dtype=float) + spends = (fold.moop_donor > 0).astype(int) microcosm_qrf(fold, 1, seed=7, weighted=True, n_jobs=1) - # One in-sample gate, fitted on every donor with that donor's weight. + # Microcosm fits one in-sample gate on every donor's row: the donor's + # predictors and whether it spends, with that donor's weight. assert len(recorded) == 1 - weights = recorded[0] + features, labels, weights = recorded[0] + assert np.array_equal(features, donors) + assert np.array_equal(labels, spends) assert weights is not None assert weights.shape == fold.weight_donor.shape assert np.allclose( @@ -409,7 +419,10 @@ def spy(self, X, y, sample_weight=None, **kwargs): recorded.clear() microcosm_qrf(fold, 1, seed=7, weighted=False, n_jobs=1) assert len(recorded) == 1 - assert recorded[0] is None + features, labels, weights = recorded[0] + assert np.array_equal(features, donors) + assert np.array_equal(labels, spends) + assert weights is None def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monkeypatch): @@ -437,6 +450,32 @@ def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monke assert re.fullmatch(r"\S+ @ ([0-9a-f]{40}|\S+)", source) +def test_environment_refuses_microcosm_imports_without_metadata(tmp_path, monkeypatch): + # Two packages import; only the first has distribution metadata. + for name in ("recorded_moop", "unrecorded_moop"): + (tmp_path / name).mkdir() + (tmp_path / name / "__init__.py").write_text("") + recorded = tmp_path / "recorded_moop-1.0.dist-info" + recorded.mkdir() + (recorded / "METADATA").write_text( + "Metadata-Version: 2.1\nName: recorded-moop\nVersion: 1.0\n" + ) + monkeypatch.syspath_prepend(str(tmp_path)) + importlib.invalidate_caches() + present = { + "recorded": ("recorded-moop", "recorded_moop"), + "absent": ("no-such-distribution-moop", "no_such_package_moop"), + } + monkeypatch.setattr(run_module, "MICROCOSM_PACKAGES", present) + found = run_module.environment() + assert found["recorded"] == "1.0" + assert found["absent"] is None + unrecorded = {"unrecorded": ("unrecorded-moop", "unrecorded_moop")} + monkeypatch.setattr(run_module, "MICROCOSM_PACKAGES", unrecorded) + with pytest.raises(RuntimeError, match="unrecorded_moop imports"): + run_module.environment() + + @settings(max_examples=10, deadline=None) @given(seed=st.integers(0, 2**31)) def test_evaluate_invariants(seed): From 1f9fd7ff3f4cfa7a2615985e83c72cfa24310a21 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 7 Nov 2026 19:12:05 -0500 Subject: [PATCH 4/5] Answer PR 223's third review: name the forest's grower, count order - The note template now says "a forest Microcosm grows on donor spenders", since "it" could mean the gate's classifier. NOTE.md re-renders with that one phrase changed. - The README now says Microcosm counts a forest's distinct donor values before any weighted resample, as _fit_forest does. Co-Authored-By: Claude Opus 5.5 --- experiments/moop_spm/NOTE.md | 2 +- experiments/moop_spm/NOTE.template.md | 2 +- experiments/moop_spm/README.md | 7 ++++--- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/experiments/moop_spm/NOTE.md b/experiments/moop_spm/NOTE.md index 05aa8cc4..6b33b1cc 100644 --- a/experiments/moop_spm/NOTE.md +++ b/experiments/moop_spm/NOTE.md @@ -10,7 +10,7 @@ We limit every method to predictors an ACS-style file carries: members' ages, se - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest it grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest Microcosm grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results diff --git a/experiments/moop_spm/NOTE.template.md b/experiments/moop_spm/NOTE.template.md index 69051c5b..b7c029e4 100644 --- a/experiments/moop_spm/NOTE.template.md +++ b/experiments/moop_spm/NOTE.template.md @@ -10,7 +10,7 @@ We limit every method to predictors an ACS-style file carries: members' ages, se - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest it grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest Microcosm grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index 450b3576..ce69edca 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -90,9 +90,10 @@ ways: - It keeps every training sample in each leaf of its MOOP forest, where microimpute keeps quantile-forest's default of at most one sample per leaf. Microcosm caps a forest's leaves at 4,096 samples only when the - donors that forest trains on hold 2 to 32 distinct values. Its MOOP - forest trains on the donor spenders, whose amounts take 7,830 to 7,880 - distinct values in each fold. + donors it hands that forest hold 2 to 32 distinct values, which it + counts before any weighted resample. Its MOOP forest trains on the + donor spenders, whose amounts take 7,830 to 7,880 distinct values in + each fold. - It reads each unit's draw at that unit's own quantile, interpolating a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute rounds each unit's quantile q to the nearest hundredth, k/100, and From 1e75f7eddea1dcfe1da2def1652f2e3f17d6ce54 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 7 Nov 2026 20:33:28 -0500 Subject: [PATCH 5/5] Answer PR 223's fourth review: count order, weighted-fit checks - The README now says the donors hold 2 to 32 distinct values before any weighted resample. In a weighted fit, Microcosm's _fit_forest draws the resample first and then counts the values of the rows it receives, so "counts before" gave the wrong order. - The property test now checks that each weighted draw equals zero or lies between the smallest and largest spender that keeps its weight, and that n_jobs=2 leaves the weighted draws unchanged, as the test module's docstring says. Co-Authored-By: Claude Opus 5.5 --- experiments/moop_spm/README.md | 7 +++---- tests/test_moop_spm_experiment.py | 12 ++++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index ce69edca..12b48c9d 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -90,10 +90,9 @@ ways: - It keeps every training sample in each leaf of its MOOP forest, where microimpute keeps quantile-forest's default of at most one sample per leaf. Microcosm caps a forest's leaves at 4,096 samples only when the - donors it hands that forest hold 2 to 32 distinct values, which it - counts before any weighted resample. Its MOOP forest trains on the - donor spenders, whose amounts take 7,830 to 7,880 distinct values in - each fold. + donors it hands that forest hold 2 to 32 distinct values before any + weighted resample. Its MOOP forest trains on the donor spenders, whose + amounts take 7,830 to 7,880 distinct values in each fold. - It reads each unit's draw at that unit's own quantile, interpolating a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute rounds each unit's quantile q to the nearest hundredth, k/100, and diff --git a/tests/test_moop_spm_experiment.py b/tests/test_moop_spm_experiment.py index 4627e1f2..7582a8b7 100644 --- a/tests/test_moop_spm_experiment.py +++ b/tests/test_moop_spm_experiment.py @@ -355,16 +355,24 @@ def test_microcosm_draws_stay_within_weighted_spenders(seed, cap_quantile): assert np.array_equal(again, unweighted) # Donors without weight shape no draw: with zero weight on every - # spender above the cap, no weighted draw exceeds it. + # spender above the cap, a weighted draw is zero or lies between the + # smallest and largest spender that keeps its weight. cap = float(np.quantile(spent, cap_quantile)) capped = dataclasses.replace( fold, weight_donor=np.where(fold.moop_donor > cap, 0.0, fold.weight_donor), ) + kept = capped.moop_donor[(capped.moop_donor > 0) & (capped.weight_donor > 0)] weighted = microcosm_qrf(capped, 3, seed=seed, weighted=True, n_jobs=1) assert weighted.shape == (3, n_recipients) - assert (weighted >= 0).all() + assert np.isfinite(weighted).all() + drawn = weighted[weighted != 0] + assert (drawn >= kept.min()).all() + assert (drawn <= kept.max()).all() assert weighted.max() <= cap + # The thread count changes no weighted draw either. + again = microcosm_qrf(capped, 3, seed=seed, weighted=True, n_jobs=2) + assert np.array_equal(again, weighted) def test_microcosm_threads_sets_and_restores_widths(monkeypatch):