diff --git a/changelog.d/moop-spm-microcosm-qrf.changed.md b/changelog.d/moop-spm-microcosm-qrf.changed.md new file mode 100644 index 00000000..64cf995b --- /dev/null +++ b/changelog.d/moop-spm-microcosm-qrf.changed.md @@ -0,0 +1 @@ +Add Microcosm's regime-gated QRF to the MOOP imputation experiment (`experiments/moop_spm`), which fits it with and without the ASEC unit weights. diff --git a/experiments/moop_spm/NOTE.md b/experiments/moop_spm/NOTE.md index f9309ce5..6b33b1cc 100644 --- a/experiments/moop_spm/NOTE.md +++ b/experiments/moop_spm/NOTE.md @@ -6,40 +6,40 @@ The American Community Survey (ACS) does not ask about medical out-of-pocket spe We use the 134,729 people, 55,401 SPM units and 53,344 households in the 2026 CPS ASEC public-use file, which covers calendar-year 2025. We impute each unit's SPM medical expenses (`SPM_MEDXPNS`), its members' MOOP plus their Medicare premiums. Census subtracts them from SPM resources dollar for dollar, so we swap each unit's reported value for an imputed one and compare the new resources with an unchanged threshold. With reported MOOP and the ASEC person weight, we compute an SPM rate of 13.11% with a 90% margin of error of 0.31 points. Census published 13.11% and 0.31 in [P60-290](https://www.census.gov/library/publications/2026/demo/p60-290.html). Subtracting MOOP raises the rate by 2.25 points (SE 0.08), and Census reports 2.28. -We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to 5 folds and impute each fold from models we fit on the rest, without survey weights. We average 10 draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. +We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to 5 folds and impute each fold from models we fit on the rest, all without survey weights except one fit in method (c). We average 10 draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests from PolicyEngine's microimpute package, alone (QRF) and inside its zero-inflated wrapper (ZI-QRF). We give each unit the value at a random quantile of its predicted distribution. In PolicyEngine's microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest Microcosm grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results -| | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:| -| SPM rate, all people (%) | 13.11 | 13.42 | 13.46 | 13.50 | 13.43 | 13.04 | 13.07 | -| Change, all people (pp, SE) | | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | -| Change, 65 and older (pp) | | +0.98 | +1.15 | +0.75 | +0.60 | -0.10 | -0.13 | -| Change, with a disability (pp) | | +1.20 | +1.40 | +0.59 | +0.41 | -0.21 | -0.22 | -| Change, Medicare (pp) | | +1.20 | +1.23 | +0.85 | +0.59 | -0.08 | -0.11 | -| Change, resources before MOOP 1.0 to 1.25 x threshold (pp) | | +14.69 | +14.37 | +4.88 | +4.19 | +0.55 | +0.61 | -| Mean MOOP, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | -| Units with zero MOOP (%) | 6.4 | 0.5 | 4.3 | 6.5 | 6.2 | 6.2 | 6.1 | -| Median MOOP | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | -| 99th percentile MOOP | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | -| Total MOOP ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | -| People newly poor / newly not poor (%) | | 1.37 / 1.07 | 1.40 / 1.06 | 1.74 / 1.36 | 1.63 / 1.31 | 1.33 / 1.40 | 1.37 / 1.41 | +| | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| SPM rate, all people (%) | 13.11 | 13.42 | 13.46 | 13.50 | 13.43 | 13.04 | 13.07 | 13.18 | 13.11 | +| Change, all people (pp, SE) | | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | +0.06 (0.07) | +0.00 (0.06) | +| Change, 65 and older (pp) | | +0.98 | +1.15 | +0.75 | +0.60 | -0.10 | -0.13 | -0.04 | -0.02 | +| Change, with a disability (pp) | | +1.20 | +1.40 | +0.59 | +0.41 | -0.21 | -0.22 | -0.17 | -0.23 | +| Change, Medicare (pp) | | +1.20 | +1.23 | +0.85 | +0.59 | -0.08 | -0.11 | -0.04 | -0.03 | +| Change, resources before MOOP 1.0 to 1.25 x threshold (pp) | | +14.69 | +14.37 | +4.88 | +4.19 | +0.55 | +0.61 | +0.42 | -0.13 | +| Mean MOOP, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | $2,434 | $2,389 | +| Units with zero MOOP (%) | 6.4 | 0.5 | 4.3 | 6.5 | 6.2 | 6.2 | 6.1 | 6.4 | 6.3 | +| Median MOOP | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | $3,602 | $3,517 | +| 99th percentile MOOP | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | $33,749 | $32,872 | +| Total MOOP ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | 850 | 842 | +| People newly poor / newly not poor (%) | | 1.37 / 1.07 | 1.40 / 1.06 | 1.74 / 1.36 | 1.63 / 1.31 | 1.33 / 1.40 | 1.37 / 1.41 | 1.47 / 1.41 | 1.41 / 1.42 | We compute rates and changes over people and MOOP statistics over SPM units, and show standard errors in parentheses. -We estimate that logit plus cell means raises the SPM rate by 0.30 percentage points (SE 0.08), or 13% of MOOP's own 2.25-point effect. The OLS variant adds 0.34 points, logit draws with donors 0.39 and PMM 0.32. QRF changes the rate by -0.07 points (SE 0.07) and ZI-QRF by -0.05 (SE 0.07), though their draws fall 3.6% and 3.7% short of the reported MOOP total. For people 65 and older, the rate changes by +0.98 points under logit plus cell means, +0.60 under PMM and -0.10 under QRF. For people with a disability, it changes by +1.20, +0.41 and -0.21. +We estimate that logit plus cell means raises the SPM rate by 0.30 percentage points (SE 0.08), or 13% of MOOP's own 2.25-point effect. The OLS variant adds 0.34 points, logit draws with donors 0.39 and PMM 0.32. All four QRF fits leave the rate within 0.1 point of its level with reported MOOP. Microcosm's QRF changes it by +0.06 points (SE 0.07) with the unit weights and by +0.00 (SE 0.06) without them, and its weighted draws fall 1.0% short of the reported MOOP total. The microimpute QRF and ZI-QRF change the rate by -0.07 (SE 0.07) and -0.05 (SE 0.07), and their draws fall 3.6% and 3.7% short. For people 65 and older, the rate changes by +0.98 points under logit plus cell means, +0.60 under PMM and -0.04 under Microcosm's weighted QRF. For people with a disability, it changes by +1.20, +0.41 and -0.17. -Logit draws with donors track the reported distribution, with 6.5% of units at zero against 6.4% and a 99th percentile of $33,147 against $33,640, yet raise the rate by 0.39 points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend $2,385 on average, and units at 2 to 3 times spend $6,086. In that first band above the line, cell means assign $3,189, donor draws $2,949, PMM $2,799 and QRF $2,352. The band's 24.9% poverty rate rises by 14.69, 4.88, 4.19 and 0.55 points under those four methods. +Logit draws with donors track the reported distribution, with 6.5% of units at zero against 6.4% and a 99th percentile of $33,147 against $33,640, yet raise the rate by 0.39 points. Microcosm's weighted QRF also tracks it, with 6.4% at zero and a 99th percentile of $33,749, and moves the rate by +0.06 points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend $2,385 on average, and units at 2 to 3 times spend $6,086. In that first band above the line, cell means assign $3,189, donor draws $2,949, PMM $2,799 and Microcosm's weighted QRF $2,434. The band's 24.9% poverty rate rises by 14.69, 4.88, 4.19 and 0.42 points under those four methods. Under the 0.5 cutoff, 0.5% of units get no MOOP, against 6.4% in the file. Drawing participation instead lowers the national change from +0.30 to +0.10 points with cell means, and from +0.59 to +0.39 with donor draws (Table 3 in `TABLE.md`). When we draw participation and assign cell means, the rate still rises 11.50 points for people at 1.0 to 1.25 times the threshold and falls 4.80 points at 1.25 to 1.5. Cell means also cut the 99th percentile from $33,640 to $15,585. Under the same cutoff, coverage-group means alone, without the age, income and size cells, raise the national rate by 2.07 points, 92% of MOOP's own effect. -Every method moves people across the line in both directions. QRF moves 1.33% of people into poverty and 1.40% out of it, and logit plus cell means moves 1.37% in and 1.07% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. +Every method moves people across the line in both directions. Microcosm's weighted QRF moves 1.47% of people into poverty and 1.41% out of it, and logit plus cell means moves 1.37% in and 1.07% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. -Census imputes MOOP for 43% of the people in the file and the Medicare premium for 13%, so part of our benchmark comes from its imputation. In the 44% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by 0.26 to 0.52 points. QRF and ZI-QRF change it by +0.01 and +0.07 points (SE 0.09 and 0.10). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. +Census imputes MOOP for 43% of the people in the file and the Medicare premium for 13%, so part of our benchmark comes from its imputation. In the 44% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by 0.26 to 0.52 points. The four QRF fits change it by +0.01 to +0.14 points (SE 0.09 to 0.10). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. For each method's change by age, disability, coverage type and resources relative to the threshold, see Table 1 in `TABLE.md`; for the code, see `README.md`. diff --git a/experiments/moop_spm/NOTE.template.md b/experiments/moop_spm/NOTE.template.md index 6abab906..b7c029e4 100644 --- a/experiments/moop_spm/NOTE.template.md +++ b/experiments/moop_spm/NOTE.template.md @@ -6,11 +6,11 @@ The American Community Survey (ACS) does not ask about medical out-of-pocket spe We use the {n_persons} people, {n_units} SPM units and {n_households} households in the 2026 CPS ASEC public-use file, which covers calendar-year 2025. We impute each unit's SPM medical expenses (`SPM_MEDXPNS`), its members' MOOP plus their Medicare premiums. Census subtracts them from SPM resources dollar for dollar, so we swap each unit's reported value for an imputed one and compare the new resources with an unchanged threshold. With reported MOOP and the ASEC person weight, we compute an SPM rate of {rate2_all_reported}% with a 90% margin of error of {reported_moe90} points. Census published {published_rate}% and {published_moe90} in [P60-290](https://www.census.gov/library/publications/2026/demo/p60-290.html). Subtracting MOOP raises the rate by {moop_effect} points (SE {moop_effect_se}), and Census reports {published_moop_effect}. -We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to {folds} folds and impute each fold from models we fit on the rest, without survey weights. We average {draws} draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. +We limit every method to predictors an ACS-style file carries: members' ages, sex, disability and current health coverage, the unit's cash income on its own and relative to its SPM threshold, family type, tenure and state. We assign whole households to {folds} folds and impute each fold from models we fit on the rest, all without survey weights except one fit in method (c). We average {draws} draws for each stochastic method and compute standard errors from the 160 replicate weights, treating each fitted model as fixed. - In method (a), we fit a logistic model of whether a unit has any MOOP. Each unit with a predicted probability of 0.5 or more gets the mean MOOP of donor spenders in its cell, and the rest get zero (logit plus cell means). In the OLS variant, predicted spenders get an OLS prediction among donor spenders instead. We form cells from coverage group, head's age, cash income relative to the threshold and unit size. - In method (b), logit draws with donors, we draw participation at random from the same model's probabilities, and each spender takes a random donor spender's MOOP from its cell. -- In method (c), we fit quantile regression forests from PolicyEngine's microimpute package, alone (QRF) and inside its zero-inflated wrapper (ZI-QRF). We give each unit the value at a random quantile of its predicted distribution. In PolicyEngine's microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. +- In method (c), we fit quantile regression forests (QRF), which give each unit the value at a random quantile of its predicted distribution. Microcosm, which builds PolicyEngine's microdata, imputes with a QRF behind a gate: a gradient-boosted classifier draws whether a unit spends, and a forest Microcosm grows on donor spenders draws the amount. Microcosm weights its fits by default, so we fit Microcosm's QRF with the ASEC unit weights, and again without them. We also fit the QRF in PolicyEngine's microimpute package, alone and inside its zero-inflated wrapper (ZI-QRF). In the microimpute [paper](../../paper/main.pdf), María Juaristi found that quantile regression forests had the lowest average quantile loss of five methods for imputing Survey of Consumer Finances (SCF) net worth onto the CPS. - We also apply predictive mean matching (PMM) within five coverage groups, where the Census researchers used eight. In each group, we regress log(1 + MOOP) on the predictors across all donors, including those with zero spending. Each recipient then takes the MOOP of a random one of the five donors whose predictions lie nearest its own. ## Results @@ -19,14 +19,14 @@ We limit every method to predictors an ACS-style file carries: members' ages, se We compute rates and changes over people and MOOP statistics over SPM units, and show standard errors in parentheses. -We estimate that logit plus cell means raises the SPM rate by {pp_all_logit_cell_mean} percentage points (SE {se_all_logit_cell_mean}), or {share_of_effect_logit_cell_mean}% of MOOP's own {moop_effect}-point effect. The OLS variant adds {pp_all_logit_reg_mean} points, logit draws with donors {pp_all_logit_draw_hotdeck} and PMM {pp_all_pmm}. QRF changes the rate by {diff_all_qrf} points (SE {se_all_qrf}) and ZI-QRF by {diff_all_zi_qrf} (SE {se_all_zi_qrf}), though their draws fall {agg_gap_qrf}% and {agg_gap_zi_qrf}% short of the reported MOOP total. For people 65 and older, the rate changes by {diff_senior_logit_cell_mean} points under logit plus cell means, {diff_senior_pmm} under PMM and {diff_senior_qrf} under QRF. For people with a disability, it changes by {diff_disabled_logit_cell_mean}, {diff_disabled_pmm} and {diff_disabled_qrf}. +We estimate that logit plus cell means raises the SPM rate by {pp_all_logit_cell_mean} percentage points (SE {se_all_logit_cell_mean}), or {share_of_effect_logit_cell_mean}% of MOOP's own {moop_effect}-point effect. The OLS variant adds {pp_all_logit_reg_mean} points, logit draws with donors {pp_all_logit_draw_hotdeck} and PMM {pp_all_pmm}. All four QRF fits leave the rate within 0.1 point of its level with reported MOOP. Microcosm's QRF changes it by {diff_all_microcosm_qrf} points (SE {se_all_microcosm_qrf}) with the unit weights and by {diff_all_microcosm_qrf_unweighted} (SE {se_all_microcosm_qrf_unweighted}) without them, and its weighted draws fall {agg_gap_microcosm_qrf}% short of the reported MOOP total. The microimpute QRF and ZI-QRF change the rate by {diff_all_qrf} (SE {se_all_qrf}) and {diff_all_zi_qrf} (SE {se_all_zi_qrf}), and their draws fall {agg_gap_qrf}% and {agg_gap_zi_qrf}% short. For people 65 and older, the rate changes by {diff_senior_logit_cell_mean} points under logit plus cell means, {diff_senior_pmm} under PMM and {diff_senior_microcosm_qrf} under Microcosm's weighted QRF. For people with a disability, it changes by {diff_disabled_logit_cell_mean}, {diff_disabled_pmm} and {diff_disabled_microcosm_qrf}. -Logit draws with donors track the reported distribution, with {zero_logit_draw_hotdeck}% of units at zero against {zero_reported}% and a 99th percentile of {p99_logit_draw_hotdeck} against {p99_reported}, yet raise the rate by {pp_all_logit_draw_hotdeck} points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend {bandmean1_reported} on average, and units at 2 to 3 times spend {bandmean4_reported}. In that first band above the line, cell means assign {bandmean1_logit_cell_mean}, donor draws {bandmean1_logit_draw_hotdeck}, PMM {bandmean1_pmm} and QRF {bandmean1_qrf}. The band's {rate_band1_reported}% poverty rate rises by {pp_band1_logit_cell_mean}, {pp_band1_logit_draw_hotdeck}, {pp_band1_pmm} and {pp_band1_qrf} points under those four methods. +Logit draws with donors track the reported distribution, with {zero_logit_draw_hotdeck}% of units at zero against {zero_reported}% and a 99th percentile of {p99_logit_draw_hotdeck} against {p99_reported}, yet raise the rate by {pp_all_logit_draw_hotdeck} points. Microcosm's weighted QRF also tracks it, with {zero_microcosm_qrf}% at zero and a 99th percentile of {p99_microcosm_qrf}, and moves the rate by {diff_all_microcosm_qrf} points. A unit falls below the line when its MOOP exceeds the margin by which its other resources clear the threshold, and reported MOOP rises with resources. Units whose resources before MOOP run 1.0 to 1.25 times their threshold spend {bandmean1_reported} on average, and units at 2 to 3 times spend {bandmean4_reported}. In that first band above the line, cell means assign {bandmean1_logit_cell_mean}, donor draws {bandmean1_logit_draw_hotdeck}, PMM {bandmean1_pmm} and Microcosm's weighted QRF {bandmean1_microcosm_qrf}. The band's {rate_band1_reported}% poverty rate rises by {pp_band1_logit_cell_mean}, {pp_band1_logit_draw_hotdeck}, {pp_band1_pmm} and {pp_band1_microcosm_qrf} points under those four methods. Under the 0.5 cutoff, {zero_logit_cell_mean}% of units get no MOOP, against {zero_reported}% in the file. Drawing participation instead lowers the national change from {diff_all_logit_cell_mean} to {diff_all_logit_draw_cell_mean} points with cell means, and from {diff_all_logit_threshold_hotdeck} to {diff_all_logit_draw_hotdeck} with donor draws (Table 3 in `TABLE.md`). When we draw participation and assign cell means, the rate still rises {pp_band1_logit_draw_cell_mean} points for people at 1.0 to 1.25 times the threshold and falls {pp_band2_logit_draw_cell_mean} points at 1.25 to 1.5. Cell means also cut the 99th percentile from {p99_reported} to {p99_logit_cell_mean}. Under the same cutoff, coverage-group means alone, without the age, income and size cells, raise the national rate by {pp_all_logit_group_mean} points, {share_of_effect_logit_group_mean}% of MOOP's own effect. -Every method moves people across the line in both directions. QRF moves {newly_poor_qrf}% of people into poverty and {newly_nonpoor_qrf}% out of it, and logit plus cell means moves {newly_poor_logit_cell_mean}% in and {newly_nonpoor_logit_cell_mean}% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. +Every method moves people across the line in both directions. Microcosm's weighted QRF moves {newly_poor_microcosm_qrf}% of people into poverty and {newly_nonpoor_microcosm_qrf}% out of it, and logit plus cell means moves {newly_poor_logit_cell_mean}% in and {newly_nonpoor_logit_cell_mean}% out. Census researchers turned to the ACS for state and substate SPM estimates, where these flows need not offset. -Census imputes MOOP for {share_persons_moop_allocated}% of the people in the file and the Medicare premium for {share_persons_premium_allocated}%, so part of our benchmark comes from its imputation. In the {share_units_no_allocation}% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by {subset_pp_low_ab_pmm} to {subset_pp_high_ab_pmm} points. QRF and ZI-QRF change it by {subset_diff_qrf} and {subset_diff_zi_qrf} points (SE {subset_se_qrf} and {subset_se_zi_qrf}). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. +Census imputes MOOP for {share_persons_moop_allocated}% of the people in the file and the Medicare premium for {share_persons_premium_allocated}%, so part of our benchmark comes from its imputation. In the {share_units_no_allocation}% of SPM units where Census imputes neither for any member, methods (a), (b) and PMM raise the rate by {subset_pp_low_ab_pmm} to {subset_pp_high_ab_pmm} points. The four QRF fits change it by {subset_diff_low_forest} to {subset_diff_high_forest} points (SE {subset_se_low_forest} to {subset_se_high_forest}). Because our donors and recipients share one survey's questions, income measures and population, we do not test the differences between surveys that an ACS imputation also faces. For each method's change by age, disability, coverage type and resources relative to the threshold, see Table 1 in `TABLE.md`; for the code, see `README.md`. diff --git a/experiments/moop_spm/README.md b/experiments/moop_spm/README.md index 2ddac855..12b48c9d 100644 --- a/experiments/moop_spm/README.md +++ b/experiments/moop_spm/README.md @@ -60,8 +60,8 @@ carries: ## Methods -We fit every method on donor units, without survey weights, and impute -held-out recipient units (`methods.py`). +We fit every method on donor units and impute held-out recipient units +(`methods.py`). Only the weighted Microcosm fit uses survey weights. | Name | Participation | Amount | |---|---|---| @@ -71,6 +71,33 @@ held-out recipient units (`methods.py`). | `pmm` | implicit | predictive mean matching on log(1 + MOOP) within five coverage groups, random one of 5 nearest donors | | (c) `qrf` | implicit | microimpute `QRF`, one random quantile per unit | | (c) `zi_qrf` | microimpute gate | microimpute `ZeroInflatedImputer` around `QRF` | +| (c) `microcosm_qrf` | Microcosm gate, a gradient-boosted classifier | Microcosm `RegimeGatedQRF` forest on donor spenders, one random quantile per unit; gate and forest use the SPM unit weights | +| (c) `microcosm_qrf_unweighted` | same, without weights | same, without weights | + +Microcosm (`PolicyEngine/microcosm`) builds PolicyEngine's microdata, +and its `microcosm-fit` package imputes with `RegimeGatedQRF`. We fit it +with its defaults (100 trees, `max_samples_leaf=None` and no +cross-fitting of the gate), once with each donor unit's SPM weight +(`SPM_WEIGHT / 100`) and once without weights. With the weights, +Microcosm passes them to the gate as sample weights and resamples +spenders in proportion to them before it grows the forest. Both fits +share a seed, so their draws use the same random numbers. Beyond the +weights, Microcosm's model differs from microimpute's `QRF` in three +ways: + +- Its gate draws whether a unit spends. microimpute's `QRF` has no + gate, and `ZeroInflatedImputer` adds one of its own. +- It keeps every training sample in each leaf of its MOOP forest, where + microimpute keeps quantile-forest's default of at most one sample per + leaf. Microcosm caps a forest's leaves at 4,096 samples only when the + donors it hands that forest hold 2 to 32 distinct values before any + weighted resample. Its MOOP forest trains on the donor spenders, whose + amounts take 7,830 to 7,880 distinct values in each fold. +- It reads each unit's draw at that unit's own quantile, interpolating + a 203-point grid whose ends lie 0.000001 from 0 and 1. microimpute + rounds each unit's quantile q to the nearest hundredth, k/100, and + reads the forest's (k + 1)/102 quantile, so its draws come from 101 + points that run from 1/102 to 101/102 (q = 0.01 reads 2/102). We form cells from the unit's coverage group (Medicare, employer, direct purchase, public, uninsured), the head's age band, cash income relative @@ -106,7 +133,19 @@ cells). ## Running -Install microimpute, then run from the repository root: +Install microimpute with its `dev` extra, and Microcosm's `graph`, +`frame` and `fit` packages at the commit we used. Microcosm needs Python +3.13 or later and installs from git: + +```bash +uv pip install -e ".[dev]" +uv pip install \ + "microcosm-graph @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-graph" \ + "microcosm-frame @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-frame" \ + "microcosm-fit @ git+https://github.com/PolicyEngine/microcosm@7080ba8a3ff4ba498575ec1a624a9064a6f619c8#subdirectory=packages/microcosm-fit" +``` + +Then run from the repository root: ```bash python -m experiments.moop_spm.run --data-dir ~/data/cps_asec_2026 @@ -115,15 +154,20 @@ python -m experiments.moop_spm.render The first command downloads the 139 MB Census zip, checks its SHA-256, caches each fold's imputations in `results/cache/`, which git ignores, -and writes `results/results.json`. +and writes `results/results.json`. `--n-jobs` (default 4) caps the +threads of microimpute's `QRF` and of each Microcosm fit; a test checks +that Microcosm's draws do not depend on it. `run.py` refits every fold when any of these change: - the settings or the experiment's source files; -- any microimpute source file; +- any source file of microimpute or of Microcosm's `fit`, `frame` and + `graph` packages; - the version of Python, numpy, pandas, scipy, scikit-learn, - quantile-forest or microimpute; + quantile-forest, microimpute or Microcosm, or the commit Microcosm came + from; - the model inputs or their row keys. -`run.py` also records each method's run time in `results.json`. The -tests in `tests/test_moop_spm_experiment.py` check the experiment's -invariants on synthetic data. +`run.py` also records each method's run time, and the commit each +Microcosm package came from, in `results.json`. The tests in +`tests/test_moop_spm_experiment.py` check the experiment's invariants on +synthetic data; the Microcosm tests skip on a machine without Microcosm. diff --git a/experiments/moop_spm/TABLE.md b/experiments/moop_spm/TABLE.md index ce282bfd..64a8a208 100644 --- a/experiments/moop_spm/TABLE.md +++ b/experiments/moop_spm/TABLE.md @@ -4,52 +4,52 @@ Source: CPS ASEC 2026 public-use file (calendar-year 2025 income), 134,729 peopl ## Table 1. SPM poverty rate with reported MOOP (%), and the change under each imputation (percentage points, SE) -| Group | People (n) | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:|---:| -| All people | 134,729 | 13.1 | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | -| **Age** | | | | | | | | | -| Under 18 | 31,444 | 13.4 | +0.17 (0.13) | +0.26 (0.14) | +0.42 (0.11) | +0.41 (0.11) | +0.00 (0.11) | +0.06 (0.11) | -| 18 to 64 | 77,083 | 12.3 | +0.13 (0.09) | +0.11 (0.09) | +0.26 (0.07) | +0.19 (0.07) | -0.08 (0.07) | -0.06 (0.07) | -| 65 and older | 26,202 | 15.4 | +0.98 (0.17) | +1.15 (0.16) | +0.75 (0.13) | +0.60 (0.15) | -0.10 (0.14) | -0.13 (0.13) | -| **Disability** | | | | | | | | | -| With a disability | 14,477 | 22.7 | +1.20 (0.22) | +1.40 (0.21) | +0.59 (0.18) | +0.41 (0.20) | -0.21 (0.18) | -0.22 (0.18) | -| No disability | 94,282 | 11.7 | +0.21 (0.09) | +0.20 (0.09) | +0.35 (0.07) | +0.28 (0.07) | -0.07 (0.07) | -0.04 (0.07) | -| Not asked (mostly under 15) | 25,970 | 13.2 | +0.15 (0.14) | +0.28 (0.15) | +0.42 (0.11) | +0.41 (0.12) | +0.02 (0.12) | +0.05 (0.12) | -| **Current health insurance** | | | | | | | | | -| Medicare | 27,328 | 16.6 | +1.20 (0.17) | +1.23 (0.16) | +0.85 (0.14) | +0.59 (0.15) | -0.08 (0.15) | -0.11 (0.13) | -| Employer | 66,520 | 3.9 | -0.21 (0.08) | -0.12 (0.08) | +0.14 (0.07) | +0.23 (0.07) | -0.02 (0.07) | +0.02 (0.08) | -| Direct purchase | 8,341 | 18.3 | +0.06 (0.39) | +2.28 (0.48) | +0.20 (0.32) | +0.71 (0.36) | -0.22 (0.36) | -0.04 (0.36) | -| Medicaid or other means-tested | 18,134 | 32.1 | +1.18 (0.23) | +0.32 (0.22) | +0.99 (0.18) | +0.33 (0.20) | +0.01 (0.17) | -0.03 (0.18) | -| TRICARE, VA or CHAMPVA | 3,174 | 7.2 | -0.17 (0.21) | -0.59 (0.36) | -0.29 (0.29) | -0.15 (0.29) | -0.01 (0.26) | -0.09 (0.30) | -| Uninsured | 11,232 | 25.8 | +0.07 (0.27) | -0.22 (0.28) | +0.06 (0.23) | -0.05 (0.22) | -0.33 (0.23) | -0.28 (0.24) | -| **Resources before MOOP / threshold** | | | | | | | | | -| Under 1.0 | 13,803 | 100.0 | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | -| 1.0 to 1.25 | 8,441 | 24.9 | +14.69 (1.01) | +14.37 (1.05) | +4.88 (0.74) | +4.19 (0.75) | +0.55 (0.79) | +0.61 (0.81) | -| 1.25 to 1.5 | 9,233 | 5.8 | -4.75 (0.50) | -3.92 (0.49) | +0.50 (0.49) | +0.23 (0.51) | -0.76 (0.48) | -0.57 (0.50) | -| 1.5 to 2.0 | 17,495 | 1.6 | -1.56 (0.15) | -1.54 (0.15) | +0.19 (0.17) | +0.17 (0.18) | -0.29 (0.17) | -0.26 (0.16) | -| 2.0 to 3.0 | 30,010 | 0.3 | -0.32 (0.07) | -0.32 (0.07) | +0.05 (0.07) | +0.03 (0.08) | -0.10 (0.07) | -0.06 (0.07) | -| 3.0 or more | 55,747 | 0.1 | -0.05 (0.02) | -0.05 (0.02) | +0.03 (0.02) | +0.02 (0.02) | +0.02 (0.02) | +0.00 (0.02) | +| Group | People (n) | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| All people | 134,729 | 13.1 | +0.30 (0.08) | +0.34 (0.08) | +0.39 (0.07) | +0.32 (0.07) | -0.07 (0.07) | -0.05 (0.07) | +0.06 (0.07) | +0.00 (0.06) | +| **Age** | | | | | | | | | | | +| Under 18 | 31,444 | 13.4 | +0.17 (0.13) | +0.26 (0.14) | +0.42 (0.11) | +0.41 (0.11) | +0.00 (0.11) | +0.06 (0.11) | +0.14 (0.10) | +0.02 (0.10) | +| 18 to 64 | 77,083 | 12.3 | +0.13 (0.09) | +0.11 (0.09) | +0.26 (0.07) | +0.19 (0.07) | -0.08 (0.07) | -0.06 (0.07) | +0.07 (0.07) | -0.01 (0.07) | +| 65 and older | 26,202 | 15.4 | +0.98 (0.17) | +1.15 (0.16) | +0.75 (0.13) | +0.60 (0.15) | -0.10 (0.14) | -0.13 (0.13) | -0.04 (0.13) | -0.02 (0.13) | +| **Disability** | | | | | | | | | | | +| With a disability | 14,477 | 22.7 | +1.20 (0.22) | +1.40 (0.21) | +0.59 (0.18) | +0.41 (0.20) | -0.21 (0.18) | -0.22 (0.18) | -0.17 (0.18) | -0.23 (0.17) | +| No disability | 94,282 | 11.7 | +0.21 (0.09) | +0.20 (0.09) | +0.35 (0.07) | +0.28 (0.07) | -0.07 (0.07) | -0.04 (0.07) | +0.09 (0.07) | +0.02 (0.07) | +| Not asked (mostly under 15) | 25,970 | 13.2 | +0.15 (0.14) | +0.28 (0.15) | +0.42 (0.11) | +0.41 (0.12) | +0.02 (0.12) | +0.05 (0.12) | +0.13 (0.11) | +0.03 (0.12) | +| **Current health insurance** | | | | | | | | | | | +| Medicare | 27,328 | 16.6 | +1.20 (0.17) | +1.23 (0.16) | +0.85 (0.14) | +0.59 (0.15) | -0.08 (0.15) | -0.11 (0.13) | -0.04 (0.14) | -0.03 (0.14) | +| Employer | 66,520 | 3.9 | -0.21 (0.08) | -0.12 (0.08) | +0.14 (0.07) | +0.23 (0.07) | -0.02 (0.07) | +0.02 (0.08) | +0.06 (0.07) | +0.00 (0.07) | +| Direct purchase | 8,341 | 18.3 | +0.06 (0.39) | +2.28 (0.48) | +0.20 (0.32) | +0.71 (0.36) | -0.22 (0.36) | -0.04 (0.36) | +0.63 (0.36) | +0.21 (0.35) | +| Medicaid or other means-tested | 18,134 | 32.1 | +1.18 (0.23) | +0.32 (0.22) | +0.99 (0.18) | +0.33 (0.20) | +0.01 (0.17) | -0.03 (0.18) | +0.12 (0.17) | +0.04 (0.18) | +| TRICARE, VA or CHAMPVA | 3,174 | 7.2 | -0.17 (0.21) | -0.59 (0.36) | -0.29 (0.29) | -0.15 (0.29) | -0.01 (0.26) | -0.09 (0.30) | -0.53 (0.35) | -0.17 (0.30) | +| Uninsured | 11,232 | 25.8 | +0.07 (0.27) | -0.22 (0.28) | +0.06 (0.23) | -0.05 (0.22) | -0.33 (0.23) | -0.28 (0.24) | -0.05 (0.23) | -0.17 (0.25) | +| **Resources before MOOP / threshold** | | | | | | | | | | | +| Under 1.0 | 13,803 | 100.0 | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +0.00 (0.00) | +| 1.0 to 1.25 | 8,441 | 24.9 | +14.69 (1.01) | +14.37 (1.05) | +4.88 (0.74) | +4.19 (0.75) | +0.55 (0.79) | +0.61 (0.81) | +0.42 (0.75) | -0.13 (0.77) | +| 1.25 to 1.5 | 9,233 | 5.8 | -4.75 (0.50) | -3.92 (0.49) | +0.50 (0.49) | +0.23 (0.51) | -0.76 (0.48) | -0.57 (0.50) | +0.03 (0.49) | -0.23 (0.49) | +| 1.5 to 2.0 | 17,495 | 1.6 | -1.56 (0.15) | -1.54 (0.15) | +0.19 (0.17) | +0.17 (0.18) | -0.29 (0.17) | -0.26 (0.16) | +0.12 (0.17) | +0.03 (0.16) | +| 2.0 to 3.0 | 30,010 | 0.3 | -0.32 (0.07) | -0.32 (0.07) | +0.05 (0.07) | +0.03 (0.08) | -0.10 (0.07) | -0.06 (0.07) | +0.04 (0.08) | -0.01 (0.08) | +| 3.0 or more | 55,747 | 0.1 | -0.05 (0.02) | -0.05 (0.02) | +0.03 (0.02) | +0.02 (0.02) | +0.02 (0.02) | +0.00 (0.02) | +0.03 (0.02) | +0.04 (0.02) | Every method imputes MOOP of zero or more, so people whose unit's resources before MOOP fall below its threshold stay poor, and the 'Under 1.0' row cannot change. ## Table 2. MOOP distribution across SPM units (unit weights) and gross flows of people (person weights) -| Statistic | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) QRF | (c) ZI-QRF | -|---|---:|---:|---:|---:|---:|---:|---:| -| Mean | $6,011 | $6,156 | $6,133 | $5,984 | $5,922 | $5,797 | $5,791 | -| Median | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | -| Share with zero MOOP | 6.4% | 0.5% | 4.3% | 6.5% | 6.2% | 6.2% | 6.1% | -| 75th percentile | $8,160 | $8,625 | $8,983 | $8,154 | $8,091 | $7,949 | $7,891 | -| 90th percentile | $14,300 | $11,436 | $11,507 | $14,276 | $14,158 | $13,820 | $13,765 | -| 99th percentile | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | -| Total ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | -| Mean, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | -| Mean, resources before MOOP 1.25 to 1.5 x threshold | $3,381 | $3,679 | $3,667 | $3,446 | $3,452 | $3,282 | $3,272 | -| Mean, resources before MOOP 1.5 to 2.0 x threshold | $4,510 | $4,788 | $4,612 | $4,546 | $4,527 | $4,335 | $4,389 | -| Mean, resources before MOOP 2.0 to 3.0 x threshold | $6,086 | $6,116 | $5,872 | $5,953 | $5,783 | $5,809 | $5,843 | -| People newly poor (%) | 0.0 | 1.37 | 1.40 | 1.74 | 1.63 | 1.33 | 1.37 | -| People newly not poor (%) | 0.0 | 1.07 | 1.06 | 1.36 | 1.31 | 1.40 | 1.41 | -| Change in rate among people in units where Census imputes no MOOP or Medicare premium (pp) | 0.00 | +0.36 (0.11) | +0.26 (0.11) | +0.52 (0.09) | +0.48 (0.10) | +0.01 (0.09) | +0.07 (0.10) | +| Statistic | Reported | (a) Logit + cell mean | (a) Logit + OLS mean | (b) Logit draw + donor | PMM | (c) microimpute QRF | (c) microimpute ZI-QRF | (c) Microcosm QRF, weighted | (c) Microcosm QRF, unweighted | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| Mean | $6,011 | $6,156 | $6,133 | $5,984 | $5,922 | $5,797 | $5,791 | $5,949 | $5,891 | +| Median | $3,700 | $6,132 | $5,891 | $3,678 | $3,624 | $3,623 | $3,610 | $3,602 | $3,517 | +| Share with zero MOOP | 6.4% | 0.5% | 4.3% | 6.5% | 6.2% | 6.2% | 6.1% | 6.4% | 6.3% | +| 75th percentile | $8,160 | $8,625 | $8,983 | $8,154 | $8,091 | $7,949 | $7,891 | $8,040 | $7,982 | +| 90th percentile | $14,300 | $11,436 | $11,507 | $14,276 | $14,158 | $13,820 | $13,765 | $14,285 | $14,139 | +| 99th percentile | $33,640 | $15,585 | $15,769 | $33,147 | $32,666 | $31,356 | $31,424 | $33,749 | $32,872 | +| Total ($ billions) | 859 | 880 | 876 | 855 | 846 | 828 | 828 | 850 | 842 | +| Mean, resources before MOOP 1.0 to 1.25 x threshold | $2,385 | $3,189 | $3,108 | $2,949 | $2,799 | $2,352 | $2,372 | $2,434 | $2,389 | +| Mean, resources before MOOP 1.25 to 1.5 x threshold | $3,381 | $3,679 | $3,667 | $3,446 | $3,452 | $3,282 | $3,272 | $3,432 | $3,344 | +| Mean, resources before MOOP 1.5 to 2.0 x threshold | $4,510 | $4,788 | $4,612 | $4,546 | $4,527 | $4,335 | $4,389 | $4,473 | $4,427 | +| Mean, resources before MOOP 2.0 to 3.0 x threshold | $6,086 | $6,116 | $5,872 | $5,953 | $5,783 | $5,809 | $5,843 | $6,071 | $5,929 | +| People newly poor (%) | 0.0 | 1.37 | 1.40 | 1.74 | 1.63 | 1.33 | 1.37 | 1.47 | 1.41 | +| People newly not poor (%) | 0.0 | 1.07 | 1.06 | 1.36 | 1.31 | 1.40 | 1.41 | 1.41 | 1.42 | +| Change in rate among people in units where Census imputes no MOOP or Medicare premium (pp) | 0.00 | +0.36 (0.11) | +0.26 (0.11) | +0.52 (0.09) | +0.48 (0.10) | +0.01 (0.09) | +0.07 (0.10) | +0.14 (0.09) | +0.05 (0.09) | ## Table 3. How the participation and amount rules of the logit approach move the rate (all people) diff --git a/experiments/moop_spm/methods.py b/experiments/moop_spm/methods.py index 90af7811..966482c3 100644 --- a/experiments/moop_spm/methods.py +++ b/experiments/moop_spm/methods.py @@ -21,6 +21,13 @@ - ``qrf``: microimpute's ``QRF`` fit on all donors, one random quantile draw per recipient. - ``zi_qrf``: microimpute's ``ZeroInflatedImputer`` around ``QRF``. +- ``microcosm_qrf``: Microcosm's ``RegimeGatedQRF``, the model that + imputes PolicyEngine's microdata, with the donors' unit weights. + Microcosm's ``fit`` asks for design weights by default and raises when a + frame carries another kind; a caller fitting a DataFrame must pass a + weight column's name, a weight vector or ``NO_WEIGHTS``. +- ``microcosm_qrf_unweighted``: the same model without weights, like + every other method here. Decomposition variants cross the participation rule (threshold or draw) with the amount rule (cell mean or donor draw), and @@ -31,7 +38,9 @@ from __future__ import annotations -from collections.abc import Callable, Sequence +import os +from collections.abc import Callable, Iterator, Sequence +from contextlib import contextmanager from dataclasses import dataclass import numpy as np @@ -170,10 +179,15 @@ def participation_probability(model: Pipeline, X: pd.DataFrame) -> np.ndarray: @dataclass class FoldData: - """Donor and recipient data for one cross-fitting fold.""" + """Donor and recipient data for one cross-fitting fold. + + ``weight_donor`` holds the donors' SPM unit weights; only the weighted + Microcosm fit reads them. + """ X_donor: pd.DataFrame moop_donor: np.ndarray + weight_donor: np.ndarray cells_donor: pd.DataFrame group_donor: np.ndarray X_recipient: pd.DataFrame @@ -353,6 +367,70 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: return draws +@contextmanager +def microcosm_threads(n_jobs: int) -> Iterator[None]: + """Cap Microcosm's threads at ``n_jobs``. + + Microcosm reads its forest-fitting and drawing widths from environment + variables that its ``qrf`` module names, and without them it fits and + draws on every core. Its gate, a scikit-learn gradient-boosted + classifier, runs on OpenMP threads, which ``threadpoolctl`` caps. The + previous settings come back on exit. + """ + from microcosm.fit import qrf as microcosm_qrf_module + from threadpoolctl import threadpool_limits + + names = ( + microcosm_qrf_module._N_JOBS_ENV, + microcosm_qrf_module._PREDICT_WORKERS_ENV, + ) + saved = {name: os.environ.get(name) for name in names} + os.environ.update({name: str(n_jobs) for name in names}) + try: + with threadpool_limits(limits=n_jobs, user_api="openmp"): + yield + finally: + for name, value in saved.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + + +def microcosm_qrf( + fold: FoldData, + n_draws: int, + seed: int, + weighted: bool, + n_jobs: int = 4, +) -> np.ndarray: + """Microcosm's ``RegimeGatedQRF`` on all donors. + + A gradient-boosted classifier draws whether each recipient spends, and + a quantile regression forest that Microcosm grows on donor spenders + gives each spender the amount at a random quantile of its predicted + distribution. With ``weighted``, the gate takes the donors' unit + weights as sample weights, and Microcosm resamples spenders in + proportion to them before it grows the forest; without it, neither the + gate nor the forest uses the weights. Each draw calls ``predict`` again + on the one fitted model, which advances the model's own random stream. + """ + from microcosm.fit import NO_WEIGHTS, RegimeGatedQRF + + train = fold.X_donor.copy() + train["moop"] = fold.moop_donor + predictors = list(fold.X_donor.columns) + weights = fold.weight_donor if weighted else NO_WEIGHTS + draws = np.empty((n_draws, len(fold.X_recipient))) + with microcosm_threads(n_jobs): + fitted = RegimeGatedQRF(seed=seed).fit( + train, predictors, ["moop"], weights=weights + ) + for r in range(n_draws): + draws[r] = fitted.predict(fold.X_recipient)["moop"].to_numpy() + return draws + + MAIN_METHODS = [ "logit_cell_mean", "logit_reg_mean", @@ -360,6 +438,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", ] DECOMPOSITION_METHODS = [ "logit_threshold_hotdeck", @@ -381,6 +461,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm": "PMM within coverage groups", "qrf": "(c) QRF (microimpute)", "zi_qrf": "(c) Zero-inflated QRF (microimpute)", + "microcosm_qrf": "(c) Regime-gated QRF (Microcosm), unit weights", + "microcosm_qrf_unweighted": "(c) Regime-gated QRF (Microcosm), unweighted", "logit_threshold_hotdeck": "Logit, p >= 0.5, donor amount in cell", "logit_draw_cell_mean": "Logit draw, cell mean", "logit_expected_value": "p x cell mean for every unit", @@ -391,6 +473,8 @@ def zi_qrf(fold: FoldData, n_draws: int, seed: int) -> np.ndarray: "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", "logit_threshold_hotdeck", "logit_draw_cell_mean", } diff --git a/experiments/moop_spm/render.py b/experiments/moop_spm/render.py index 04831c1b..bfc5020a 100644 --- a/experiments/moop_spm/render.py +++ b/experiments/moop_spm/render.py @@ -24,6 +24,8 @@ "pmm", "qrf", "zi_qrf", + "microcosm_qrf", + "microcosm_qrf_unweighted", ] SHORT = { "reported": "Reported", @@ -31,8 +33,10 @@ "logit_reg_mean": "(a) Logit + OLS mean", "logit_draw_hotdeck": "(b) Logit draw + donor", "pmm": "PMM", - "qrf": "(c) QRF", - "zi_qrf": "(c) ZI-QRF", + "qrf": "(c) microimpute QRF", + "zi_qrf": "(c) microimpute ZI-QRF", + "microcosm_qrf": "(c) Microcosm QRF, weighted", + "microcosm_qrf_unweighted": "(c) Microcosm QRF, unweighted", } DECOMPOSITION = [ "logit_cell_mean", @@ -339,6 +343,28 @@ def placeholder_values(results: dict) -> dict[str, str]: raise ValueError("A method (a), (b) or PMM no longer raises the subset rate.") values["subset_pp_low_ab_pmm"] = f"{100 * min(subset_ab_pmm):.2f}" values["subset_pp_high_ab_pmm"] = f"{100 * max(subset_ab_pmm):.2f}" + # The note says all four QRF fits leave the national rate within 0.1 + # point and fall short of the reported MOOP total. + forests = ["qrf", "zi_qrf", "microcosm_qrf", "microcosm_qrf_unweighted"] + if max(abs(methods[m]["diff"][0]) for m in forests) >= 0.001: + raise ValueError("A QRF fit moves the national rate by 0.1 point or more.") + totals = [methods[m]["distribution"]["aggregate_billions"] for m in forests] + if max(totals) >= reported_total: + raise ValueError("A QRF fit no longer falls short of the reported total.") + # It says four methods raise the rate of the first band above the line. + band1 = keyed["pre_moop_ratio::1.0 to 1.25"] + cited = ["logit_cell_mean", "logit_draw_hotdeck", "pmm", "microcosm_qrf"] + if min(methods[m]["diff"][band1] for m in cited) <= 0: + raise ValueError("A method the note cites no longer raises the band's rate.") + subset_forests = [methods[m]["no_allocation_subset"] for m in forests] + values["subset_diff_low_forest"] = signed(min(s["diff"] for s in subset_forests)) + values["subset_diff_high_forest"] = signed(max(s["diff"] for s in subset_forests)) + values["subset_se_low_forest"] = ( + f"{100 * min(s['diff_se'] for s in subset_forests):.2f}" + ) + values["subset_se_high_forest"] = ( + f"{100 * max(s['diff_se'] for s in subset_forests):.2f}" + ) return values diff --git a/experiments/moop_spm/results/results.json b/experiments/moop_spm/results/results.json index 0015c2c2..e687f82e 100644 --- a/experiments/moop_spm/results/results.json +++ b/experiments/moop_spm/results/results.json @@ -1356,6 +1356,236 @@ }, "n_draws": 10 }, + "microcosm_qrf": { + "rate": [ + 0.13176597711983554, + 0.13526968715039864, + 0.12350002246414134, + 0.15340594010547154, + 0.2248051223160781, + 0.11747645511385332, + 0.13305603599806434, + 0.16518960209642802, + 0.039906728623815765, + 0.18932364377891436, + 0.3223734578052057, + 0.06627867410088427, + 0.25705105851270915, + 1.0, + 0.25319871622296414, + 0.05838029863988854, + 0.016825135077092105, + 0.0036201119433489437, + 0.000798666807527483 + ], + "rate_se": [ + 0.0017722283598928515, + 0.003213551830044547, + 0.0019635171995840786, + 0.003145575335591901, + 0.004570227562425885, + 0.0017890591770403663, + 0.0034203544079126503, + 0.003258369600304733, + 0.0012402801655167996, + 0.007340907278622133, + 0.006704386028836305, + 0.008397985771672481, + 0.0069289837720845505, + 0.0, + 0.0064046245896981525, + 0.002575172671985775, + 0.0008808686908466718, + 0.00036387923913772964, + 0.00012341934682562108 + ], + "diff": [ + 0.0006492289734812418, + 0.001356539466790374, + 0.0007271289585814439, + -0.00037309587107262066, + -0.001697528458331904, + 0.0008522919105011101, + 0.0012577162752293591, + -0.0003615060214874899, + 0.0006332811849191072, + 0.006294465433623164, + 0.0012330146010093856, + -0.005308201685261957, + -0.0004650027951850211, + 0.0, + 0.004150763059049073, + 0.0003451013307301451, + 0.0012113353767259135, + 0.0004145096552255717, + 0.00029358445461573086 + ], + "diff_se": [ + 0.0006592651961442288, + 0.0010223477206047426, + 0.0007442494544711315, + 0.0013388474853659532, + 0.0018367416554727917, + 0.0006962035984257459, + 0.0011316013507247148, + 0.001399497208018087, + 0.0007312309884633065, + 0.0036334331101260772, + 0.001715176458800662, + 0.0034876761358560294, + 0.002348819525701735, + 0.0, + 0.007528622170789294, + 0.004931367011484733, + 0.0016992785907958133, + 0.0007953055936745014, + 0.00021537566922358558 + ], + "newly_poor_share": 0.0147271338655482, + "newly_nonpoor_share": 0.014077904892067034, + "overall_draw_sd": 0.00046779452762348413, + "distribution": { + "mean": 5948.790858172932, + "share_zero": 0.06368620791710222, + "aggregate_billions": 850.0712999217154, + "near_line_mean": 3671.0686281868802, + "median": 3601.6532323149245, + "p75": 8040.322086535487, + "p90": 14284.542837359739, + "p95": 19501.498348512174, + "p99": 33748.81173189083, + "band_means": [ + 1453.6003032561512, + 2434.0878138343346, + 3431.6513280482445, + 4473.238904220103, + 6070.872709541648, + 8641.984509555059 + ], + "mean_draw_sd": 37.017322597518294 + }, + "no_allocation_subset": { + "rate": 0.1202665144927313, + "diff": 0.001378939779788993, + "diff_se": 0.000939223230646948 + }, + "n_draws": 10 + }, + "microcosm_qrf_unweighted": { + "rate": [ + 0.13106692164163733, + 0.13415258046530917, + 0.12266759186609895, + 0.1535806780580723, + 0.22420695950786557, + 0.11683665432179541, + 0.13205388625435582, + 0.165254917347693, + 0.0393048547945922, + 0.18510263194184223, + 0.32149653668439665, + 0.06989252912718669, + 0.2557872081718448, + 1.0, + 0.24774896156061657, + 0.05572065656492517, + 0.015926979409537096, + 0.0030581291317636405, + 0.0009537681944727924 + ], + "rate_se": [ + 0.0017695074323104496, + 0.003213453284334693, + 0.0019564415761225773, + 0.0031537878382792957, + 0.004652394622314318, + 0.0017813163206770075, + 0.003432591884034214, + 0.0032725193303407593, + 0.0012324462825733747, + 0.007340458726984088, + 0.0067616772408449904, + 0.008757042095872686, + 0.006905335606267363, + 0.0, + 0.006152360771847005, + 0.0022250301228301096, + 0.0007600690732998304, + 0.00030587397587153307, + 0.0001374811138293 + ], + "diff": [ + -4.982650471696548e-05, + 0.0002394327817009101, + -0.00010530163946094462, + -0.00019835791847186424, + -0.0022956912665444418, + 0.0002124911184431949, + 0.00025556653152083997, + -0.0002961907702225264, + 3.140735569553965e-05, + 0.0020734535965510337, + 0.0003560934802003257, + -0.001694346658959539, + -0.0017288531360493686, + 0.0, + -0.0012989916032984938, + -0.0023145407442332275, + 0.00031317970917090453, + -0.00014747315635973155, + 0.00044868584156104025 + ], + "diff_se": [ + 0.0006483601963995772, + 0.0010427416199333315, + 0.0007169516960344145, + 0.0013375249925742245, + 0.001736854270344326, + 0.0006897482765244226, + 0.0011607888236397864, + 0.0013996269222166927, + 0.0007072886785143316, + 0.003482219843123647, + 0.0017802852979193254, + 0.0029699212851560337, + 0.0024659140628588844, + 0.0, + 0.00773088623665069, + 0.004935477370410093, + 0.0015855809293758682, + 0.000754633029987932, + 0.00021749338188776913 + ], + "newly_poor_share": 0.014148643352674075, + "newly_nonpoor_share": 0.014198469857391266, + "overall_draw_sd": 0.0004589807548556485, + "distribution": { + "mean": 5891.13011170698, + "share_zero": 0.0626552384039673, + "aggregate_billions": 841.8316850367131, + "near_line_mean": 3614.174309373083, + "median": 3516.8740708910273, + "p75": 7981.77211733802, + "p90": 14139.202845924421, + "p95": 19092.281833557445, + "p99": 32871.98515295489, + "band_means": [ + 1423.7467982065887, + 2389.264924735185, + 3343.7693548217867, + 4427.242685978768, + 5928.645050553944, + 8617.923648416621 + ], + "mean_draw_sd": 65.8135035294827 + }, + "no_allocation_subset": { + "rate": 0.11937030819262767, + "diff": 0.00048273347968535996, + "diff_se": 0.000879829562190696 + }, + "n_draws": 10 + }, "no_moop": { "rate": [ 0.10859685755998726, @@ -1649,11 +1879,16 @@ ], "source_sha256": { "cps_asec.py": "647857a0fa249c44", - "methods.py": "8ce97d8ddd38df23", + "methods.py": "4a36486e83ad2f88", "evaluate.py": "09daf780b35188ad", - "run.py": "324dad8ce5a851e0" + "run.py": "87dbb0975d32e8de" }, "microimpute_sha256": "e14c884a85e90486", + "microcosm_sha256": { + "microcosm.fit": "797ad4f288a8c2e5", + "microcosm.frame": "a5630b096c4ba177", + "microcosm.graph": "ee26504154cb597e" + }, "environment": { "python": "3.13.13", "numpy": "2.5.4", @@ -1661,17 +1896,22 @@ "scipy": "1.18.1", "scikit_learn": "1.8.0", "quantile_forest": "1.4.2", - "microimpute": "3.1.4" + "microimpute": "3.2.0", + "microcosm_fit": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_frame": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" }, "input_sha256": "5a129f5d767fff81", - "tag": "a82da82512c894e6", + "tag": "6ca09058c0101fce", "n_jobs": 4 }, "timings_seconds": { - "logit_family": 38.13895297050476, - "pmm": 0.5446643829345703, - "qrf": 68.79297137260437, - "zi_qrf": 185.8473162651062 + "logit_family": 39.928866147994995, + "pmm": 1.5424842834472656, + "qrf": 172.0643048286438, + "zi_qrf": 475.4314589500427, + "microcosm_qrf": 228.54575967788696, + "microcosm_qrf_unweighted": 284.83655405044556 }, "environment": { "python": "3.13.13", @@ -1680,6 +1920,9 @@ "scipy": "1.18.1", "scikit_learn": "1.8.0", "quantile_forest": "1.4.2", - "microimpute": "3.1.4" + "microimpute": "3.2.0", + "microcosm_fit": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_frame": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8", + "microcosm_graph": "0.1.0 @ 7080ba8a3ff4ba498575ec1a624a9064a6f619c8" } } diff --git a/experiments/moop_spm/run.py b/experiments/moop_spm/run.py index d910e96b..e45a274f 100644 --- a/experiments/moop_spm/run.py +++ b/experiments/moop_spm/run.py @@ -12,9 +12,11 @@ The first run downloads the Census public-use zip (139 MB) into ``--data-dir`` and checks its SHA-256. Per-fold imputations are cached in ``/cache`` so an interrupted run resumes. The cache key covers the -settings, the experiment's source files, every microimpute source file, -the versions of Python and the numerical packages, and the model inputs -with their row keys, so a change to any of them refits every fold. +settings, the experiment's source files, every source file of microimpute +and of Microcosm's ``fit``, ``frame`` and ``graph`` packages, the versions +of Python and the numerical packages, the commit Microcosm came from, and +the model inputs with their row keys, so a change to any of them refits +every fold. """ from __future__ import annotations @@ -26,7 +28,7 @@ import platform import time from collections.abc import Callable -from importlib.metadata import version +from importlib.metadata import PackageNotFoundError, distribution, version from pathlib import Path import numpy as np @@ -54,6 +56,7 @@ FoldData, cell_keys, logit_family, + microcosm_qrf, pmm, qrf, zi_qrf, @@ -70,6 +73,14 @@ "quantile_forest": "quantile-forest", "microimpute": "microimpute", } +# Microcosm's distributions and the packages they install. They install +# from git, and their version numbers do not change from commit to +# commit, so we record the commit and hash the source. +MICROCOSM_PACKAGES = { + "microcosm_fit": ("microcosm-fit", "microcosm.fit"), + "microcosm_frame": ("microcosm-frame", "microcosm.frame"), + "microcosm_graph": ("microcosm-graph", "microcosm.graph"), +} PUBLISHED = json.loads((HERE / "published_2025.json").read_text()) @@ -85,9 +96,51 @@ def config_hash(config: dict) -> str: return hashlib.sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()[:16] -def environment() -> dict[str, str]: - """Return the Python version and the versions in ``PACKAGES``.""" - versions = {name: version(dist) for name, dist in PACKAGES.items()} +def installed_source(dist: str) -> str | None: + """Return a distribution's version and source, or None if it is absent. + + For an install from git, it reports the commit that pip or uv recorded + in the distribution's ``direct_url.json`` (PEP 610). It raises, rather + than returning None, when the metadata names no version. + """ + try: + found = distribution(dist) + except PackageNotFoundError: + return None + # ``metadata.get`` reads a missing field as None without the warning, + # and later KeyError, that ``Distribution.version`` gives. + found_version = found.metadata.get("Version") + if not found_version: + raise RuntimeError(f"{dist}'s distribution metadata names no version.") + text = found.read_text("direct_url.json") + if not text: + return found_version + direct = json.loads(text) + source = direct.get("vcs_info", {}).get("commit_id") or direct.get("url") + return f"{found_version} @ {source}" + + +def environment() -> dict[str, str | None]: + """Return the versions of Python, ``PACKAGES`` and Microcosm. + + Microcosm's entries carry the commit each package came from, or None + for an absent package. It raises when a Microcosm package imports + without distribution metadata, since the run would then hash the + package's source into the cache key without recording where it came + from. + """ + versions: dict[str, str | None] = { + name: version(dist) for name, dist in PACKAGES.items() + } + for name, (dist, module) in MICROCOSM_PACKAGES.items(): + source = installed_source(dist) + if source is None and installed_package_sha256(module) is not None: + raise RuntimeError( + f"{module} imports, but no {dist} distribution metadata " + "records its version or source; reinstall it from git " + "(see README.md)." + ) + versions[name] = source return {"python": platform.python_version(), **versions} @@ -104,6 +157,32 @@ def package_sha256(package: str) -> str: return digest.hexdigest()[:16] +def installed_package_sha256(package: str) -> str | None: + """Return ``package_sha256``, or None when Python finds no such package. + + When the package exists but fails to import, the import error + propagates. + """ + try: + return package_sha256(package) + except ModuleNotFoundError as error: + parts = package.split(".") + parents = {".".join(parts[:i]) for i in range(1, len(parts) + 1)} + if error.name in parents: + return None + raise + + +def require_microcosm() -> None: + """Fail before any fitting when Microcosm's ``fit`` package is absent.""" + if installed_source("microcosm-fit") is None: + raise SystemExit( + "The Microcosm methods need microcosm-fit, which installs from " + "git (see README.md). Pass --skip-forests to run without the " + "forest methods." + ) + + def frame_sha256(*frames: pd.DataFrame) -> str: """Hash frames' column names, row keys and values.""" digest = hashlib.sha256() @@ -137,6 +216,10 @@ def build_config( "cell_levels": [list(level) for level in CELL_LEVELS], "source_sha256": {name: sha256_of(HERE / name)[:16] for name in SOURCES}, "microimpute_sha256": package_sha256("microimpute"), + "microcosm_sha256": { + module: installed_package_sha256(module) + for _, module in MICROCOSM_PACKAGES.values() + }, "environment": environment(), "input_sha256": frame_sha256(units, X, cells, fold_frame), } @@ -194,6 +277,7 @@ def run_fold( data = FoldData( X_donor=X.loc[donor], moop_donor=units["moop"].to_numpy()[donor], + weight_donor=units["unit_weight"].to_numpy()[donor], cells_donor=cells.loc[donor], group_donor=units["group"].to_numpy()[donor], X_recipient=X.loc[recipient], @@ -220,6 +304,22 @@ def run_fold( start = time.time() out["zi_qrf"] = zi_qrf(data, args.draws, seed=args.seed + fold) timings["zi_qrf"] = time.time() - start + + # Both Microcosm fits share a seed, so their draws use the same + # uniforms and differ only through the weights. + for method, weighted in ( + ("microcosm_qrf", True), + ("microcosm_qrf_unweighted", False), + ): + start = time.time() + out[method] = microcosm_qrf( + data, + args.draws, + seed=args.seed + fold, + weighted=weighted, + n_jobs=args.n_jobs, + ) + timings[method] = time.time() - start out["_timings"] = np.array([list(timings.values())]) out["_timing_names"] = np.array(list(timings.keys())) return out @@ -236,7 +336,8 @@ def main() -> None: parser.add_argument( "--skip-forests", action="store_true", - help="Skip QRF and ZI-QRF (for quick checks).", + help="Skip the forest methods, microimpute's and Microcosm's " + "(for quick checks).", ) parser.add_argument( "--max-households", @@ -245,6 +346,8 @@ def main() -> None: help="Subsample households (for quick checks).", ) args = parser.parse_args() + if not args.skip_forests: + require_microcosm() output = args.output cache = output / "cache" diff --git a/tests/test_moop_spm_experiment.py b/tests/test_moop_spm_experiment.py index 4063c4d4..7582a8b7 100644 --- a/tests/test_moop_spm_experiment.py +++ b/tests/test_moop_spm_experiment.py @@ -15,12 +15,26 @@ - The logistic-participation variants follow their stated rules. - Census's MOOP and Medicare premium imputations both count as allocated. - Extracted CSVs that differ from the pinned zip are extracted again. -- The fold cache misses whenever the environment, microimpute's source - or the model inputs change. +- The fold cache misses whenever the environment, microimpute's or + Microcosm's source or the model inputs change. +- ``installed_source`` raises, rather than reading a package as absent, + when its metadata names no version, and ``environment`` raises when a + Microcosm package imports without distribution metadata. +- Microcosm's draws are zero or lie within the range of the spenders + that carry weight, and they do not depend on the number of threads. +- Microcosm fits its gate on every donor's row: the donor's predictors + and whether it spends, with its unit weight in the weighted fit and no + weights in the unweighted fit. + +The tests of Microcosm's draws, threads and gate skip on a machine +without Microcosm, which installs from git. """ import argparse +import dataclasses +import importlib import os +import re import zipfile import numpy as np @@ -48,6 +62,8 @@ assign_cells, fit_participation, logit_family, + microcosm_qrf, + microcosm_threads, nearest_donors, participation_probability, ) @@ -57,6 +73,8 @@ collect_folds, config_hash, frame_sha256, + installed_package_sha256, + installed_source, ) SETTINGS = settings( @@ -280,6 +298,7 @@ def features(n): return FoldData( X_donor=X_donor, moop_donor=moop, + weight_donor=rng.uniform(100, 3_000, n_donor), cells_donor=cells_donor, group_donor=cells_donor["group"].to_numpy(), X_recipient=features(n_recipient), @@ -315,6 +334,156 @@ def test_logit_family_rules(seed): assert np.all((drawn == 0) | np.isclose(drawn, means[i])) +@settings(max_examples=3, deadline=None) +@given(seed=st.integers(0, 2**31), cap_quantile=st.floats(0.3, 0.9)) +def test_microcosm_draws_stay_within_weighted_spenders(seed, cap_quantile): + pytest.importorskip("microcosm.fit") + fold = _fold(seed) + n_recipients = len(fold.X_recipient) + spent = fold.moop_donor[fold.moop_donor > 0] + + unweighted = microcosm_qrf(fold, 3, seed=seed, weighted=False, n_jobs=1) + assert unweighted.shape == (3, n_recipients) + assert np.isfinite(unweighted).all() + # A draw is zero or lies between the smallest and largest spender. + drawn = unweighted[unweighted != 0] + assert (drawn >= spent.min()).all() + assert (drawn <= spent.max()).all() + assert not np.array_equal(unweighted[0], unweighted[1]) + # The thread count changes no draw. + again = microcosm_qrf(fold, 3, seed=seed, weighted=False, n_jobs=2) + assert np.array_equal(again, unweighted) + + # Donors without weight shape no draw: with zero weight on every + # spender above the cap, a weighted draw is zero or lies between the + # smallest and largest spender that keeps its weight. + cap = float(np.quantile(spent, cap_quantile)) + capped = dataclasses.replace( + fold, + weight_donor=np.where(fold.moop_donor > cap, 0.0, fold.weight_donor), + ) + kept = capped.moop_donor[(capped.moop_donor > 0) & (capped.weight_donor > 0)] + weighted = microcosm_qrf(capped, 3, seed=seed, weighted=True, n_jobs=1) + assert weighted.shape == (3, n_recipients) + assert np.isfinite(weighted).all() + drawn = weighted[weighted != 0] + assert (drawn >= kept.min()).all() + assert (drawn <= kept.max()).all() + assert weighted.max() <= cap + # The thread count changes no weighted draw either. + again = microcosm_qrf(capped, 3, seed=seed, weighted=True, n_jobs=2) + assert np.array_equal(again, weighted) + + +def test_microcosm_threads_sets_and_restores_widths(monkeypatch): + qrf_module = pytest.importorskip("microcosm.fit.qrf") + fit_name, draw_name = qrf_module._N_JOBS_ENV, qrf_module._PREDICT_WORKERS_ENV + monkeypatch.delenv(fit_name, raising=False) + monkeypatch.setenv(draw_name, "7") + with microcosm_threads(3): + # Microcosm's own resolvers read the cap. + assert qrf_module._fit_n_jobs() == 3 + assert qrf_module._predict_workers() == 3 + assert fit_name not in os.environ + assert os.environ[draw_name] == "7" + + +def test_microcosm_gate_takes_donor_weights_in_order(monkeypatch): + pytest.importorskip("microcosm.fit") + from sklearn.ensemble import HistGradientBoostingClassifier + + recorded = [] + original = HistGradientBoostingClassifier.fit + + def spy(self, X, y, sample_weight=None, **kwargs): + weights = sample_weight + if weights is not None: + weights = np.array(weights, dtype=float) + recorded.append((np.array(X, dtype=float), np.array(y), weights)) + return original(self, X, y, sample_weight=sample_weight, **kwargs) + + monkeypatch.setattr(HistGradientBoostingClassifier, "fit", spy) + fold = _fold(7) + donors = fold.X_donor.to_numpy(dtype=float) + spends = (fold.moop_donor > 0).astype(int) + microcosm_qrf(fold, 1, seed=7, weighted=True, n_jobs=1) + # Microcosm fits one in-sample gate on every donor's row: the donor's + # predictors and whether it spends, with that donor's weight. + assert len(recorded) == 1 + features, labels, weights = recorded[0] + assert np.array_equal(features, donors) + assert np.array_equal(labels, spends) + assert weights is not None + assert weights.shape == fold.weight_donor.shape + assert np.allclose( + weights / weights.sum(), fold.weight_donor / fold.weight_donor.sum() + ) + # A reversed donor order would fail the same check. + assert not np.allclose( + weights / weights.sum(), + fold.weight_donor[::-1] / fold.weight_donor.sum(), + ) + + recorded.clear() + microcosm_qrf(fold, 1, seed=7, weighted=False, n_jobs=1) + assert len(recorded) == 1 + features, labels, weights = recorded[0] + assert np.array_equal(features, donors) + assert np.array_equal(labels, spends) + assert weights is None + + +def test_installed_source_and_hash_tolerate_only_absent_packages(tmp_path, monkeypatch): + assert installed_source("no-such-distribution-moop") is None + assert installed_package_sha256("no_such_package_moop.sub") is None + # A package that is present but fails to import still raises. + broken = tmp_path / "broken_package_moop" + broken.mkdir() + (broken / "__init__.py").write_text("import no_such_dependency_moop\n") + # So does a distribution whose metadata names no version. + versionless = tmp_path / "versionless_moop-0.0.dist-info" + versionless.mkdir() + (versionless / "METADATA").write_text( + "Metadata-Version: 2.1\nName: versionless-moop\n" + ) + monkeypatch.syspath_prepend(str(tmp_path)) + importlib.invalidate_caches() + with pytest.raises(ModuleNotFoundError): + installed_package_sha256("broken_package_moop") + with pytest.raises(RuntimeError, match="names no version"): + installed_source("versionless-moop") + source = installed_source("microcosm-fit") + if source is not None: + # A git install records its commit. + assert re.fullmatch(r"\S+ @ ([0-9a-f]{40}|\S+)", source) + + +def test_environment_refuses_microcosm_imports_without_metadata(tmp_path, monkeypatch): + # Two packages import; only the first has distribution metadata. + for name in ("recorded_moop", "unrecorded_moop"): + (tmp_path / name).mkdir() + (tmp_path / name / "__init__.py").write_text("") + recorded = tmp_path / "recorded_moop-1.0.dist-info" + recorded.mkdir() + (recorded / "METADATA").write_text( + "Metadata-Version: 2.1\nName: recorded-moop\nVersion: 1.0\n" + ) + monkeypatch.syspath_prepend(str(tmp_path)) + importlib.invalidate_caches() + present = { + "recorded": ("recorded-moop", "recorded_moop"), + "absent": ("no-such-distribution-moop", "no_such_package_moop"), + } + monkeypatch.setattr(run_module, "MICROCOSM_PACKAGES", present) + found = run_module.environment() + assert found["recorded"] == "1.0" + assert found["absent"] is None + unrecorded = {"unrecorded": ("unrecorded-moop", "unrecorded_moop")} + monkeypatch.setattr(run_module, "MICROCOSM_PACKAGES", unrecorded) + with pytest.raises(RuntimeError, match="unrecorded_moop imports"): + run_module.environment() + + @settings(max_examples=10, deadline=None) @given(seed=st.integers(0, 2**31)) def test_evaluate_invariants(seed): @@ -426,7 +595,7 @@ def fit(fold: int) -> dict[str, np.ndarray]: return fit -def test_cache_misses_when_environment_or_microimpute_changes(tmp_path, monkeypatch): +def test_cache_misses_when_environment_or_model_source_changes(tmp_path, monkeypatch): args, units, X, cells, folds = _cache_inputs() base_env = run_module.environment() monkeypatch.setattr(run_module, "environment", lambda: base_env) @@ -454,7 +623,21 @@ def test_cache_misses_when_environment_or_microimpute_changes(tmp_path, monkeypa assert calls == [0, 1, 2] assert (fresh["qrf"] == 99.0).all() + # Microcosm from another commit refits. + moved = {**base_env, "microcosm_fit": "0.1.0 @ another-commit"} + monkeypatch.setattr(run_module, "environment", lambda: moved) + assert config_hash(build_config(args, units, X, cells, folds)) != tag + monkeypatch.setattr(run_module, "environment", lambda: base_env) + assert config_hash(build_config(args, units, X, cells, folds)) == tag + # Edited Microcosm source refits, and so does edited microimpute source. + original = run_module.installed_package_sha256 + monkeypatch.setattr( + run_module, "installed_package_sha256", lambda package: "edited" + ) + assert config_hash(build_config(args, units, X, cells, folds)) != tag + monkeypatch.setattr(run_module, "installed_package_sha256", original) + assert config_hash(build_config(args, units, X, cells, folds)) == tag monkeypatch.setattr(run_module, "package_sha256", lambda package: "edited") assert config_hash(build_config(args, units, X, cells, folds)) != tag