From b06c5331fa22a4ff4090d5db456280e18cb5fcde Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 27 Sep 2026 05:55:30 -0400 Subject: [PATCH 1/3] =?UTF-8?q?Score=20the=20#982=20real=20pool=20at=20fra?= =?UTF-8?q?me=20weight;=20relabel=20the=20=C3=972=20tables?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #982 receipts weighted the PUF-clone half x2 ("as if this half were the whole population") and compared that with SOI Table 1.1. In the real pool the survey half (clone index 0) carries the other 50% of the weight and has no units at $5M+ proxy AGI, so before calibration and main's capital-gains tail stage the pool reaches about half of SOI at $5M+ after #1033, not SOI. - Add actual_pool_bands.py (lint-clean restyle of the recovered script; it imports the SOI bands and income items from score_variants.py, and gives byte-identical JSON and stdout to the original on synthetic frames) and its recorded output results/actual_pool_bands.json. - README: new "Real pool at frame weight" section; label the existing tables "PUF-clone half ×2 (distribution check, not a level comparison)"; correct the Colorado limit (2.1x SOI with one record at 81%, not ~4x at 84%; the record's frame weight is 434, and 867 was its x2 weight). - changelog.d fragment (fixed). Refs #982, #940, #1033. Co-Authored-By: Claude Opus 5.5 --- changelog.d/982-real-pool-bands.fixed.md | 1 + experiments/982-puf-self-prediction/README.md | 96 ++++++++--- .../actual_pool_bands.py | 154 ++++++++++++++++++ .../results/actual_pool_bands.json | 112 +++++++++++++ 4 files changed, 344 insertions(+), 19 deletions(-) create mode 100644 changelog.d/982-real-pool-bands.fixed.md create mode 100644 experiments/982-puf-self-prediction/actual_pool_bands.py create mode 100644 experiments/982-puf-self-prediction/results/actual_pool_bands.json diff --git a/changelog.d/982-real-pool-bands.fixed.md b/changelog.d/982-real-pool-bands.fixed.md new file mode 100644 index 0000000000..146a6e08b7 --- /dev/null +++ b/changelog.d/982-real-pool-bands.fixed.md @@ -0,0 +1 @@ +Corrected the #982 experiment README: its PUF-clone-half ×2 tables are now labeled distribution checks, and a new section scores the real pool at frame weight, which before calibration reaches about half of SOI at $5M+ proxy AGI after #1033. diff --git a/experiments/982-puf-self-prediction/README.md b/experiments/982-puf-self-prediction/README.md index 7b7f2edd0b..2acab4d321 100644 --- a/experiments/982-puf-self-prediction/README.md +++ b/experiments/982-puf-self-prediction/README.md @@ -1,10 +1,13 @@ # PUF imputation self-prediction (#982): receipts -Issue: PolicyEngine/microcosm#982. Branch `us-982-puf-demographic-rank-predictors`. -Scripts in this folder: `compare_predictors.py` (fit and predict per predictor -variant) and `score_variants.py` (score against SOI). Scored output: -`results/scores_8_trees.json`; input counts and per-run metadata (trees, rows, -peak memory): `results/inputs_and_runs.json`. +Issue: PolicyEngine/microcosm#982. Fix merged in #1033 (branch +`us-982-puf-demographic-rank-predictors`). Scripts in this folder: +`compare_predictors.py` (fit and predict per predictor variant), +`score_variants.py` (score the PUF-clone half ×2 against SOI) and +`actual_pool_bands.py` (score the real pool, both halves at frame weight). +Scored output: `results/scores_8_trees.json` and +`results/actual_pool_bands.json`; input counts and per-run metadata (trees, +rows, peak memory): `results/inputs_and_runs.json`. ## The defect @@ -16,7 +19,9 @@ variant below), PUF-clone wages have rank correlation 1.000 with the same unit's survey wages; among units with positive survey wages, 99.98% land within 10% of their survey value; and no clone exceeds the survey maximum. Survey topcodes and underreporting therefore carried into the PUF half, which -is where the missing $5M+ tail came from (#958). +is why that half lacked the $5M+ tail (#958). The survey half, which carries +the other half of the weight, has no units there at all; see +[Real pool at frame weight](#real-pool-at-frame-weight). ## The fix @@ -39,12 +44,19 @@ middle. All numbers below are pre-calibration, from one fit on the saved 2026-09-12 donor frame (211,677 PUF tax units) predicted onto the 231,007 saved PUF-clone -recipients, weights = PUF-clone half x2. AGI is a proxy: the sum of the 15 -imputed income items. Run with 8 trees per variant (memory was constrained); -every variant uses the same configuration. SOI references: Table 1.1 TY2023 -counts and AGI; Table 1.4 shares. +recipients, weights = PUF-clone half ×2 ("as if this half were the whole +population"). AGI is a proxy: the sum of the 15 imputed income items. Run with +8 trees per variant (memory was constrained); every variant uses the same +configuration. SOI references: Table 1.1 TY2023 counts and AGI; Table 1.4 +shares. -### Returns by size of proxy AGI +The ×2 weighting compares the variants' distributions with one another and +with SOI's shape. It is not a level comparison with SOI: in the real pool the +survey half carries the other 50% of the weight and has no units at $5M+ proxy +AGI. For levels, see +[Real pool at frame weight](#real-pool-at-frame-weight). + +### Returns by size of proxy AGI: PUF-clone half ×2 (distribution check, not a level comparison) | Band | Old design | Rank only | Rank + flag (chosen) | Within-group rank + flag | SOI 1.1 | |---|---|---|---|---|---| @@ -54,10 +66,10 @@ counts and AGI; Table 1.4 shares. | $5M-10M | 23,600 | 47,582 | 48,629 | 49,569 | 49,262 | | $10M+ | 8,186 | 31,404 | 33,920 | 29,041 | 30,382 | -AGI in $10M+: old $185B, chosen $1,037B, SOI $908B. Capital gains share of -$10M+ AGI: old 0.1%, chosen 42.6%, SOI Table 1.4 39.5%. +AGI in $10M+ (×2): old $185B, chosen $1,037B, SOI $908B. Capital gains share +of $10M+ AGI: old 0.1%, chosen 42.6%, SOI Table 1.4 39.5%. -### Wages: self-prediction and participation +### Wage self-prediction and participation: PUF-clone half ×2 (distribution check, not a level comparison) | Measure | Old | Rank only | Rank + flag | Within-group | |---|---|---|---|---| @@ -77,13 +89,59 @@ that includes many small separate returns (for example dependents filing their own), cutting imputed wages to $7.93T; the pooled rank keeps $9.68T with the same participation fidelity. +## Real pool at frame weight + +The PUF-detail expansion (`microcosm.build.us_runtime.puf_support`) splits +each unit's weight evenly between two clone channels. Clone index 0, the +native survey channel, keeps the survey values; clone index 1 +(`PUF_TAX_DETAIL_CLONE_INDEX`) receives the imputed ones. +`actual_pool_bands.py` combines both halves at frame weight and writes +`results/actual_pool_bands.json`. The figures are from the same fit as above +(8 trees, proxy AGI), before calibration and before main's capital-gains tail +stage (#567). "Old design" is the `current` variant; "#1033" is +`demographic_midrank6_earn`, the rank + flag design that #1033 merged. + +Clone 0 carries 50.0% of the tax-unit weight (`clone0_weight_share`) and has +no units at $5M+ proxy AGI in either variant: in both $5M+ bands, +`clone1_returns` equals `returns`. Its proxy AGI sums the 11 of the 15 items +that its person table carries; estate, farm, partnership and S-corporation +income are absent (`clone0_items_used`). + +| Band | Old design | #1033 | SOI Table 1.1 | +|---|---|---|---| +| $1M–1.5M returns | 701k (1.90×) | 483k (1.31×) | 369k | +| $5M–10M returns | 11.8k (0.24×) | 24.3k (0.49×) | 49.3k | +| $10M+ returns | 4.1k (0.13×) | 17.0k (0.56×) | 30.4k | +| $10M+ AGI | $92B (0.10×) | $519B (0.57×) | $908B | +| Colorado income above $1M | $20.2B (0.64×; largest record 28%) | $66.3B (2.1×; largest record 81%) | $31.7B (Historic Table 2, via #940) | + +"Largest record" is the share of the state's income above $1M carried by its +single largest contributor. + +#1033 roughly quadruples the pool's proxy AGI at $5M+ ($174B to $675B) and +lifts its $5M+ returns from 15.9k to 41.3k. On its own, though, it reaches +about half of SOI there (52% of the 79.6k returns and 54% of the $1,244B), not +SOI. At $5M+ the real-pool figures are exactly half of the ×2 ones (for +example 48,629 to 24,315 returns at $5M–10M), because only the PUF-clone half +has units there. The bands in between ($1.5M–5M) sit at 0.59–0.84× SOI in +both designs. At $1M–1.5M the survey half alone holds 284k returns, 77% of +SOI's count, in both designs. All five bands, with AGI and record counts, are +in `results/actual_pool_bands.json`. + +These figures correct the ×2 level comparisons first posted on #982, #940 and +#1033; a correction is posted on each. + ## Known limits -- Colorado (#940) is not fixed by this change. The chosen variant has 15 - Colorado records at $1M+ proxy AGI; the largest (survey six-item total - $1.1M, weight 867) draws $125.4M of proxy AGI and carries 84% of the state's - income above $1M (`colorado_1m_plus` in the scores), so state top-tail totals - are noisy until state x AGI-band amount targets (#940) enter calibration. +- Colorado (#940) is not fixed by this change. In the PUF-clone half the + chosen variant has 15 Colorado records at $1M+ proxy AGI. The largest + (survey six-item total $1.1M) draws $125.4M of proxy AGI at frame weight + 434, shown as 867 in the ×2 scores. In the real pool at frame weight, + Colorado's income above $1M is $66.3B, 2.1× SOI's $31.7B, and that one + record carries 81% of it. The ×2 scores put the total at $128.5B, about 4× + SOI, with that record at 84% (`colorado_1m_plus`); those are not levels. + State top-tail totals therefore stay noisy until state × AGI-band amount + targets (#940) enter calibration. - All variants ran at 8 trees because memory was constrained; production fits more trees. The comparison is like for like across variants. - The within-10% and wage-total rows are pre-calibration; calibration targets diff --git a/experiments/982-puf-self-prediction/actual_pool_bands.py b/experiments/982-puf-self-prediction/actual_pool_bands.py new file mode 100644 index 0000000000..551d29ff6c --- /dev/null +++ b/experiments/982-puf-self-prediction/actual_pool_bands.py @@ -0,0 +1,154 @@ +"""Real-pool proxy-AGI bands at frame weight (microcosm#982 correction). + +Usage:: + + python actual_pool_bands.py .parquet> + +``score_variants.py`` weights the PUF-clone half x2 ("as if this half were the +whole population"). The real pool is the survey half (clone index 0, survey +values, half the weight) plus the PUF-clone half (clone index 1, imputed +values, the other half), before main's capital-gains tail stage. This script +combines both halves at frame weight on the same proxy AGI and compares the +bands with SOI Table 1.1. The survey half sums whichever of the 15 proxy-AGI +items its person table carries; ``clone0_items_used`` in the output lists them. + +Writes ``actual_pool_bands.json`` into the input directory and prints a +summary. Inputs are restricted build artifacts on the author's machine; the +checkpoint path is recorded as provenance, not as a portable interface. +""" + +from __future__ import annotations + +import json +import os +import sys +import warnings + +import numpy as np +import pandas as pd +from score_variants import COLORADO_SOI_INCOME_ABOVE_1M_BN, INCOME, SOI_TABLE_1_1 + +from microcosm.build.frame_checkpoint import load_frame_checkpoint + +warnings.filterwarnings("ignore") + +CHECKPOINTS = os.path.expanduser( + "~/spm-rebuild-20260908/rollout/codex-continuation-20260912/" + "genuine-build-76325eb7-20260912/base-checkpoints" +) +VARIANTS = ("current", "demographic_midrank6_earn") +COLORADO_FIPS = 8 + + +def survey_half(frame) -> tuple[pd.DataFrame, float, list[str]]: + """Clone index 0 at frame weight: survey proxy AGI, weight and state per unit. + + Also returns clone 0's share of all tax-unit weight and the income items + found on the person table. + """ + + person, tax_unit = frame.table("person"), frame.table("tax_unit") + household = frame.table("household").copy() + household["w"] = frame.resolve_weights("household").values + clone = tax_unit["tax_unit_support_clone_index"].to_numpy() + have = [column for column in INCOME if column in person.columns] + survey = ( + person[have] + .astype(float) + .sum(axis=1) + .groupby(person["person_tax_unit_id"]) + .sum() + ) + unit_household = ( + person.groupby("person_tax_unit_id")["person_household_id"] + .first() + .reindex(tax_unit["tax_unit_id"]) + .to_numpy() + ) + by_household = household.set_index("household_id") + weight = by_household["w"].reindex(unit_household).to_numpy() + state = by_household["state_fips"].reindex(unit_household).to_numpy() + zero = clone == 0 + half = pd.DataFrame( + { + "agi": survey.reindex(tax_unit["tax_unit_id"][zero]).fillna(0).to_numpy(), + "w": weight[zero], + "state": state[zero], + "clone": 0, + } + ) + return half, float(weight[zero].sum() / weight.sum()), have + + +def pool_report(pool: pd.DataFrame) -> dict[str, object]: + """SOI Table 1.1 bands and Colorado's income above $1M for one pool.""" + + bands = {} + for band, (low, high, returns, soi_agi) in SOI_TABLE_1_1.items(): + mask = (pool.agi >= low) & (pool.agi < high) + bands[band] = { + "returns": float(pool.w[mask].sum()), + "soi_returns": returns, + "agi_bn": float((pool.w * pool.agi)[mask].sum() / 1e9), + "soi_agi_bn": soi_agi, + "records": int(mask.sum()), + "clone1_returns": float(pool.w[mask & (pool.clone == 1)].sum()), + } + colorado = pool.state == COLORADO_FIPS + above = pool.w * np.maximum(pool.agi - 1e6, 0) + return { + "bands": bands, + "colorado_income_above_1m_bn": float(above[colorado].sum() / 1e9), + "colorado_soi_bn": COLORADO_SOI_INCOME_ABOVE_1M_BN, + "colorado_largest_record_share": float( + above[colorado].max() / above[colorado].sum() + ), + } + + +def main() -> None: + directory = sys.argv[1] + frame = load_frame_checkpoint( + f"{CHECKPOINTS}/002_clone_feature_extraction.frame.h5" + ).frame + survey, survey_share, items = survey_half(frame) + out = { + "clone0_weight_share": survey_share, + "clone0_items_used": items, + "variants": {}, + } + for variant in VARIANTS: + imputed = pd.read_parquet(f"{directory}/imputed_{variant}.parquet") + clone1 = pd.DataFrame( + { + "agi": imputed[INCOME].sum(axis=1).to_numpy(), + "w": imputed["w"].to_numpy(), + "state": imputed["state_fips"].to_numpy(), + "clone": 1, + } + ) + pool = pd.concat([survey, clone1], ignore_index=True) + out["variants"][variant] = pool_report(pool) + with open(f"{directory}/actual_pool_bands.json", "w") as handle: + json.dump(out, handle, indent=1) + + print(f"clone0 weight share {out['clone0_weight_share']:.3f}") + for variant, report in out["variants"].items(): + print("==", variant) + for band, x in report["bands"].items(): + print( + f" {band:12} returns {x['returns']:>9,.0f} vs SOI " + f"{x['soi_returns']:>8,} ({x['returns'] / x['soi_returns']:.2f}x) " + f"AGI ${x['agi_bn']:6.0f}B vs ${x['soi_agi_bn']:4.0f}B " + f"records {x['records']:5} clone1 {x['clone1_returns']:>9,.0f}" + ) + print( + " Colorado income above $1M " + f"${report['colorado_income_above_1m_bn']:.1f}B vs SOI " + f"${report['colorado_soi_bn']:.1f}B; largest record share " + f"{report['colorado_largest_record_share']:.0%}" + ) + + +if __name__ == "__main__": + main() diff --git a/experiments/982-puf-self-prediction/results/actual_pool_bands.json b/experiments/982-puf-self-prediction/results/actual_pool_bands.json new file mode 100644 index 0000000000..5e7acd5a9a --- /dev/null +++ b/experiments/982-puf-self-prediction/results/actual_pool_bands.json @@ -0,0 +1,112 @@ +{ + "clone0_weight_share": 0.5, + "clone0_items_used": [ + "employment_income_before_lsr", + "self_employment_income_before_lsr", + "taxable_interest_income", + "qualified_dividend_income", + "non_qualified_dividend_income", + "short_term_capital_gains", + "long_term_capital_gains_before_response", + "taxable_private_pension_income", + "taxable_ira_distributions", + "rental_income", + "miscellaneous_income" + ], + "variants": { + "current": { + "bands": { + "1m_to_1_5m": { + "returns": 701147.4560873657, + "soi_returns": 368931, + "agi_bn": 820.1167999742612, + "soi_agi_bn": 443.658, + "records": 1751, + "clone1_returns": 417016.76188669726 + }, + "1_5m_to_2m": { + "returns": 123041.63164029716, + "soi_returns": 147290, + "agi_bn": 210.17016121755844, + "soi_agi_bn": 252.903, + "records": 289, + "clone1_returns": 99921.38839026116 + }, + "2m_to_5m": { + "returns": 119112.43471266434, + "soi_returns": 203229, + "agi_bn": 314.19123083447545, + "soi_agi_bn": 602.653, + "records": 284, + "clone1_returns": 100370.2883022731 + }, + "5m_to_10m": { + "returns": 11800.218136098976, + "soi_returns": 49262, + "agi_bn": 81.29907128772709, + "soi_agi_bn": 336.335, + "records": 31, + "clone1_returns": 11800.218136098976 + }, + "10m_plus": { + "returns": 4093.195927116907, + "soi_returns": 30382, + "agi_bn": 92.27042532710882, + "soi_agi_bn": 907.901, + "records": 9, + "clone1_returns": 4093.195927116907 + } + }, + "colorado_income_above_1m_bn": 20.21137092030577, + "colorado_soi_bn": 31.7, + "colorado_largest_record_share": 0.28371840606041343 + }, + "demographic_midrank6_earn": { + "bands": { + "1m_to_1_5m": { + "returns": 482653.3947451953, + "soi_returns": 368931, + "agi_bn": 565.6567147347971, + "soi_agi_bn": 443.658, + "records": 1221, + "clone1_returns": 198522.7005445268 + }, + "1_5m_to_2m": { + "returns": 96760.34899440594, + "soi_returns": 147290, + "agi_bn": 164.41986705107027, + "soi_agi_bn": 252.903, + "records": 243, + "clone1_returns": 73640.10574436994 + }, + "2m_to_5m": { + "returns": 125304.60878687656, + "soi_returns": 203229, + "agi_bn": 353.22466083387354, + "soi_agi_bn": 602.653, + "records": 318, + "clone1_returns": 106562.4623764853 + }, + "5m_to_10m": { + "returns": 24314.73234462075, + "soi_returns": 49262, + "agi_bn": 156.12486777592076, + "soi_agi_bn": 336.335, + "records": 52, + "clone1_returns": 24314.73234462075 + }, + "10m_plus": { + "returns": 16959.818039246165, + "soi_returns": 30382, + "agi_bn": 518.7226807804055, + "soi_agi_bn": 907.901, + "records": 38, + "clone1_returns": 16959.818039246165 + } + }, + "colorado_income_above_1m_bn": 66.3266036367742, + "colorado_soi_bn": 31.7, + "colorado_largest_record_share": 0.8131946934821087 + } + } +} \ No newline at end of file From bc2a37574aa4ba77688a77be4d7ddddc981380bb Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 27 Sep 2026 06:35:18 -0400 Subject: [PATCH 2/3] =?UTF-8?q?State=20the=20$1.5M-5M=20range=20for=20retu?= =?UTF-8?q?rns=20and=20AGI;=20mark=20=C3=972=20wage=20totals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-up (independent Opus review of #1037 approved; one minor and two nits taken): the 0.59-0.84x range held for returns only, so the README now gives AGI too (0.52-0.83x); "roughly quadruples" becomes "nearly quadruples (3.9x)"; the wage totals in the prose are marked x2. Co-Authored-By: Claude Opus 5.5 --- experiments/982-puf-self-prediction/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/experiments/982-puf-self-prediction/README.md b/experiments/982-puf-self-prediction/README.md index 2acab4d321..b038e22fb1 100644 --- a/experiments/982-puf-self-prediction/README.md +++ b/experiments/982-puf-self-prediction/README.md @@ -86,8 +86,8 @@ ranks level with low but positive PUF incomes; rank alone handed positive wages to 47.9% (weighted) of survey units with no wages or self-employment income. Why pooled rather than within-group ranks: ranking earners only among earners matched survey earners to a PUF earner distribution that includes many small separate returns (for example dependents filing their -own), cutting imputed wages to $7.93T; the pooled rank keeps $9.68T with the -same participation fidelity. +own), cutting imputed wages to $7.93T (×2); the pooled rank keeps $9.68T (×2) +with the same participation fidelity. ## Real pool at frame weight @@ -118,13 +118,13 @@ income are absent (`clone0_items_used`). "Largest record" is the share of the state's income above $1M carried by its single largest contributor. -#1033 roughly quadruples the pool's proxy AGI at $5M+ ($174B to $675B) and -lifts its $5M+ returns from 15.9k to 41.3k. On its own, though, it reaches -about half of SOI there (52% of the 79.6k returns and 54% of the $1,244B), not +#1033 nearly quadruples (3.9×) the pool's proxy AGI at $5M+ ($174B to $675B) +and lifts its $5M+ returns from 15.9k to 41.3k. On its own, though, it +reaches about half of SOI there (52% of the 79.6k returns and 54% of the $1,244B), not SOI. At $5M+ the real-pool figures are exactly half of the ×2 ones (for example 48,629 to 24,315 returns at $5M–10M), because only the PUF-clone half has units there. The bands in between ($1.5M–5M) sit at 0.59–0.84× SOI in -both designs. At $1M–1.5M the survey half alone holds 284k returns, 77% of +returns and 0.52–0.83× in AGI, in both designs. At $1M–1.5M the survey half alone holds 284k returns, 77% of SOI's count, in both designs. All five bands, with AGI and record counts, are in `results/actual_pool_bands.json`. From 094aaa91c3600976d249437a95ea6604397e7d97 Mon Sep 17 00:00:00 2001 From: Disk manager Date: Mon, 9 Nov 2026 08:10:41 -0500 Subject: [PATCH 3/3] State that #1102 bound the #940 targets and scope the real-pool figures The Colorado limit said state top-tail totals stay noisy until the #940 state x AGI-band targets enter calibration; #1102 added them on 2026-10-11. The scope sentence now names #1251 and #1259, which also postdate this fit. Smaller wording fixes from the same review: the script writes into its input directory, the four absent items are columns of the 15-item list, the $1.5M-5M ranges span the two designs, and two passives become active. Co-Authored-By: Claude Opus 5.5 --- changelog.d/982-real-pool-bands.fixed.md | 2 +- experiments/982-puf-self-prediction/README.md | 25 +++++++++++-------- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/changelog.d/982-real-pool-bands.fixed.md b/changelog.d/982-real-pool-bands.fixed.md index 146a6e08b7..2a0679562d 100644 --- a/changelog.d/982-real-pool-bands.fixed.md +++ b/changelog.d/982-real-pool-bands.fixed.md @@ -1 +1 @@ -Corrected the #982 experiment README: its PUF-clone-half ×2 tables are now labeled distribution checks, and a new section scores the real pool at frame weight, which before calibration reaches about half of SOI at $5M+ proxy AGI after #1033. +Corrected the #982 experiment README: it now labels its PUF-clone-half ×2 tables as distribution checks, and a new section scores the real pool at frame weight, which before calibration reaches about half of SOI at $5M+ proxy AGI after #1033. diff --git a/experiments/982-puf-self-prediction/README.md b/experiments/982-puf-self-prediction/README.md index b038e22fb1..cacedc81e2 100644 --- a/experiments/982-puf-self-prediction/README.md +++ b/experiments/982-puf-self-prediction/README.md @@ -96,16 +96,20 @@ each unit's weight evenly between two clone channels. Clone index 0, the native survey channel, keeps the survey values; clone index 1 (`PUF_TAX_DETAIL_CLONE_INDEX`) receives the imputed ones. `actual_pool_bands.py` combines both halves at frame weight and writes -`results/actual_pool_bands.json`. The figures are from the same fit as above -(8 trees, proxy AGI), before calibration and before main's capital-gains tail -stage (#567). "Old design" is the `current` variant; "#1033" is -`demographic_midrank6_earn`, the rank + flag design that #1033 merged. +`actual_pool_bands.json` into its input directory; +`results/actual_pool_bands.json` holds the recorded copy. The figures come +from the same fit as above (8 trees, proxy AGI), before calibration and before +main's capital-gains tail stage (#567). The fit also predates #1251 (the +donor's partnership and S-corporation income moved to Schedule E line 32) and +#1259 (a bound on each QRF draw), and these figures reflect neither. "Old +design" is the `current` variant; "#1033" is `demographic_midrank6_earn`, the +rank + flag design that #1033 merged. Clone 0 carries 50.0% of the tax-unit weight (`clone0_weight_share`) and has no units at $5M+ proxy AGI in either variant: in both $5M+ bands, `clone1_returns` equals `returns`. Its proxy AGI sums the 11 of the 15 items -that its person table carries; estate, farm, partnership and S-corporation -income are absent (`clone0_items_used`). +that its person table carries; the 15-item list's estate, farm, partnership +and S-corporation columns are absent (`clone0_items_used`). | Band | Old design | #1033 | SOI Table 1.1 | |---|---|---|---| @@ -124,12 +128,12 @@ reaches about half of SOI there (52% of the 79.6k returns and 54% of the $1,244B SOI. At $5M+ the real-pool figures are exactly half of the ×2 ones (for example 48,629 to 24,315 returns at $5M–10M), because only the PUF-clone half has units there. The bands in between ($1.5M–5M) sit at 0.59–0.84× SOI in -returns and 0.52–0.83× in AGI, in both designs. At $1M–1.5M the survey half alone holds 284k returns, 77% of +returns and 0.52–0.83× in AGI across the two designs. At $1M–1.5M the survey half alone holds 284k returns, 77% of SOI's count, in both designs. All five bands, with AGI and record counts, are in `results/actual_pool_bands.json`. These figures correct the ×2 level comparisons first posted on #982, #940 and -#1033; a correction is posted on each. +#1033; each of those threads carries a correction. ## Known limits @@ -140,8 +144,9 @@ These figures correct the ×2 level comparisons first posted on #982, #940 and Colorado's income above $1M is $66.3B, 2.1× SOI's $31.7B, and that one record carries 81% of it. The ×2 scores put the total at $128.5B, about 4× SOI, with that record at 84% (`colorado_1m_plus`); those are not levels. - State top-tail totals therefore stay noisy until state × AGI-band amount - targets (#940) enter calibration. + #1102 later added state × AGI-band return and amount targets (#940) to the + US calibration target surface; this pre-calibration run predates them and + shows nothing about a build that binds them. - All variants ran at 8 trees because memory was constrained; production fits more trees. The comparison is like for like across variants. - The within-10% and wage-total rows are pre-calibration; calibration targets