Derive US local-jurisdiction flags (Denver, Yonkers, Wilmington, Philadelphia) from household geography

changelog.d/us-local-jurisdiction-flags.added.md

@@ -0,0 +1 @@
+Derive the policyengine-us input-only local flags `in_denver`, `in_yonkers`, `in_wilmington` and `pa_philadelphia_wage_tax_resident` from household geography through a SHA-pinned registry checked against the 2020 Gazetteer, P.L. 94-171 counts and the block ladder. The block-ladder line writes all four; the PUMA and pooled ACS lines write the county-rule flags and record the place-rule flags (Yonkers, Wilmington) as unavailable instead of writing `False`. The L0/refit export and the fiscal refresh release builder re-derive the flags from an existing base's own geography, and the builder derives them again after frozen-support selection and on the export support, so a base built before this change, or a selection that keeps only block-anchored rows, no longer exports without them. Both ladder gates now fail a table that lacks a flag its geography decides. The annual static-aging lane, which keeps its parent's columns and base-year bytes, now refuses a parent that lacks or contradicts a flag its geography decides, before any model work, and records the parent's flag receipt in the candidate manifest.

docs/evidence/spec-engine/us-f0-coverage.json

@@ -1642,13 +1642,13 @@
           "compiler_ir.node_slices"
         ],
         "expected": {
-          "map_sha256": "07fe9086b57a15ab1c2551fce5c30aebfa55175d76a308f7dbb268ac49da747a",
-          "protocol_sha256": "4ff992324004d4c09d4165bc8d24ccff15b9d84858836851b12d532c870920f4"
+          "map_sha256": "19daf13d0415792816c230e775de357c878091a628fd02bbe4184159db356f90",
+          "protocol_sha256": "c1b42729c9b655fc599c39a55a66fdf71d3fd5eba64bc3c1d37cb09a5ff09596"
         },
         "failures": [],
         "observed": {
-          "map_sha256": "07fe9086b57a15ab1c2551fce5c30aebfa55175d76a308f7dbb268ac49da747a",
-          "protocol_sha256": "4ff992324004d4c09d4165bc8d24ccff15b9d84858836851b12d532c870920f4"
+          "map_sha256": "19daf13d0415792816c230e775de357c878091a628fd02bbe4184159db356f90",
+          "protocol_sha256": "c1b42729c9b655fc599c39a55a66fdf71d3fd5eba64bc3c1d37cb09a5ff09596"
         },
         "status": "covered"
       },
@@ -1663,7 +1663,7 @@
           "compiler_ir.seed_stream_map"
         ],
         "expected": {
-          "implementation_sha256": "4ff992324004d4c09d4165bc8d24ccff15b9d84858836851b12d532c870920f4",
+          "implementation_sha256": "c1b42729c9b655fc599c39a55a66fdf71d3fd5eba64bc3c1d37cb09a5ff09596",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1684,7 +1684,7 @@
         },
         "failures": [],
         "observed": {
-          "implementation_sha256": "4ff992324004d4c09d4165bc8d24ccff15b9d84858836851b12d532c870920f4",
+          "implementation_sha256": "c1b42729c9b655fc599c39a55a66fdf71d3fd5eba64bc3c1d37cb09a5ff09596",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1873,7 +1873,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "69c2e4fcc8ebdf1e6257ccaf4787b736c4822beda5f4ee01a53ffa3945f4936f"
+          "sha256": "94ed381f17dc9cafb80396b2ea0f02ed8745d9359ec3f4d551a65a5a7f2fc26d"
         },
         "failures": [],
         "observed": {
@@ -1897,7 +1897,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "69c2e4fcc8ebdf1e6257ccaf4787b736c4822beda5f4ee01a53ffa3945f4936f"
+          "sha256": "94ed381f17dc9cafb80396b2ea0f02ed8745d9359ec3f4d551a65a5a7f2fc26d"
         },
         "status": "covered"
       },
@@ -1961,7 +1961,7 @@
             "take_up_contract",
             "us_qbi_reconciliation_contract"
           ],
-          "sha256": "7248c2831f3772e75f22dbabcbb5794d5d81138009a38aae8410e43aa05fb656"
+          "sha256": "e31727246ecd547071778fb91c2f6b102cf024f3c0cc64252bbc8040b9fb1201"
         },
         "status": "covered"
       },
@@ -2585,7 +2585,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "0c713aabfd0afc3703415fea86a8210a3f4346918f774f371fb16aa663348c1d"
+      "spec_sha256": "66507c9d0a09b288ad118424d8256da4ab914d1e957ba2ed653fe26e83a482a8"
     }
   },
   "report_schema_version": 3,
@@ -2595,7 +2595,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "0c713aabfd0afc3703415fea86a8210a3f4346918f774f371fb16aa663348c1d"
+    "spec_sha256": "66507c9d0a09b288ad118424d8256da4ab914d1e957ba2ed653fe26e83a482a8"
   },
   "status": "pass"
 }

docs/us-annual-static-aging.md

@@ -41,12 +41,23 @@ fields. Fixed-format inputs, including missing nullable booleans that require
 that storage format, are rejected before building and need a separate certified
 layout contract.
 
+Because the lane adds no column, the parent must already carry every
+local-jurisdiction flag its own geography decides (see
+`packages/microcosm-build/src/microcosm/build/us_runtime/US_LOCAL_JURISDICTIONS.md`):
+`in_denver` and `pa_philadelphia_wage_tax_resident` when every household has a
+county, and also `in_yonkers` and `in_wilmington` when every household has a
+place and a block. The builder reads the parent first and refuses one that lacks
+such a flag, or carries one that disagrees with its geography, before any model
+or output work. Rebuild a parent that predates the flags before it seeds annual
+files.
+
 The output directory must be new. A failed attempt retains its partial files and
 `build_status.json`; rerunning requires another directory. `annual_manifest.json`
 is written last, after all years finish and source/input hashes are rechecked.
 It contains:
 
-- `base`: source dataset, year, parent release claim, path, and SHA256.
+- `base`: source dataset, year, parent release claim, path, SHA256, and the
+  parent's local-jurisdiction flag receipt (`local_jurisdiction_flags`).
 - `inputs`, `model`, and `runtime`: SSA pin, model commit and source hashes, actual
   implementation hashes, and dependency versions.
 - `metadata.dataset_years`: a map from the source dataset to year-keyed annual

packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py

@@ -350,7 +350,7 @@
     "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25",
     "authority": "bf681aedc859fd6398ef72da68f584069b7d25b3085c549bf99f82e8324c4eef",
     "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd",
-    "full_checkpoint": "69c2e4fcc8ebdf1e6257ccaf4787b736c4822beda5f4ee01a53ffa3945f4936f",
+    "full_checkpoint": '94ed381f17dc9cafb80396b2ea0f02ed8745d9359ec3f4d551a65a5a7f2fc26d',
     "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7",
     "graph_nodes": "1b208fff4f6d878c522147fee617177544bda8963c940f98a4bfc37507cb955f",
     "geography_assignment": "44e7f5f6cd7ceeef85d532c3d46306d03de4f4ee811376e6d29920742746b65c",
@@ -379,9 +379,12 @@
     # gains the passive partnership/S-corp leaf, so the PUF QRF targets change.
     # microcosm#1259's recipient-share bound in puf_support moves both again.
     # microcosm#1261 moves both again: the PUF donor carries partnership and
-    # S-corp line-32 shares as separate leaves.
-    "seed_map": '07fe9086b57a15ab1c2551fce5c30aebfa55175d76a308f7dbb268ac49da747a',
-    "seed_protocol": '4ff992324004d4c09d4165bc8d24ccff15b9d84858836851b12d532c870920f4',
+    # S-corp line-32 shares as separate leaves. microcosm#1444 moves both:
+    # housing_inputs puts each ACS rent draw on its own household's head.
+    # microcosm#1453 moves both again: geography_ladder and puma_ladder write
+    # the local-jurisdiction flags.
+    "seed_map": '19daf13d0415792816c230e775de357c878091a628fd02bbe4184159db356f90',
+    "seed_protocol": 'c1b42729c9b655fc599c39a55a66fdf71d3fd5eba64bc3c1d37cb09a5ff09596',
     "source_manifest": "608c3621b1e4a108e57c8f83d16a184acdb3d05e16a81b895f7f72dbf5d0685c",
     "take_up": "9522ce40f7dea569312dd7a7beb474e5a5afadd2f1e10cb5876534c3ef623d35",
     "tail": "071d418bb49976f4682c320d2ad07097c177de3a645d2524f235e1ff33ac83f9",

packages/microcosm-build/src/microcosm/build/us_annual_static_aging.py

@@ -3,6 +3,9 @@
 Each projection starts from the same base frame. Only one projected year is
 held in memory at a time. The original H5 is copied byte for byte, and all later
 files retain its entity/column/row layout with an explicit ``_time_period``.
+No file adds a column, so the parent must already carry every local-jurisdiction
+flag its own geography decides; the lane refuses a parent that lacks one before
+any model work.
 Run ``python -m microcosm.build.us_annual_static_aging --help``.
 """
 
@@ -54,6 +57,32 @@ def _check_pin(path: Path, expected: str) -> None:
         raise ValueError(f"SHA256 mismatch for {path}")
 
 
+def _parent_local_jurisdiction_flags(frame: Frame) -> dict:
+    """Require the parent to carry its local-jurisdiction flags, with a receipt.
+
+    The lane copies the parent's base year byte for byte and keeps the
+    parent's columns in every projected year, so it cannot write a flag the
+    parent lacks: the engine would default it to ``False`` for every resident
+    in every year.
+    """
+
+    # Imported here so that importing this module (for --help, or for the
+    # serializer registry's _write_year) does not load the us_runtime package.
+    from microcosm.build.us_runtime.local_jurisdictions import (
+        us_local_jurisdiction_artifact_receipt,
+    )
+
+    try:
+        return us_local_jurisdiction_artifact_receipt(frame)
+    except ValueError as error:
+        raise ValueError(
+            "The annual lane keeps its parent's columns and base-year bytes, so "
+            "it needs a parent that already carries every local-jurisdiction "
+            "flag its own geography decides; rebuild the parent with the flags. "
+            f"{error}"
+        ) from error
+
+
 def _model_provenance(*, source: Path, commit: str, version: str) -> dict:
     import policyengine_us
 
@@ -238,13 +267,8 @@ def build_annual_static_aging(
     ssa = Path(ssa_csv).expanduser().absolute()
     _check_pin(base, base_sha256)
     _check_pin(ssa, ssa_sha256)
-    model_kwargs = {
-        "source": Path(model_source),
-        "commit": model_commit,
-        "version": model_version,
-    }
-    model = _model_provenance(**model_kwargs)
-    runtime = _runtime_provenance()
+    # Read the parent before any model work, so a parent this lane cannot
+    # carry forward fails first.
     with pd.HDFStore(base, "r") as store:
         _period(store, base_year)
         tables = {
@@ -261,6 +285,14 @@ def build_annual_static_aging(
     frame = Frame(
         tables, US_SCHEMA, {"household": Weights(weights, WeightKind.CALIBRATED)}
     )
+    local_jurisdiction_flags = _parent_local_jurisdiction_flags(frame)
+    model_kwargs = {
+        "source": Path(model_source),
+        "commit": model_commit,
+        "version": model_version,
+    }
+    model = _model_provenance(**model_kwargs)
+    runtime = _runtime_provenance()
     base_tables = {
         entity: table.loc[:, columns[entity]]
         for entity, table in engine_tables(
@@ -297,6 +329,7 @@ def build_annual_static_aging(
             "parent_release": parent_release,
             "path": str(base),
             "sha256": base_sha256,
+            "local_jurisdiction_flags": local_jurisdiction_flags,
         },
         "inputs": {"ssa": {"path": str(ssa), "sha256": ssa_sha256}},
         "model": model,

packages/microcosm-build/src/microcosm/build/us_runtime/US_LOCAL_JURISDICTIONS.md

@@ -0,0 +1,121 @@
+# US local-jurisdiction flags
+
+policyengine-us 2.2.1 (and 2.60.4) declares four local-tax jurisdictions as
+plain inputs with no formula. A flag stays `False` unless the dataset writes
+it:
+
+| Flag | Entity | Rule | 2020 geography | Population | Engine consumer |
+| --- | --- | --- | --- | ---: | --- |
+| `in_denver` | household | county | county 08031 (same blocks as Denver city, 08/20000) | 715,522 | Denver property-tax relief eligibility |
+| `in_yonkers` | household | place | NY place 84000 (Yonkers city) | 211,569 | `ny_yonkers_income_tax` |
+| `in_wilmington` | household | place | DE place 77580 (Wilmington city) | 70,898 | `de_wilmington_earned_income_tax` |
+| `pa_philadelphia_wage_tax_resident` | person | county | county 42101 (same blocks as Philadelphia city, 42/60000) | 1,603,797 | `pa_philadelphia_wage_tax` |
+
+Local flags that have a formula on `county_fips` (`in_nyc`, `in_la` and the
+rest) need nothing: the engine computes them from the county the geography
+line writes.
+
+## Registry
+
+`us_local_jurisdictions.json` holds the table above, and
+`local_jurisdictions.py` pins its SHA-256. A county rule holds when
+`county_fips` equals the registry county. A place rule holds when the first two
+digits of `county_fips` equal the registry state and `place_fips` equals the
+registry place code; place codes repeat across states, so the state is part of
+the match. A person flag takes its household's value.
+
+`tools/verify_us_local_jurisdiction_registry.py` checks the registry against
+three Census sources:
+
+- the 2020 Gazetteer place and county files (GEOID, ANSI code, name, land and
+  water area; a county-rule jurisdiction's county and city have identical
+  areas);
+- the 2020 P.L. 94-171 `P1_001N` count for each place and county; and
+- any block ladder: the blocks whose household would get a flag sum to the
+  registry population, and a county-rule flag covers the same blocks as its
+  city.
+
+`block_ladder_checks` records the result for ladders `7ba39b95` and `6840b990`.
+With `--record` the tool writes a ladder's counts into the registry and prints
+the new SHA-256 to pin.
+
+## Geography lines
+
+- **Block ladder** (`with_household_us_geography_ladder`): one block fixes
+  county and place, so the line writes all four flags.
+- **Block-location draw** (microcosm#1047): call
+  `with_us_local_jurisdiction_flags(frame, rungs=BLOCK_GEOGRAPHY_RUNGS)` once
+  `county_fips` and `place_fips` are final.
+- **PUMA ladder** (`with_household_us_puma_ladder`, the stacked pool): the
+  ladder draws a county but no place. The line writes `in_denver` and the
+  Philadelphia flag and never writes `in_yonkers` or `in_wilmington`; it also
+  removes a stale column. The assignment summary records those two as
+  `unavailable`, and the engine ABI lock lists them as
+  `unavailable_place_jurisdiction_input`.
+- **Pooled ACS line** (`with_optional_acs_spine`): donor rows arrive with
+  block-line flags and ACS rows without. The pool re-derives the county-rule
+  flags for every row from the final county and removes the place-rule flags.
+  Without a PUMA ladder it removes every flag, because ACS rows then have no
+  county. `build_optional_acs_multispine` records the pool's receipt as
+  `provenance["local_jurisdiction_flags"]` on both paths.
+
+Missing, null or malformed geography raises; no line writes `False` for a
+household whose county or place it does not know.
+
+## Existing artifacts
+
+A base H5 built before this registry carries no flags. Two steps start from
+an existing artifact rather than a geography line, and both call
+`rederive_us_local_jurisdiction_flags`:
+
+- the L0/refit export (`export_us_l0_refit_h5`) calls it on the selected
+  support and records the receipt as `local_jurisdiction_flags` in its
+  summary; and
+- the fiscal refresh release builder calls it on the loaded base or prepared
+  pool, again on the frozen support when a selection source reduces it, and
+  again on the export support after the L0 or full-pool weights attach. It
+  records each receipt, keyed `base`, `frozen_support` and `export`, under
+  `support_value_repairs.local_jurisdiction_flags` in the calibration
+  diagnostics.
+
+A third step, the annual static-aging lane (`us_annual_static_aging`), keeps its
+parent's columns and base-year bytes, so it cannot write the flags. It calls
+`us_local_jurisdiction_artifact_receipt` instead, which refuses a parent that
+lacks a flag its geography decides or carries one that disagrees with it, and
+records the receipt as `base.local_jurisdiction_flags` in the candidate
+manifest. A parent built before the flags existed therefore cannot seed annual
+files until a rebuild adds them.
+
+The helper reads the rungs from the artifact's own geography
+(`us_local_jurisdiction_rungs_present`): the county rung when `county_fips` is
+on every row, and the place rung only when every row also has `place_fips` and
+a 15-digit `block_geoid`. A pooled artifact whose ACS rows have no block
+therefore gets the county-rule flags only. A flag the artifact already carries
+must agree with its geography (household and person flags both), or the step
+fails.
+
+The rungs belong to the whole table, so removing rows can add one: a frozen
+support or L0 support that keeps only block-anchored donors decides
+`in_yonkers` and `in_wilmington` even though the pooled base did not. That is
+why the builder derives the flags again after each step that removes rows.
+Deriving a support that already carries exactly the flags it decides returns
+the same frame, so these repeat calls copy no tables.
+
+## Gates
+
+Both ladder gates fail:
+
+- a flag column that disagrees with its rule;
+- a null or non-boolean flag;
+- a place-rule flag on a table with no complete `place_fips`; and
+- a missing flag that the line's own rung decides. The block-ladder gate
+  requires `in_denver`, `in_yonkers` and `in_wilmington`; the PUMA-ladder gate
+  requires `in_denver`.
+
+The gates see only the household table, so they check the household flags.
+`us_local_jurisdiction_flag_failures(household, person=...)` also checks that
+each person carries its household's value.
+
+The Philadelphia flag records residence only. `pa_philadelphia_wage_tax` also
+needs `pa_philadelphia_wage_tax_taxable_wages`, which no line supplies yet, so
+the tax stays zero in built datasets.

packages/microcosm-build/src/microcosm/build/us_runtime/US_PUMA_LADDER.md

@@ -26,6 +26,11 @@ from its `seed`.
 The dense ACS-spine artifact is filtered to state / congressional district /
 county for local analysis because every record carries all three.
 
+The ladder also writes the county-rule local-jurisdiction flags (`in_denver`,
+`pa_philadelphia_wage_tax_resident`) from the drawn county. It has no place
+rung, so it never writes `in_yonkers` or `in_wilmington`, and its summary
+records them as unavailable ([US_LOCAL_JURISDICTIONS.md](US_LOCAL_JURISDICTIONS.md)).
+
 ## Why one block pass
 
 2020 tabulation blocks nest in 2020 census tracts, and 2020 tracts nest in 2020