Allocate Census residuals through shared reference profiles

changelog.d/nz-census-residual-profiles.added.md

@@ -0,0 +1,5 @@
+Add shared, validated proportional residual allocation and reference-profile
+resolution. New Zealand's d1886 Census size and tenure profiles retain regional
+totals, disclose computed residuals, and add the V4 small-household sensitivity.
+Validated roster preparation supplies size indicators and binds preparation to
+the effective reference and interval identities.

docs/nz-calibration-targets.md

@@ -36,6 +36,25 @@ Ranked by calibration value for a dollar-level take-up estimate. **$** = enables
 | 14 | **NZ Superannuation demographics** | MSD Fact Sheets (NZS/VP) + Annual Report (941,511 recipients; $23.19B) | $ / # | Largest transfer by dollars; near-universal but needs the age×region recipient frame. [FETCH-OK] |
 | 15 | **Minimum wage series + average weekly earnings** | MBIE minimum wage ($23.95 from Apr 2026) + Stats NZ QES/LEED | $ | Anchors the bottom of the earnings distribution and the national/TA earnings margin. [FETCH-OK min wage; SNIPPET QES/LEED] |
 
+**d1886 modelling convention, 23 October 2026.** Microcosm allocates Census
+counts with unstated household size across stated size categories in proportion
+to their stated counts, assuming missing at random. Each region's allocated
+amount is its independently published household total minus its stated size
+leaves. The national calculation is 61,725 households; regional calculations sum
+to 61,713 because the independently published cells do not reconcile exactly.
+These regional differences and allocated leaves are Microcosm calculations,
+rather than Stats NZ published residual cells or allocated estimates.
+
+Tenure uses the same proportional convention for owned, including family
+trusts, and not owned. Its national difference is 426; regional differences sum
+to 429. V4 allocates the size residual only to one- and two-person households in
+their stated proportions. Fractional targets are retained. Allocation precedes
+one application of the B2 ERP bridge. Region 99 remains separate and uses the
+disclosed national-growth proxy. Raw discrepancies and national diagnostics
+remain in `reference_inventory.json` alongside computed regional profile
+receipts. Model household size counts validated usual-resident roster members,
+with sizes eight and above sharing the final category.
+
 **Dollar-level calibration design (numerator/denominator):**
 - **Denominator (full-entitlement modelled $):** our own rules × calibrated population. Treasury DistributionExplorer CSVs (#5) are TAWA-modelled; AS values assigned by TAWA's take-up model; an independent comparison, not full entitlement, for AS; TAWA assumes full take-up for the WFF credits ([TAWA methodology](https://treasury.govt.nz/sites/default/files/2024-08/tawa-model-methodology.pdf), "Modelling take-up").
 - **Numerator (actual $ paid):** MSD Annual Report expenditure schedule (#2) and AN24/01 fiscal totals (#10); IRD WfF expenditure (#4) for in-work credits.

docs/nz-take-up-engine-plan.md

@@ -16,14 +16,17 @@ Support Survey 2022, findings pack 7); an IDI-based estimate puts Working for
 Families (WFF) take-up at about 87% in 2020 (McLeod and Wilson 2022, targets
 inventory M5).
 
-## Status: 7 October 2026
+## Status: 23 October 2026
 
 The spec-only [NZ country package](../packages/microcosm-build/src/microcosm/build/nz/country_package.json)
-compiles through the shared country seam. It is a declared layout, not a
-calibrated population or a release: every target reference is an unactivated
-placeholder, the threshold-bearing gates read "awaiting D1", and no NZ graph
-node exists yet. Donor records are US support records and are never presented
-as New Zealand microdata.
+compiles through the shared country seam. It declares active demographic
+references and profile-resolved Census size and tenure references, with other
+unresolved target definitions parked. The release gates carry reviewed
+thresholds (d1376: every calibration target within 5%, demography within 2%,
+administrative anchors within 5%); no production fit evidence exists for them
+yet, and graph assembly and calibrated population evidence remain separate.
+Donor records are US support records and are never presented as New Zealand
+microdata.
 
 The package replaces the stage-era scaffold of
 [#821](https://github.com/PolicyEngine/microcosm/pull/821). It keeps #821's
@@ -62,7 +65,7 @@ nz.create (CREATE) ── nz.open (FILTER keep-all: transport, geography, receip
                          │      │      └── nz.scn.S0 … nz.scn.S5 (FILTER each)
                          │      ├── nz.validate (FILTER: WFF tripwire, parked)
                          │      └── nz.terminal (FILTER: bands, diagnostics, gates, export)
-                         └── nz.v.V1 … nz.v.V3 (FILTER each: variant → calibrate → AS → gap)
+                         └── nz.v.V1 … nz.v.V4 (FILTER each: variant → calibrate → AS → gap)
 ```
 
 - **CREATE is to build the benefit units.** Entity ids and person
@@ -97,8 +100,9 @@ nz.create (CREATE) ── nz.open (FILTER keep-all: transport, geography, receip
 | `as_area_crosswalk.json` | Work and Income area definitions; the TA → AS area population shares are a placeholder (`rows: []`, `status: requires_harvest`) until harvested | `nz.geo.support` | MC9 |
 | `axiom_rules_bindings.json` | rulespec-nz pin, five module bindings, variables, the `2026-27` tax-year period | every Axiom-bound node | MC1 |
 | `target_references.json` | the calibration set (MC5 list), generated cell-pinned by `tools/generate_nz_target_references.py` and summarized in `reference_inventory.json`; unresolved definitions stay parked with named reasons | `nz.targets.compile` | MC5 |
+| `reference_profiles.json`, `calibration_bindings.json` | d1886 MAR and small-household residual declarations, plus validated usual-resident roster counts and all 136 region-by-size indicators; the shared resolver produces effective references before activation | reference resolution and calibration preparation | MC5 |
 | `as_rate_bridge.json` | reg 17 base rate and reg 18 cutout composition | `nz.as.bridge.*` | MC11 |
-| `scenarios.json` | S0–S5, V1–V3, knobs, gap conventions (×52) | scenario and variant branches | MC3, MC12 |
+| `scenarios.json` | S0–S5, V1–V4, knobs, gap conventions (×52); V4 selects the small-household residual sensitivity | scenario and variant branches | MC3, MC12 |
 | `holdout_references.json` | AS comparators and the WFF tripwire | gap and validation nodes only | MC2, MC5 |
 | `gates.json` | the gate battery, evaluated with the shared transport bindings (`transport/gate_bindings.py`); fit and admin-aggregate tolerances follow d1376 (each target within 5%, demography within 2%, administrative anchors within 5%); support and release-input coverage remain applicable and block builds until their producers and evidence checks exist | `nz.gates.calibrated` | MC15 |
 | `export_contract.json` | the closed export contract | export nodes | — |
@@ -111,12 +115,23 @@ transport.
 ### Reference sets
 
 - **Calibration (MC5):** active are region × age population (all sexes, four
-  age bands) and tenure × region (owned, including family trusts, and not
-  owned). Census 2023 tenure is scaled to June 2025 by each region's ERP ratio,
-  and region 99 by the national ratio. Declared but parked until their
-  definitions or producers exist: Census household size and family type, IRD
+  age bands). Size × region and tenure × region (owned, including family
+  trusts, and not owned) require the d1886 profile resolver before execution.
+  Their allocated Census 2023 counts receive each region's June 2025 ERP ratio
+  once; region 99 uses the national-growth proxy. Declared but parked until
+  their definitions or producers exist: Census family type, IRD
   taxable-income bands (counts and totals), MSD main-benefit recipients, and NZ
   Super and Veterans Pension recipients (counted separately).
+
+G8-r1b implements Max's d1886 residual-allocation convention through shared
+target arithmetic and the common reference-profile resolver. Central size and
+tenure targets close to each region's published Census household total before
+the existing ERP bridge. Calibration variant V4 concentrates only the size
+residual onto one- and two-person households. The implementation preserves raw
+discrepancies and independent national diagnostics, and supplies validated size
+columns before activation. G8-r2 reuses the same resolver for ageing profiles
+and interval preparation; central ageing remains explicitly deferred and V1/V2
+ageing transformations remain unavailable until that work lands.
 - **Pre-calibration:** IRD wage and salary bands, MBIE bond rents by TA, Stats
   NZ household financial net worth, TA population, and the same MSD recipient
   facts as the calibration set (the take-up draws calibrate to them).

packages/microcosm-build/src/microcosm/build/calibration_columns.py

@@ -1,15 +1,15 @@
 """Prepare simple calibration columns from an explicit, engine-free contract.
 
 Geography and canonical concepts must already be present. This helper creates
-only constants and conjunctions of declared predicates; receipt assignment and
-statistical-unit construction belong to their own producers.
+constants, validated roster counts and conjunctions of declared predicates;
+receipt assignment and statistical-unit construction belong to their own producers.
 """
 
 from __future__ import annotations
 
 import math
 from collections.abc import Mapping
-from numbers import Real
+from numbers import Integral, Real
 from typing import Any
 
 import numpy as np
@@ -49,6 +49,8 @@ def prepare_calibration_columns(
             _validate_input(tables[entity][name], domain, f"{entity}.{name}")
 
     out = {entity: table.copy(deep=True) for entity, table in tables.items()}
+    derived = _prepare_roster_counts(out, bindings.get("roster_counts", ()), required)
+    prepared: dict[str, dict[str, object]] = {}
     seen: set[tuple[str, str]] = set()
     for entry in columns:
         if not isinstance(entry, Mapping):
@@ -57,7 +59,12 @@ def prepare_calibration_columns(
         if entity not in required or not isinstance(name, str) or not name:
             raise ValueError("Calibration column needs a declared entity and name.")
         key = (entity, name)
-        if key in seen or name in required[entity] or name.endswith("_id"):
+        if (
+            key in seen
+            or name in required[entity]
+            or name in derived.get(entity, ())
+            or name.endswith("_id")
+        ):
             raise ValueError(
                 f"Calibration output {entity}.{name} is repeated or structural."
             )
@@ -66,7 +73,7 @@ def prepare_calibration_columns(
             value = entry["constant"]
             if "predicates" in entry or not _finite_number(value):
                 raise ValueError(f"Calibration constant {entity}.{name} is invalid.")
-            out[entity][name] = value
+            prepared.setdefault(entity, {})[name] = value
             continue
         predicates = entry.get("predicates")
         if not isinstance(predicates, (list, tuple)) or not predicates:
@@ -76,7 +83,10 @@ def prepare_calibration_columns(
             if not isinstance(predicate, Mapping):
                 raise ValueError("Calibration predicates must be objects.")
             column = predicate.get("column")
-            if column not in required[entity] or "value" not in predicate:
+            if (
+                column not in {*required[entity], *derived.get(entity, ())}
+                or "value" not in predicate
+            ):
                 raise ValueError(
                     f"Calibration predicate needs a validated {entity} input."
                 )
@@ -93,7 +103,15 @@ def prepare_calibration_columns(
             else:
                 raise ValueError(f"Unsupported calibration predicate {operator!r}.")
             mask &= part.to_numpy(dtype=bool)
-        out[entity][name] = mask
+        prepared.setdefault(entity, {})[name] = mask
+    for entity, values in prepared.items():
+        original_order = list(out[entity].columns)
+        computed = pd.DataFrame(values, index=out[entity].index)
+        out[entity] = pd.concat(
+            [out[entity].drop(columns=list(values), errors="ignore"), computed], axis=1
+        )
+        order = original_order + [name for name in values if name not in original_order]
+        out[entity] = out[entity].loc[:, order]
     return out
 
 
@@ -103,10 +121,82 @@ def _finite_number(value: object) -> bool:
     )
 
 
+def _identity(value: object) -> tuple[type, object]:
+    if (
+        isinstance(value, bool)
+        or not isinstance(value, (str, Integral))
+        or isinstance(value, str)
+        and not value
+    ):
+        raise ValueError(
+            "Calibration membership identities must be strings or integers."
+        )
+    return (str, value) if isinstance(value, str) else (int, int(value))
+
+
+def _prepare_roster_counts(
+    tables: dict[str, pd.DataFrame], entries: object, required: Mapping[str, Any]
+) -> dict[str, set[str]]:
+    if not isinstance(entries, (list, tuple)):
+        raise ValueError("Calibration roster_counts must be a list.")
+    derived: dict[str, set[str]] = {}
+    for entry in entries:
+        if not isinstance(entry, Mapping):
+            raise ValueError("Calibration roster count must be an object.")
+        entity, members, name = (
+            entry.get("entity"),
+            entry.get("member_entity"),
+            entry.get("name"),
+        )
+        group_id, member_id, membership = (
+            entry.get("entity_id"),
+            entry.get("member_id"),
+            entry.get("membership"),
+        )
+        if (
+            entity not in required
+            or members not in required
+            or not isinstance(name, str)
+            or not name
+            or name.endswith("_id")
+            or name in required[entity]
+            or name in derived.get(entity, ())
+            or group_id not in required[entity]
+            or member_id not in required[members]
+            or membership not in required[members]
+            or not isinstance(entry.get("representation"), str)
+            or not entry["representation"]
+        ):
+            raise ValueError(
+                "Calibration roster count needs declared identities and representation."
+            )
+        groups = [_identity(value) for value in tables[entity][group_id].tolist()]
+        identities = [_identity(value) for value in tables[members][member_id].tolist()]
+        if len(set(groups)) != len(groups) or len(set(identities)) != len(identities):
+            raise ValueError(
+                "Calibration roster contains duplicate membership identities."
+            )
+        counts = dict.fromkeys(groups, 0)
+        for value in tables[members][membership].tolist():
+            key = _identity(value)
+            if key not in counts:
+                raise ValueError("Calibration roster contains orphan membership.")
+            counts[key] += 1
+        if any(count == 0 for count in counts.values()):
+            raise ValueError("Calibration roster contains empty households or groups.")
+        tables[entity][name] = [counts[key] for key in groups]
+        derived.setdefault(entity, set()).add(name)
+    return derived
+
+
 def _validate_input(values: pd.Series, domain: Mapping[str, Any], name: str) -> None:
     if values.isna().any():
         raise ValueError(f"Calibration input {name} contains missing values.")
-    if domain.get("kind") == "nonnegative_integer":
+    if domain.get("kind") == "identity":
+        for value in values.tolist():
+            _identity(value)
+        valid = True
+    elif domain.get("kind") == "nonnegative_integer":
         valid = all(
             _finite_number(value) and value >= 0 and value == int(value)
             for value in values.tolist()