UK #1334: compute the local metrics from their declarations, and measure the local HMRC rows over each person's year

changelog.d/uk-annual-income-local-rows.changed.md

@@ -0,0 +1 @@
+The five local HMRC income rows now measure each person's income over a year, as the national rows do: the 12 months before each FRS interview, a proxy for the tax year that HMRC's statistics count. The constituency and local-authority declarations of the self-employment and employment amounts and counts and of the higher rate taxpayers read `self_employment_income_after_loss_relief_annual`, `employment_income_annual`, `income_tax_annual` and `tax_band_annual` for their values, their taxpayer test and their rate band. Target ids, facts, values and metric names do not change. The incumbent extractor reads an incumbent without annual income descriptors on its current values, as rule 1 scores it nationally.

changelog.d/uk-annual-income-national-rows.changed.md

@@ -1 +1 @@
-The national HMRC income rows now measure each person's income over a year, as HMRC's statistics count income over the tax year. The year is a proxy for the tax year: the 12 months before each FRS interview (April 2024 to March 2025), applied to the engine's 2025 amounts and fitted to HMRC targets for calendar 2025 (the SPI 2023-24 rows uprated to it, and the Income Tax liabilities rows' calendar-2025 window). The 49 declarations of `hmrc_spi`, `hmrc_spi_region`, `hmrc_itl` and `hmrc_itl_marginal_rate` read the annual measures for their values, their total-income bands and filters, their taxpayer test and their rate band (`employment_income_annual`, `self_employment_income_after_loss_relief_annual`, `private_pension_income_annual`, `state_pension_annual`, `total_income_after_loss_relief_annual`, `income_tax_annual`, `tax_band_annual`), and `obr.income_tax` reads `income_tax_annual`. Dividends, property and savings income have no spells and stay as they are. Target ids, facts, values and metric names do not change: only the person-side measure each row adds up moves, so a recent leaver whose year of income makes them liable to income tax counts as a taxpayer, and a part-year earner sits in the band of what they received. Poverty, benefits and the engine's published tax keep reading current income. Reform costings on files built from this code are lower: in 2025 the engine's income tax on current income falls 1.0% (£324.7bn to £321.5bn against the build before this change, on policyengine-uk 2.147.2; now below OBR's receipts by construction, since the calibration fits OBR's total on the year's income), and static costings of threshold changes fall 1.7% to 1.9% (a £1,000 rise in the personal allowance from £8.98bn to £8.81bn, and 1p on the basic rate from £6.62bn to £6.51bn), closer to HMRC's 2025-26 ready reckoner (January 2025, which includes behaviour: £6.55bn for 1p, and £0.78bn for a £100 rise and £9.65bn for a 10% rise, which put £1,000 near £7.7bn to £7.8bn). The five local `hmrc` rows, the CGT taxable-income bands and the salary-sacrifice reliefs stay on current income for now; the local rows follow once the local line computes its metrics from their declarations.
+The national HMRC income rows now measure each person's income over a year, as HMRC's statistics count income over the tax year. The year is a proxy for the tax year: the 12 months before each FRS interview (April 2024 to March 2025), applied to the engine's 2025 amounts and fitted to HMRC targets for calendar 2025 (the SPI 2023-24 rows uprated to it, and the Income Tax liabilities rows' calendar-2025 window). The 49 declarations of `hmrc_spi`, `hmrc_spi_region`, `hmrc_itl` and `hmrc_itl_marginal_rate` read the annual measures for their values, their total-income bands and filters, their taxpayer test and their rate band (`employment_income_annual`, `self_employment_income_after_loss_relief_annual`, `private_pension_income_annual`, `state_pension_annual`, `total_income_after_loss_relief_annual`, `income_tax_annual`, `tax_band_annual`), and `obr.income_tax` reads `income_tax_annual`. Dividends, property and savings income have no spells and stay as they are. Target ids, facts, values and metric names do not change: only the person-side measure each row adds up moves, so a recent leaver whose year of income makes them liable to income tax counts as a taxpayer, and a part-year earner sits in the band of what they received. Poverty, benefits and the engine's published tax keep reading current income. Reform costings on files built from this code are lower. These figures come from a pre-Budget arm at `e399d21ef`, signed with a throwaway key and not attested, against #1338's certified build at `829c1bc26`, both on policyengine-uk 2.147.2; the first certified build from main confirms them, and this note changes with it if they move materially. In 2025 the engine's income tax on current income falls 1.0% (£324.7bn to £321.5bn; now below OBR's receipts on this build, since the calibration fits OBR's total on the year's income, and part-year entrants pull annual income below current income while leavers push it above), and static costings of threshold changes fall 1.7% to 1.9% (a £1,000 rise in the personal allowance from £8.98bn to £8.81bn, and 1p on the rUK basic rate from £6.62bn to £6.51bn). These are full-year liabilities of every UK taxpayer, which HMRC's 2025-26 ready reckoner (January 2025, with behaviour) is not, for two reasons. Its first-year column counts Self Assessment tax when it is paid, so it holds little of a change's Self Assessment effect: fitted to HMRC's own later columns, a full year is 9% to 11% higher, about £8.45bn for £1,000 on the allowance (from £0.98bn for 1% and £9.65bn for 10%) and £7.28bn for 1p (from £6.55bn). And it is the UK government's exchequer impact, which leaves out the devolved Scottish and Welsh elements and adds the change in their block grants. On that scope, in calendar 2025, the allowance rise costs £8.54bn on this arm against £8.71bn on #1338's build, moving from 3.1% to 1.1% above HMRC's full year, and 1p, with the rUK property income that the engine's main-rate parameter does not reach, raises £7.07bn against £7.18bn, moving from 1.3% to 2.9% below. Moved to 2025-26, this arm costs about 1% more on both, so like for like the allowance sits about 1.9% above HMRC and 1p about 1.9% below (microcosm#1370). The CGT taxable-income bands and the salary-sacrifice reliefs stay on current income.

changelog.d/uk-local-metrics-from-declarations.changed.md

@@ -0,0 +1 @@
+The local line computes each metric from its declaration. `compute_household_metrics` reads every constituency, local-authority and BRMA metric's `policyengine` binding in `uk_population_targets.json` through the national rows' binding interpreter (`prepared_binding_values`), on columns `UKMeasureResolver` computes, and sums person and benefit-unit bindings onto their household with the engine's `map_result`. A rebind therefore moves the national and local lines together, and a local declaration that carries a key the local line does not honour (a measurement period, a band, a mapping other than to the household, or a filter that names an entity, a concept or a reduction) refuses. Ten local declarations change so they say what the code computed: the four UC child bands count UC benefit units by `uc_calibration_child_count`, the FRS-role composition the national UC-by-children rows read; the private rents read `benunit_rent` summed to the household; the high-value homes and the gas-connected households read resolver measures that apply the stored period and price basis (`owner_occupied_home_over_gbp_*`, and the new `domestic_gas_connected` beside `domestic_electricity_kwh` and `domestic_gas_kwh`). The interpreter gains `in` lists, lower-inclusive ranges and subtraction in value expressions, and refuses a predicate that declares `input_period` or `price_basis`, which it cannot honour. On a national frame, the 59 local declarations' 43 metrics (61 columns over the three grains) are identical to the old code's, bit for bit, except `ons/equiv_housing_costs`, which no active target reads: it differs by float32 rounding because the difference is now taken in float64. The local references change only the entity they record: `benunit` for the UC child bands, `household` for the private rents. The incumbent extractor reads the incumbent file through one resolver, so its three `high_value_homes/*` columns move on a Microcosm-release incumbent: the old extractor read the stored main residence value and tenure at 2025, where the engine uprates the value, against thresholds deflated to the dataset period, while the resolver reads them at the frame's period, as the build's local line already did. The local rule-1 comparison therefore moves on those three metrics; the build's local line does not. An incumbent without FRS benefit-unit roles, such as the enhanced FRS, cannot measure the UC child bands: the local extractor does not prune listed measures, so it refuses and names `benunit.uc_calibration_child_count`, which the national scorer's signed incumbent-unresolvable register already records.

packages/microcosm-build/src/microcosm/build/target_materialization.py

@@ -959,6 +959,25 @@ def _prepared_column_values(
     )
 
 
+def prepared_binding_values(
+    adapter: Any,
+    entity: str,
+    binding: Mapping[str, Any],
+    *,
+    period: int | str | None = None,
+) -> np.ndarray:
+    """One binding's masked values at its own grain, as the national rows read it.
+
+    Its value variable or expression, kept where its filters and conditions
+    hold and zero elsewhere, through the same interpreter that materializes
+    every national target. A caller that needs one binding's column (the local
+    line's household metrics) reads it here, so a rebind moves every line
+    that reads the declaration.
+    """
+
+    return _prepared_column_values(adapter, entity, binding, period=period)
+
+
 class MeasurementPeriodError(RuntimeError):
     """A binding declares a measurement period its read route cannot honour.
 
@@ -1016,8 +1035,9 @@ def _value_variables(binding: Mapping[str, Any]) -> tuple[str, ...]:
     """The value variables a plain-column binding reads."""
 
     if "value_expression" in binding:
-        parts = str(binding["value_expression"]).split("+")
-        return tuple(part.strip() for part in parts if part.strip())
+        return tuple(
+            name for _, name in _expression_terms(str(binding["value_expression"]))
+        )
     return (str(binding["value_variable"]),)
 
 
@@ -1088,18 +1108,42 @@ def _entity_reduction(adapter: Any, reduction: Mapping[str, Any]) -> np.ndarray:
     return np.asarray(adapter.entity_reduction(reduction), dtype=float)
 
 
+#: Predicate keys a declaration may carry that this interpreter does not
+#: honour: a predicate naming one would be read at the wrong period or price
+#: basis, so it refuses; such a row binds a measure that applies them instead.
+_UNHONOURED_PREDICATE_KEYS = ("input_period", "price_basis")
+
+
 def _predicate_mask(
     adapter: Any,
     default_entity: str,
     predicate: Mapping[str, Any],
 ) -> np.ndarray:
     if "reduce" in predicate and hasattr(adapter, "household_condition"):
         return np.asarray(adapter.household_condition(predicate), dtype=bool)
+    unhonoured = [key for key in _UNHONOURED_PREDICATE_KEYS if key in predicate]
+    if unhonoured:
+        raise ValueError(
+            f"predicate {dict(predicate)!r} declares {unhonoured}, which the "
+            "binding interpreter does not honour; bind a measure that applies "
+            "them instead."
+        )
     entity = str(predicate.get("entity") or default_entity)
     variable = str(predicate.get("variable") or predicate.get("concept"))
     values = _column(adapter, entity, variable)
     if "operator" in predicate:
         operator, expected = str(predicate["operator"]), predicate["value"]
+    elif "in" in predicate:
+        operator, expected = "in", predicate["in"]
+    elif "lower" in predicate or "upper" in predicate:
+        # A range: the lower bound inclusive, the upper bound exclusive.
+        numeric = values.astype(float)
+        mask = np.ones(numeric.shape, dtype=bool)
+        if predicate.get("lower") is not None:
+            mask &= numeric >= float(predicate["lower"])
+        if predicate.get("upper") is not None:
+            mask &= numeric < float(predicate["upper"])
+        return mask
     else:
         operator, expected = "==", predicate["equals"]
     return _compare(values, operator, expected)
@@ -1125,6 +1169,20 @@ def _compare(values: np.ndarray, operator: str, expected: object) -> np.ndarray:
     raise ValueError(f"unsupported predicate operator {operator!r}")
 
 
+_EXPRESSION_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+
+
+def _expression_terms(expression: str) -> tuple[tuple[float, str], ...]:
+    """A value expression's signed terms: names joined by ``+`` or ``-``."""
+
+    tokens = re.split(r"\s*([+-])\s*", expression.strip())
+    names = tokens[0::2]
+    if not names or any(not _EXPRESSION_NAME.fullmatch(name) for name in names):
+        raise ValueError(f"unsupported value_expression {expression!r}")
+    signs = (1.0, *(1.0 if sign == "+" else -1.0 for sign in tokens[1::2]))
+    return tuple(zip(signs, names, strict=True))
+
+
 def _expression(
     adapter: Any,
     entity: str,
@@ -1133,15 +1191,13 @@ def _expression(
     period: int | str | None = None,
     route_stored: bool = False,
 ) -> np.ndarray:
-    parts = [part.strip() for part in expression.split("+")]
-    if not parts or any(not part for part in parts):
-        raise ValueError(f"unsupported value_expression {expression!r}")
     total: np.ndarray | None = None
-    for part in parts:
+    for sign, name in _expression_terms(expression):
         values = _value_column(
-            adapter, entity, part, period=period, route_stored=route_stored
+            adapter, entity, name, period=period, route_stored=route_stored
         )
-        total = values.copy() if total is None else total + values
+        term = values if sign > 0 else -values
+        total = term.copy() if total is None else total + term
     if total is None:
         raise ValueError(f"unsupported value_expression {expression!r}")
     return total

packages/microcosm-build/src/microcosm/build/uk/local_binding_adjudications.json

@@ -10,9 +10,9 @@
       "expires_on": "2027-01-14"
     },
     "hmrc_spi_frame_model_proxy": {
-      "reason": "Accept the stated basis: the constituency and local-authority HMRC families bind on the modeled income_tax > 0 frame, the same SPI-frame proxy the national hmrc.* surface binds through (uk_population_targets.json bindings), so both grains share one frame and the standing cross-grain rule can reconcile them. The proxy's known biases (zero-liability taxpayers excluded; modeled liabilities the SPI would not observe included) are accepted for the first candidate and re-measured per family in the microcosm#762 receipts. Fence evidence: uk_local_target_census.json#/binding_fences/hmrc_spi_frame_model_proxy.",
+      "reason": "Accept the stated basis: the constituency and local-authority HMRC families bind on the modeled income_tax_annual > 0 frame (each person's income tax over the 12 months before their FRS interview, a proxy for the tax year), the same SPI-frame proxy the national hmrc.* surface binds through (uk_population_targets.json bindings), so both grains share one frame and the standing cross-grain rule can reconcile them. The proxy's known biases (zero-liability taxpayers excluded; modeled liabilities the SPI would not observe included) are accepted for the first candidate and re-measured per family in the microcosm#762 receipts. Fence evidence: uk_local_target_census.json#/binding_fences/hmrc_spi_frame_model_proxy.",
       "approved_by": "juaristi22",
-      "adjudication": "microcosm#762 (A1, rulings 2026-09-02); renewed three months on 2026-10-14 by juaristi22 (microcosm#1278): both grains still bind the HMRC area families on the modeled taxpayer frame.",
+      "adjudication": "microcosm#762 (A1, rulings 2026-09-02); renewed three months on 2026-10-14 by juaristi22 (microcosm#1278): both grains still bind the HMRC area families on the modeled taxpayer frame; re-worded 2026-10-23 by juaristi22 (microcosm#1347 review): the frame is each person's year of income (income_tax_annual > 0) at both grains, since microcosm#1344 moved the national rows and microcosm#1347 the area rows.",
       "approved_on": "2026-10-14",
       "expires_on": "2027-01-14"
     },