Publish calibration diagnostics as schema 9, each hierarchy label declared once

README.md

@@ -359,8 +359,9 @@ the historical `record` and `projection`) penalize distance from the design
 weights; see [penalized calibration toward the design weights](docs/calibration-l2-basis.md)
 for the algebra, the evidence and the measured frontier.
 
-National and ACS local-area builds now use the same typed schema-8 calibration
-diagnostics writer. The local builder adds its Census population marginals to a
+National and ACS local-area builds now use the same typed calibration
+diagnostics writer, which publishes schema 9: the schema-8 document builders
+assemble, with each hierarchy label declared once in `hierarchy_catalogs`. The local builder adds its Census population marginals to a
 versioned `TargetRegistry`, including provider, category, geography, and target
 hierarchy, before calibration. Both builders always attempt diagnostics after
 the calibrated dataset exists. If construction, validation, serialization, or
@@ -375,11 +376,11 @@ UK release workflows stop when diagnostics are unavailable because their later
 release checks require that evidence; historical schema-6 and schema-7 UK
 artifacts remain readable through isolated compatibility validation.
 
-A Draft 2020-12 JSON Schema for schema 8, generated from those models, is
-committed at
-`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v8.schema.json`
-for consumers that check the file without importing them, such as the
-collector. Its description lists the model rules a JSON Schema cannot express;
+Draft 2020-12 JSON Schemas for schema 9 and schema 8, generated from those
+models, are committed at
+`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json`
+and `calibration_diagnostics.v8.schema.json` beside it, for consumers that check
+the file without importing them, such as the collector. Its description lists the model rules a JSON Schema cannot express;
 see the [package README](packages/microcosm-diagnostics/README.md).
 
 Standard publication uploads the locally built `releases/<id>/` artifacts to

changelog.d/1341-diagnostics-schema-9-catalogs.changed.md

@@ -0,0 +1 @@
+Calibration diagnostics are published as schema 9: each provider, category, geography and dimension label is declared once in `hierarchy_catalogs`, and target rows refer to it by ID, so one geography or dimension value can no longer carry two labels in a published file (microcosm#1341). Producers still assemble schema 8; the writer (`catalog_calibration_diagnostics`) builds the catalogs and refuses rows that label one identifier two ways, with the failure code `label_conflict` and no file, unless a reviewed label choice in `microcosm.calibrate.hierarchy_labels` names the label. The UK choices give a `geography` dimension value its area's label (the payload's geography label, `UK_GEOGRAPHY_ID_TO_LABEL`, or the ONS local-authority names), which resolves the 88 areas the October dense release names two ways; the US choices name the label chosen in review for each of the three values two US tables word differently ("No adjusted gross income", "All returns, total", "Age 85 plus"), covering only the two spellings Chronicle registers as reviewed (chronicle#377), so a third spelling is still refused; the Belgian choice writes `measure` = `country_total` as "Country total", covering only the four spellings Belgium's Chronicle packages give it, without which a Belgian build would stop at its diagnostics step. Every disagreement a choice settles is logged as a warning. The release contract, the UK local scorer and the dense assembler accept schema 8 and 9, and a generated `calibration_diagnostics.v9.schema.json` ships beside the schema 8 file.

docs/agent-guide.md

@@ -415,14 +415,15 @@ module instead of copying it or reconstructing alternate views in consumers.
 
 ## Calibration diagnostics JSON Schema
 
-`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v8.schema.json`
-is generated from the schema-8 models in `microcosm.diagnostics.schema`. After
-changing one, regenerate it with
+`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json`
+(what builds publish) and `calibration_diagnostics.v8.schema.json` beside it
+(documents published before schema 9) are generated from the models in
+`microcosm.diagnostics.schema`. After changing one, regenerate both with
 `uv run --no-sync python tools/generate_calibration_diagnostics_json_schema.py`.
 A new model validator also needs an entry in
-`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS` and a document in
-`test_json_schema.py` that the schema accepts and the models refuse; the
-diagnostics tests fail until both exist.
+`CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS` (or the schema-8 list) and a
+document in `test_json_schema_v9.py` (or `test_json_schema.py`) that the schema
+accepts and the models refuse; the diagnostics tests fail until both exist.
 
 ## Review this guide
 

docs/calibration-target-hierarchy.md

@@ -1,7 +1,8 @@
 # Calibration target hierarchy
 
-Microcosm diagnostics schema 8 publishes a fixed sequence of hierarchy tier
-types for every registry-backed calibration target:
+Microcosm calibration diagnostics (schema 8, and schema 9 which current
+builds publish) carry a fixed sequence of hierarchy tier types for every
+registry-backed calibration target:
 
 1. provider
 2. category
@@ -127,6 +128,100 @@ that labels the IRS SOI district rows translated to the 119th Congress plan.
 That release's materialize stage assembles its admin and population targets
 into one registry before the engine pass, so a conflict stops it at the start.
 
+## Diagnostics schema 9: labels declared once
+
+Schema 8 copies every provider, category, geography and dimension label onto
+every target row. The registry check above covers geography ids only, and
+Chronicle reports a value two packages word differently only as a warning, so
+a schema 8 file could still label one identifier two ways. Published examples:
+the US national release `populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z`
+labels the AGI band `under_1` "No adjusted gross income" (106 targets) and "No
+adjusted gross income and deficit" (2), and the October 2026 UK dense release
+gives 88 areas two names, because Chronicle's local facts carry the area as a
+dimension named `geography` whose value is the lower-case GSS code and each
+publisher names it its own way (HMRC "Hartlepool UA", ONS "Hartlepool").
+
+Schema 9 (`CalibrationDiagnosticsV9`) declares each of those labels once, in a
+top-level `hierarchy_catalogs` object, and each row's `hierarchy` refers to
+them by id:
+
+```json
+{
+  "schema_version": 9,
+  "hierarchy_catalogs": {
+    "providers": [{"id": "obr", "label": "Office for Budget Responsibility"}],
+    "categories": [
+      {"id": "obr.efo_receipts", "label": "Economic and fiscal outlook receipts",
+       "provider_id": "obr"}
+    ],
+    "geographies": [
+      {"id": "K02000001", "label": "United Kingdom", "level": "country"}
+    ],
+    "dimensions": [
+      {"id": "obr.efo_line", "label": "Economic and fiscal outlook line",
+       "values": [{"id": "income_tax", "label": "Income tax"}]}
+    ]
+  },
+  "targets": [
+    {
+      "name": "obr.income_tax@2025",
+      "target_name": "obr.income_tax",
+      "hierarchy": {
+        "category_id": "obr.efo_receipts",
+        "geography_id": "K02000001",
+        "dimensions": [{"id": "obr.efo_line", "value_id": "income_tax"}],
+        "target": {"id": "obr.income_tax", "label": "Income tax receipts"}
+      }
+    }
+  ]
+}
+```
+
+The provider is the category's; providers and categories move into catalogs
+with geographies and dimensions, mirroring the authoring contract's catalogs,
+so a category cannot appear under two providers either. Each value is listed
+under its own dimension. A geography id has one label and one level. The
+target's own label stays on the row, and every other field is schema 8's.
+
+Producers still assemble schema 8 (`CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION`);
+the writer publishes schema 9 (`catalog_calibration_diagnostics`). It refuses,
+with the failure code `label_conflict` and no file, rows that give one id more
+than one label (or one geography more than one level) unless a reviewed label
+choice resolves it (`microcosm.calibrate.hierarchy_labels`):
+
+- UK: a `geography` dimension value takes its area's label: the payload's own
+  geography label for that code, else `UK_GEOGRAPHY_ID_TO_LABEL`, else the ONS
+  April 2023 local-authority names in `local_authority_names.json` (the UK
+  build's writer adds those). This resolves all 88 dense-release areas, two of
+  them constituencies that only the payload's geography names, and the
+  national releases' "Yorkshire And The Humber".
+- US: `under_1` is "No adjusted gross income", `all` is "All returns, total",
+  and `age` `85_plus` is "Age 85 plus", the labels chosen in review on
+  2026-10-21. A US feed re-pin brings the same labels from Chronicle
+  (chronicle#377 for `under_1`), so these choices matter only for feeds and
+  releases built before it. Each choice covers only the two spellings
+  Chronicle registers as reviewed variants; a third spelling is refused.
+- Belgium: `measure` `country_total` is "Country total", the label chosen in
+  review on 2026-10-22 and the one Chronicle's canonical register gives it
+  (chronicle#377). Each Belgian package (SPF Finances, ONSS, ONEM, NBB)
+  labels its national figure with its own measure name, so without this choice
+  the transport graph's diagnostics step raises and a Belgian build fails. The
+  choice covers only those four spellings; another is refused.
+
+Every disagreement a choice settles is logged as a warning naming the id, each
+spelling with its first target, and the label written, so a choice never hides
+a variant.
+
+A choice applies only where the rows disagree, so an id every row labels one
+way keeps its label. Any other disagreement stops the writer until the source
+is corrected or a reviewed choice is added. The same catalogs are written by
+the transport diagnostics kernel and the UK graph's final validation kernel,
+which raise instead of writing.
+
+Releases published before schema 9 keep their schema-8 files;
+`parse_calibration_diagnostics` reads both, and
+`expand_calibration_diagnostics` rebuilds schema 8 from schema 9.
+
 ## Propagation stages
 
 The expected data flow is:
@@ -143,9 +238,11 @@ The expected data flow is:
    readable for historical artifacts.
 7. Solver compilation carries the hierarchy unchanged from `TargetSpec` to
    `Target`.
-8. Diagnostics schema 8 serializes `targets[].hierarchy` without constructing
-   provider, category, geography, dimension, or target labels.
-9. Release validation requires every schema-8 tier and label.
+8. Diagnostics assembly (schema 8) serializes `targets[].hierarchy` without
+   constructing provider, category, geography, dimension, or target labels;
+   the writer publishes schema 9, declaring each label once in
+   `hierarchy_catalogs`.
+9. Release validation requires every tier and label, in schema 8 or 9.
 10. The calibration dashboard selects its parser from top-level
     `schema_version` and builds navigation from the ordered hierarchy.
 

docs/transport-graph-kernels.md

@@ -106,7 +106,9 @@ that rejects; if it lowers them, the solver's clamp is the tighter bound.
 
 `diagnostics.calibration@1` rebuilds the completed calibration from the
 `problem`, `solution` and `result` artifacts without solving, and emits the
-schema-8 model. Each row describes its target as compiled (its own entity,
+schema-9 document (the schema-8 model with its hierarchy labels in
+`hierarchy_catalogs`; a second label for one identifier without a chosen label
+fails the node). Each row describes its target as compiled (its own entity,
 measure and filter), with its hierarchy from the surface's registry. The
 kernel refuses a population whose installed weights are not the solution's,
 a result from another solve, and a surface other than the one the problem

packages/microcosm-build/src/microcosm/build/transport/terminal_kernels.py

@@ -65,7 +65,11 @@
     decode_problem,
     decode_solution,
 )
-from microcosm.diagnostics import CalibrationDiagnosticsV8
+from microcosm.calibrate.hierarchy_labels import hierarchy_label_choices
+from microcosm.diagnostics import (
+    CalibrationDiagnosticsV8,
+    catalog_calibration_diagnostics,
+)
 from microcosm.frame import EntitySchema, Frame, engine_tables
 from microcosm.frame.adapters.axiom import AxiomEntityTableDataset
 from microcosm.graph import (
@@ -278,7 +282,13 @@ def run(self, context: KernelContext) -> KernelResult:
                 "surface_sha256": surface.sha256,
             },
         )
-        model = CalibrationDiagnosticsV8.model_validate(payload)
+        # Published as schema 9: the hierarchy labels go into catalogs, and a
+        # row set that labels one identifier two ways without a chosen label
+        # is refused (HierarchyLabelConflictError).
+        model = catalog_calibration_diagnostics(
+            CalibrationDiagnosticsV8.model_validate(payload),
+            label_choices=hierarchy_label_choices(surface.registry.country),
+        )
         diagnostics = canonical_json(
             model.model_dump(mode="python", exclude_defaults=True)
         )

packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py

@@ -33,6 +33,7 @@
     effective_sample_size,
 )
 from microcosm.calibrate.diagnostics import past_cap_census
+from microcosm.calibrate.hierarchy_labels import UK_HIERARCHY_LABEL_CHOICES
 from microcosm.calibrate.solve import CalibrationResult
 from microcosm.diagnostics import (
     UK_DIAGNOSTICS_SCHEMA_VERSION,
@@ -42,17 +43,21 @@
     UK_WITHHELD_PUBLISHED_OPTIONS,
     CalibrationDiagnosticsV8,
     DiagnosticsWriteOutcome,
+    HierarchyLabelChoices,
     failed_diagnostics_outcome,
 )
 from microcosm.diagnostics import (
     write_calibration_diagnostics as write_typed_calibration_diagnostics,
 )
 from microcosm.frame import Frame
 
+from .local_authority_input import load_uk_local_authority_names_resource
+
 __all__ = [
     "UK_DIAGNOSTICS_SCHEMA_VERSION",
     "UK_TARGET_GEOGRAPHY_LEVELS",
     "uk_calibration_diagnostics_payload",
+    "uk_hierarchy_label_choices",
     "uk_consumption_drift",
     "uk_fit_by_family",
     "uk_support_limited_misses",
@@ -1403,6 +1408,25 @@ def uk_calibration_diagnostics_payload(
     )
 
 
+def uk_hierarchy_label_choices() -> HierarchyLabelChoices:
+    """The UK's label choices, with the ONS April 2023 local-authority names.
+
+    UK publishers name one area several ways, and Chronicle's local facts
+    carry the area as a ``geography`` dimension value (HMRC "Hartlepool UA",
+    ONS "Hartlepool"). Diagnostics schema 9 holds one label per area: the
+    payload's own geography label for that code, else Microcosm's catalogs,
+    ``UK_GEOGRAPHY_ID_TO_LABEL`` and ``local_authority_names.json``
+    (:mod:`microcosm.calibrate.hierarchy_labels`).
+    """
+
+    areas = load_uk_local_authority_names_resource()["areas"]
+    return UK_HIERARCHY_LABEL_CHOICES.merged(
+        HierarchyLabelChoices(
+            geographies={code: str(entry["name"]) for code, entry in areas.items()}
+        )
+    )
+
+
 def write_uk_calibration_diagnostics(
     result: CalibrationResult,
     path: Path | str,
@@ -1438,4 +1462,6 @@ def write_uk_calibration_diagnostics(
         )
     except Exception as error:  # diagnostics must not escape without validation
         return failed_diagnostics_outcome(error, path=output)
-    return write_typed_calibration_diagnostics(diagnostics, output)
+    return write_typed_calibration_diagnostics(
+        diagnostics, output, label_choices=uk_hierarchy_label_choices()
+    )

packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py

@@ -946,8 +946,14 @@ def run(self, context: KernelContext) -> KernelResult:
             )
             diagnostics = evidence.artifacts["target_diagnostics"]
             support = support_frame.to_dict(orient="records")
+            from microcosm.diagnostics import (
+                CalibrationDiagnosticsV8,
+                catalog_calibration_diagnostics,
+            )
+
             from .diagnostics import (
                 uk_calibration_diagnostics_payload,
+                uk_hierarchy_label_choices,
                 uk_withhold_brma_figures,
             )
 
@@ -995,10 +1001,18 @@ def run(self, context: KernelContext) -> KernelResult:
                 },
                 withheld=withheld,
             )
+            # Published as schema 9, as the canonical writer publishes it:
+            # one label per area, whichever publisher named it
+            # (uk_hierarchy_label_choices); an unresolved second label is
+            # refused (HierarchyLabelConflictError).
+            published_diagnostics = catalog_calibration_diagnostics(
+                CalibrationDiagnosticsV8.model_validate(complete_diagnostics),
+                label_choices=uk_hierarchy_label_choices(),
+            ).model_dump(mode="python", exclude_defaults=True)
             output_artifacts.update(
                 {
                     "calibration_diagnostics": encode_stage_evidence(
-                        _json_safe(complete_diagnostics)
+                        _json_safe(published_diagnostics)
                     ),
                     "target_diagnostics_csv": pd.DataFrame(
                         uk_withhold_brma_figures(diagnostics, withheld)

packages/microcosm-build/tests/engine_free/shared/test_transport_terminal_kernels.py

@@ -58,7 +58,8 @@
 )
 from microcosm.calibrate.artifacts import PROBLEM_TYPE, RESULT_TYPE, SOLUTION_TYPE
 from microcosm.diagnostics import (
-    CalibrationDiagnosticsV8,
+    CalibrationDiagnosticsV9,
+    expand_calibration_diagnostics,
     parse_calibration_diagnostics,
     write_calibration_diagnostics,
 )
@@ -144,12 +145,15 @@ def test_full_build_passes_its_gates_readback_and_package(toy_run) -> None:
 # ---------------------------------------------------------------------------
 
 
-def test_diagnostics_validate_against_schema_8(toy_run, tmp_path) -> None:
+def test_diagnostics_validate_against_schema_9(toy_run, tmp_path) -> None:
     payload = toy_run.artifact(DIAGNOSTICS, "diagnostics")
-    model = parse_calibration_diagnostics(json.loads(payload))
+    published = parse_calibration_diagnostics(json.loads(payload))
 
-    assert isinstance(model, CalibrationDiagnosticsV8)
-    assert model.schema_version == DIAGNOSTICS_TYPE.schema_version == 8
+    assert isinstance(published, CalibrationDiagnosticsV9)
+    assert published.schema_version == DIAGNOSTICS_TYPE.schema_version == 9
+    # The rows' labels sit in the published catalogs; expanded, each row
+    # carries its schema-8 hierarchy again.
+    model = expand_calibration_diagnostics(published)
     assert model.target_registry.country == COUNTRY
     assert model.n_records == 6
     assert [row.target_name for row in model.targets] == [
@@ -178,7 +182,7 @@ def test_diagnostics_validate_against_schema_8(toy_run, tmp_path) -> None:
     outcome = write_calibration_diagnostics(model, tmp_path / "diagnostics.json")
     assert outcome.status == "available"
     assert (
-        canonical_json(model.model_dump(mode="python", exclude_defaults=True))
+        canonical_json(published.model_dump(mode="python", exclude_defaults=True))
         == payload
     )
 

packages/microcosm-build/tests/engine_free/uk/test_uk_dense_release_assembler.py

@@ -493,16 +493,22 @@ def _candidate_dir(
     return candidate, spine, incumbent_manifest
 
 
+@pytest.mark.parametrize("schema_version", [8, 9])
 def test_current_diagnostics_are_validated_then_copied_without_rewriting(
     tmp_path: Path,
     monkeypatch,
+    schema_version: int,
 ) -> None:
+    # A candidate written before schema 9 carries schema 8; both carry the
+    # calibration hierarchy and are copied byte for byte.
     assembler = _load("assemble_uk_dense_release_dir")
     source = tmp_path / "source.json"
     destination = tmp_path / "release.json"
-    source_bytes = b'{"schema_version":8,"source":"exact bytes"}\n'
+    source_bytes = (
+        f'{{"schema_version":{schema_version},"source":"exact bytes"}}\n'.encode()
+    )
     source.write_bytes(source_bytes)
-    diagnostics = {"schema_version": 8, "source": "exact bytes"}
+    diagnostics = {"schema_version": schema_version, "source": "exact bytes"}
     captured: dict[str, object] = {}
 
     def canonical_parser(payload: object) -> object: