diff --git a/README.md b/README.md index e38a0d0e3..3a128b988 100644 --- a/README.md +++ b/README.md @@ -359,8 +359,9 @@ the historical `record` and `projection`) penalize distance from the design weights; see [penalized calibration toward the design weights](docs/calibration-l2-basis.md) for the algebra, the evidence and the measured frontier. -National and ACS local-area builds now use the same typed schema-8 calibration -diagnostics writer. The local builder adds its Census population marginals to a +National and ACS local-area builds now use the same typed calibration +diagnostics writer, which publishes schema 9: the schema-8 document builders +assemble, with each hierarchy label declared once in `hierarchy_catalogs`. The local builder adds its Census population marginals to a versioned `TargetRegistry`, including provider, category, geography, and target hierarchy, before calibration. Both builders always attempt diagnostics after the calibrated dataset exists. If construction, validation, serialization, or @@ -375,11 +376,11 @@ UK release workflows stop when diagnostics are unavailable because their later release checks require that evidence; historical schema-6 and schema-7 UK artifacts remain readable through isolated compatibility validation. -A Draft 2020-12 JSON Schema for schema 8, generated from those models, is -committed at -`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v8.schema.json` -for consumers that check the file without importing them, such as the -collector. Its description lists the model rules a JSON Schema cannot express; +Draft 2020-12 JSON Schemas for schema 9 and schema 8, generated from those +models, are committed at +`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json` +and `calibration_diagnostics.v8.schema.json` beside it, for consumers that check +the file without importing them, such as the collector. Its description lists the model rules a JSON Schema cannot express; see the [package README](packages/microcosm-diagnostics/README.md). Standard publication uploads the locally built `releases//` artifacts to diff --git a/changelog.d/1341-diagnostics-schema-9-catalogs.changed.md b/changelog.d/1341-diagnostics-schema-9-catalogs.changed.md new file mode 100644 index 000000000..6011ff9c6 --- /dev/null +++ b/changelog.d/1341-diagnostics-schema-9-catalogs.changed.md @@ -0,0 +1 @@ +Calibration diagnostics are published as schema 9: each provider, category, geography and dimension label is declared once in `hierarchy_catalogs`, and target rows refer to it by ID, so one geography or dimension value can no longer carry two labels in a published file (microcosm#1341). Producers still assemble schema 8; the writer (`catalog_calibration_diagnostics`) builds the catalogs and refuses rows that label one identifier two ways, with the failure code `label_conflict` and no file, unless a reviewed label choice in `microcosm.calibrate.hierarchy_labels` names the label. The UK choices give a `geography` dimension value its area's label (the payload's geography label, `UK_GEOGRAPHY_ID_TO_LABEL`, or the ONS local-authority names), which resolves the 88 areas the October dense release names two ways; the US choices name the label chosen in review for each of the three values two US tables word differently ("No adjusted gross income", "All returns, total", "Age 85 plus"), covering only the two spellings Chronicle registers as reviewed (chronicle#377), so a third spelling is still refused; the Belgian choice writes `measure` = `country_total` as "Country total", covering only the four spellings Belgium's Chronicle packages give it, without which a Belgian build would stop at its diagnostics step. Every disagreement a choice settles is logged as a warning. The release contract, the UK local scorer and the dense assembler accept schema 8 and 9, and a generated `calibration_diagnostics.v9.schema.json` ships beside the schema 8 file. diff --git a/docs/agent-guide.md b/docs/agent-guide.md index 10bc99fdd..947e43d9b 100644 --- a/docs/agent-guide.md +++ b/docs/agent-guide.md @@ -415,14 +415,15 @@ module instead of copying it or reconstructing alternate views in consumers. ## Calibration diagnostics JSON Schema -`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v8.schema.json` -is generated from the schema-8 models in `microcosm.diagnostics.schema`. After -changing one, regenerate it with +`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json` +(what builds publish) and `calibration_diagnostics.v8.schema.json` beside it +(documents published before schema 9) are generated from the models in +`microcosm.diagnostics.schema`. After changing one, regenerate both with `uv run --no-sync python tools/generate_calibration_diagnostics_json_schema.py`. A new model validator also needs an entry in -`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS` and a document in -`test_json_schema.py` that the schema accepts and the models refuse; the -diagnostics tests fail until both exist. +`CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS` (or the schema-8 list) and a +document in `test_json_schema_v9.py` (or `test_json_schema.py`) that the schema +accepts and the models refuse; the diagnostics tests fail until both exist. ## Review this guide diff --git a/docs/calibration-target-hierarchy.md b/docs/calibration-target-hierarchy.md index e2f96929d..163c677e3 100644 --- a/docs/calibration-target-hierarchy.md +++ b/docs/calibration-target-hierarchy.md @@ -1,7 +1,8 @@ # Calibration target hierarchy -Microcosm diagnostics schema 8 publishes a fixed sequence of hierarchy tier -types for every registry-backed calibration target: +Microcosm calibration diagnostics (schema 8, and schema 9 which current +builds publish) carry a fixed sequence of hierarchy tier types for every +registry-backed calibration target: 1. provider 2. category @@ -127,6 +128,100 @@ that labels the IRS SOI district rows translated to the 119th Congress plan. That release's materialize stage assembles its admin and population targets into one registry before the engine pass, so a conflict stops it at the start. +## Diagnostics schema 9: labels declared once + +Schema 8 copies every provider, category, geography and dimension label onto +every target row. The registry check above covers geography ids only, and +Chronicle reports a value two packages word differently only as a warning, so +a schema 8 file could still label one identifier two ways. Published examples: +the US national release `populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z` +labels the AGI band `under_1` "No adjusted gross income" (106 targets) and "No +adjusted gross income and deficit" (2), and the October 2026 UK dense release +gives 88 areas two names, because Chronicle's local facts carry the area as a +dimension named `geography` whose value is the lower-case GSS code and each +publisher names it its own way (HMRC "Hartlepool UA", ONS "Hartlepool"). + +Schema 9 (`CalibrationDiagnosticsV9`) declares each of those labels once, in a +top-level `hierarchy_catalogs` object, and each row's `hierarchy` refers to +them by id: + +```json +{ + "schema_version": 9, + "hierarchy_catalogs": { + "providers": [{"id": "obr", "label": "Office for Budget Responsibility"}], + "categories": [ + {"id": "obr.efo_receipts", "label": "Economic and fiscal outlook receipts", + "provider_id": "obr"} + ], + "geographies": [ + {"id": "K02000001", "label": "United Kingdom", "level": "country"} + ], + "dimensions": [ + {"id": "obr.efo_line", "label": "Economic and fiscal outlook line", + "values": [{"id": "income_tax", "label": "Income tax"}]} + ] + }, + "targets": [ + { + "name": "obr.income_tax@2025", + "target_name": "obr.income_tax", + "hierarchy": { + "category_id": "obr.efo_receipts", + "geography_id": "K02000001", + "dimensions": [{"id": "obr.efo_line", "value_id": "income_tax"}], + "target": {"id": "obr.income_tax", "label": "Income tax receipts"} + } + } + ] +} +``` + +The provider is the category's; providers and categories move into catalogs +with geographies and dimensions, mirroring the authoring contract's catalogs, +so a category cannot appear under two providers either. Each value is listed +under its own dimension. A geography id has one label and one level. The +target's own label stays on the row, and every other field is schema 8's. + +Producers still assemble schema 8 (`CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION`); +the writer publishes schema 9 (`catalog_calibration_diagnostics`). It refuses, +with the failure code `label_conflict` and no file, rows that give one id more +than one label (or one geography more than one level) unless a reviewed label +choice resolves it (`microcosm.calibrate.hierarchy_labels`): + +- UK: a `geography` dimension value takes its area's label: the payload's own + geography label for that code, else `UK_GEOGRAPHY_ID_TO_LABEL`, else the ONS + April 2023 local-authority names in `local_authority_names.json` (the UK + build's writer adds those). This resolves all 88 dense-release areas, two of + them constituencies that only the payload's geography names, and the + national releases' "Yorkshire And The Humber". +- US: `under_1` is "No adjusted gross income", `all` is "All returns, total", + and `age` `85_plus` is "Age 85 plus", the labels chosen in review on + 2026-10-21. A US feed re-pin brings the same labels from Chronicle + (chronicle#377 for `under_1`), so these choices matter only for feeds and + releases built before it. Each choice covers only the two spellings + Chronicle registers as reviewed variants; a third spelling is refused. +- Belgium: `measure` `country_total` is "Country total", the label chosen in + review on 2026-10-22 and the one Chronicle's canonical register gives it + (chronicle#377). Each Belgian package (SPF Finances, ONSS, ONEM, NBB) + labels its national figure with its own measure name, so without this choice + the transport graph's diagnostics step raises and a Belgian build fails. The + choice covers only those four spellings; another is refused. + +Every disagreement a choice settles is logged as a warning naming the id, each +spelling with its first target, and the label written, so a choice never hides +a variant. + +A choice applies only where the rows disagree, so an id every row labels one +way keeps its label. Any other disagreement stops the writer until the source +is corrected or a reviewed choice is added. The same catalogs are written by +the transport diagnostics kernel and the UK graph's final validation kernel, +which raise instead of writing. + +Releases published before schema 9 keep their schema-8 files; +`parse_calibration_diagnostics` reads both, and +`expand_calibration_diagnostics` rebuilds schema 8 from schema 9. + ## Propagation stages The expected data flow is: @@ -143,9 +238,11 @@ The expected data flow is: readable for historical artifacts. 7. Solver compilation carries the hierarchy unchanged from `TargetSpec` to `Target`. -8. Diagnostics schema 8 serializes `targets[].hierarchy` without constructing - provider, category, geography, dimension, or target labels. -9. Release validation requires every schema-8 tier and label. +8. Diagnostics assembly (schema 8) serializes `targets[].hierarchy` without + constructing provider, category, geography, dimension, or target labels; + the writer publishes schema 9, declaring each label once in + `hierarchy_catalogs`. +9. Release validation requires every tier and label, in schema 8 or 9. 10. The calibration dashboard selects its parser from top-level `schema_version` and builds navigation from the ordered hierarchy. diff --git a/docs/transport-graph-kernels.md b/docs/transport-graph-kernels.md index 3111794ce..708829993 100644 --- a/docs/transport-graph-kernels.md +++ b/docs/transport-graph-kernels.md @@ -106,7 +106,9 @@ that rejects; if it lowers them, the solver's clamp is the tighter bound. `diagnostics.calibration@1` rebuilds the completed calibration from the `problem`, `solution` and `result` artifacts without solving, and emits the -schema-8 model. Each row describes its target as compiled (its own entity, +schema-9 document (the schema-8 model with its hierarchy labels in +`hierarchy_catalogs`; a second label for one identifier without a chosen label +fails the node). Each row describes its target as compiled (its own entity, measure and filter), with its hierarchy from the surface's registry. The kernel refuses a population whose installed weights are not the solution's, a result from another solve, and a surface other than the one the problem diff --git a/packages/microcosm-build/src/microcosm/build/transport/terminal_kernels.py b/packages/microcosm-build/src/microcosm/build/transport/terminal_kernels.py index fb699fff1..6a72adf28 100644 --- a/packages/microcosm-build/src/microcosm/build/transport/terminal_kernels.py +++ b/packages/microcosm-build/src/microcosm/build/transport/terminal_kernels.py @@ -65,7 +65,11 @@ decode_problem, decode_solution, ) -from microcosm.diagnostics import CalibrationDiagnosticsV8 +from microcosm.calibrate.hierarchy_labels import hierarchy_label_choices +from microcosm.diagnostics import ( + CalibrationDiagnosticsV8, + catalog_calibration_diagnostics, +) from microcosm.frame import EntitySchema, Frame, engine_tables from microcosm.frame.adapters.axiom import AxiomEntityTableDataset from microcosm.graph import ( @@ -278,7 +282,13 @@ def run(self, context: KernelContext) -> KernelResult: "surface_sha256": surface.sha256, }, ) - model = CalibrationDiagnosticsV8.model_validate(payload) + # Published as schema 9: the hierarchy labels go into catalogs, and a + # row set that labels one identifier two ways without a chosen label + # is refused (HierarchyLabelConflictError). + model = catalog_calibration_diagnostics( + CalibrationDiagnosticsV8.model_validate(payload), + label_choices=hierarchy_label_choices(surface.registry.country), + ) diagnostics = canonical_json( model.model_dump(mode="python", exclude_defaults=True) ) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py index 7bcf1ea29..e05063395 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py @@ -33,6 +33,7 @@ effective_sample_size, ) from microcosm.calibrate.diagnostics import past_cap_census +from microcosm.calibrate.hierarchy_labels import UK_HIERARCHY_LABEL_CHOICES from microcosm.calibrate.solve import CalibrationResult from microcosm.diagnostics import ( UK_DIAGNOSTICS_SCHEMA_VERSION, @@ -42,6 +43,7 @@ UK_WITHHELD_PUBLISHED_OPTIONS, CalibrationDiagnosticsV8, DiagnosticsWriteOutcome, + HierarchyLabelChoices, failed_diagnostics_outcome, ) from microcosm.diagnostics import ( @@ -49,10 +51,13 @@ ) from microcosm.frame import Frame +from .local_authority_input import load_uk_local_authority_names_resource + __all__ = [ "UK_DIAGNOSTICS_SCHEMA_VERSION", "UK_TARGET_GEOGRAPHY_LEVELS", "uk_calibration_diagnostics_payload", + "uk_hierarchy_label_choices", "uk_consumption_drift", "uk_fit_by_family", "uk_support_limited_misses", @@ -1403,6 +1408,25 @@ def uk_calibration_diagnostics_payload( ) +def uk_hierarchy_label_choices() -> HierarchyLabelChoices: + """The UK's label choices, with the ONS April 2023 local-authority names. + + UK publishers name one area several ways, and Chronicle's local facts + carry the area as a ``geography`` dimension value (HMRC "Hartlepool UA", + ONS "Hartlepool"). Diagnostics schema 9 holds one label per area: the + payload's own geography label for that code, else Microcosm's catalogs, + ``UK_GEOGRAPHY_ID_TO_LABEL`` and ``local_authority_names.json`` + (:mod:`microcosm.calibrate.hierarchy_labels`). + """ + + areas = load_uk_local_authority_names_resource()["areas"] + return UK_HIERARCHY_LABEL_CHOICES.merged( + HierarchyLabelChoices( + geographies={code: str(entry["name"]) for code, entry in areas.items()} + ) + ) + + def write_uk_calibration_diagnostics( result: CalibrationResult, path: Path | str, @@ -1438,4 +1462,6 @@ def write_uk_calibration_diagnostics( ) except Exception as error: # diagnostics must not escape without validation return failed_diagnostics_outcome(error, path=output) - return write_typed_calibration_diagnostics(diagnostics, output) + return write_typed_calibration_diagnostics( + diagnostics, output, label_choices=uk_hierarchy_label_choices() + ) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py index 735733f80..b04616288 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py @@ -946,8 +946,14 @@ def run(self, context: KernelContext) -> KernelResult: ) diagnostics = evidence.artifacts["target_diagnostics"] support = support_frame.to_dict(orient="records") + from microcosm.diagnostics import ( + CalibrationDiagnosticsV8, + catalog_calibration_diagnostics, + ) + from .diagnostics import ( uk_calibration_diagnostics_payload, + uk_hierarchy_label_choices, uk_withhold_brma_figures, ) @@ -995,10 +1001,18 @@ def run(self, context: KernelContext) -> KernelResult: }, withheld=withheld, ) + # Published as schema 9, as the canonical writer publishes it: + # one label per area, whichever publisher named it + # (uk_hierarchy_label_choices); an unresolved second label is + # refused (HierarchyLabelConflictError). + published_diagnostics = catalog_calibration_diagnostics( + CalibrationDiagnosticsV8.model_validate(complete_diagnostics), + label_choices=uk_hierarchy_label_choices(), + ).model_dump(mode="python", exclude_defaults=True) output_artifacts.update( { "calibration_diagnostics": encode_stage_evidence( - _json_safe(complete_diagnostics) + _json_safe(published_diagnostics) ), "target_diagnostics_csv": pd.DataFrame( uk_withhold_brma_figures(diagnostics, withheld) diff --git a/packages/microcosm-build/tests/engine_free/shared/test_transport_terminal_kernels.py b/packages/microcosm-build/tests/engine_free/shared/test_transport_terminal_kernels.py index 8864f859c..70a4bd187 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_transport_terminal_kernels.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_transport_terminal_kernels.py @@ -58,7 +58,8 @@ ) from microcosm.calibrate.artifacts import PROBLEM_TYPE, RESULT_TYPE, SOLUTION_TYPE from microcosm.diagnostics import ( - CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, + expand_calibration_diagnostics, parse_calibration_diagnostics, write_calibration_diagnostics, ) @@ -144,12 +145,15 @@ def test_full_build_passes_its_gates_readback_and_package(toy_run) -> None: # --------------------------------------------------------------------------- -def test_diagnostics_validate_against_schema_8(toy_run, tmp_path) -> None: +def test_diagnostics_validate_against_schema_9(toy_run, tmp_path) -> None: payload = toy_run.artifact(DIAGNOSTICS, "diagnostics") - model = parse_calibration_diagnostics(json.loads(payload)) + published = parse_calibration_diagnostics(json.loads(payload)) - assert isinstance(model, CalibrationDiagnosticsV8) - assert model.schema_version == DIAGNOSTICS_TYPE.schema_version == 8 + assert isinstance(published, CalibrationDiagnosticsV9) + assert published.schema_version == DIAGNOSTICS_TYPE.schema_version == 9 + # The rows' labels sit in the published catalogs; expanded, each row + # carries its schema-8 hierarchy again. + model = expand_calibration_diagnostics(published) assert model.target_registry.country == COUNTRY assert model.n_records == 6 assert [row.target_name for row in model.targets] == [ @@ -178,7 +182,7 @@ def test_diagnostics_validate_against_schema_8(toy_run, tmp_path) -> None: outcome = write_calibration_diagnostics(model, tmp_path / "diagnostics.json") assert outcome.status == "available" assert ( - canonical_json(model.model_dump(mode="python", exclude_defaults=True)) + canonical_json(published.model_dump(mode="python", exclude_defaults=True)) == payload ) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_dense_release_assembler.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_dense_release_assembler.py index f3f42873c..076b55736 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_dense_release_assembler.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_dense_release_assembler.py @@ -493,16 +493,22 @@ def _candidate_dir( return candidate, spine, incumbent_manifest +@pytest.mark.parametrize("schema_version", [8, 9]) def test_current_diagnostics_are_validated_then_copied_without_rewriting( tmp_path: Path, monkeypatch, + schema_version: int, ) -> None: + # A candidate written before schema 9 carries schema 8; both carry the + # calibration hierarchy and are copied byte for byte. assembler = _load("assemble_uk_dense_release_dir") source = tmp_path / "source.json" destination = tmp_path / "release.json" - source_bytes = b'{"schema_version":8,"source":"exact bytes"}\n' + source_bytes = ( + f'{{"schema_version":{schema_version},"source":"exact bytes"}}\n'.encode() + ) source.write_bytes(source_bytes) - diagnostics = {"schema_version": 8, "source": "exact bytes"} + diagnostics = {"schema_version": schema_version, "source": "exact bytes"} captured: dict[str, object] = {} def canonical_parser(payload: object) -> object: diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_diagnostics.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_diagnostics.py index a7ef17e6f..9fab28362 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_diagnostics.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_diagnostics.py @@ -13,6 +13,7 @@ UK_DIAGNOSTICS_SCHEMA_VERSION, UK_TARGET_GEOGRAPHY_LEVELS, uk_calibration_diagnostics_payload, + uk_hierarchy_label_choices, uk_support_limited_misses, uk_weakest_areas_by_fit, uk_weakest_families, @@ -21,7 +22,7 @@ write_uk_calibration_diagnostics, ) from microcosm.calibrate import ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, CalibrationHierarchy, HierarchyCategory, HierarchyGeography, @@ -31,10 +32,13 @@ diagnostics_payload, score_targets, ) +from microcosm.calibrate.geography_constants import UK_GEOGRAPHY_ID_TO_LABEL from microcosm.diagnostics import ( CalibrationDiagnosticsV8, DiagnosticsWriteFailure, DiagnosticsWriteSuccess, + HierarchyLabelChoices, + catalog_calibration_diagnostics, ) from microcosm.frame import EntitySchema, Frame, WeightKind, Weights from test_support.microcosm_diagnostics.json_schema import ( @@ -562,7 +566,7 @@ def test_payload_preserves_common_schema_and_adds_versioned_uk_evidence() -> Non build={"release_id": "fixture"}, ) - assert payload["schema_version"] == CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + assert payload["schema_version"] == CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION assert "ess_fraction" not in payload assert all("geography_level" not in row for row in payload["targets"]) assert payload["target_registry"] == { @@ -939,7 +943,8 @@ def test_writer_round_trips_strict_json(tmp_path: Path) -> None: assert isinstance(outcome, DiagnosticsWriteSuccess) assert outcome.path == path - assert json.loads(path.read_text(encoding="utf-8")) == ( + assert outcome.schema_version == 9 + assembled = CalibrationDiagnosticsV8.model_validate( uk_calibration_diagnostics_payload( result, frame, @@ -947,6 +952,11 @@ def test_writer_round_trips_strict_json(tmp_path: Path) -> None: target_registry=registry, ) ) + assert json.loads(path.read_text(encoding="utf-8")) == json.loads( + catalog_calibration_diagnostics( + assembled, label_choices=uk_hierarchy_label_choices() + ).model_dump_json(exclude_defaults=True) + ) invalid = write_uk_calibration_diagnostics( result, path, @@ -1058,13 +1068,16 @@ def test_writer_delegates_a_typed_payload_to_the_canonical_writer( def canonical_writer( diagnostics: CalibrationDiagnosticsV8, destination: Path | str, + *, + label_choices: HierarchyLabelChoices | None = None, ) -> DiagnosticsWriteSuccess: captured["diagnostics"] = diagnostics captured["destination"] = Path(destination) + captured["label_choices"] = label_choices return DiagnosticsWriteSuccess( status="available", path=Path(destination), - schema_version=8, + schema_version=9, sha256="a" * 64, ) @@ -1084,6 +1097,26 @@ def canonical_writer( assert isinstance(outcome, DiagnosticsWriteSuccess) assert isinstance(captured["diagnostics"], CalibrationDiagnosticsV8) assert captured["destination"] == path + assert captured["label_choices"] == uk_hierarchy_label_choices() + + +def test_uk_label_choices_name_every_area_microcosm_has_a_catalog_for() -> None: + # UK publishers name one area several ways ("Hartlepool UA" from HMRC, + # "Hartlepool" from ONS). Diagnostics schema 9 holds one label per area, + # taken from Microcosm's own catalogs: UK_GEOGRAPHY_ID_TO_LABEL for the + # UK, its nations and English regions, and the ONS April 2023 names in + # local_authority_names.json for local authorities. + choices = uk_hierarchy_label_choices() + + assert choices.geography_dimensions == frozenset({"geography"}) + assert dict(choices.dimension_values) == {} + for code, label in UK_GEOGRAPHY_ID_TO_LABEL.items(): + assert choices.geographies[code] == label + assert choices.geographies["E06000001"] == "Hartlepool" + assert choices.geographies["E06000023"] == "Bristol, City of" + assert choices.geographies["E08000013"] == "St. Helens" + assert choices.geographies["W06000001"] == "Isle of Anglesey" + assert len(choices.geographies) == len(UK_GEOGRAPHY_ID_TO_LABEL) + 361 def test_geography_level_vocabulary_includes_future_release_levels() -> None: diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_local_candidate_scorer.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_local_candidate_scorer.py index 745a468a2..42554ad50 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_local_candidate_scorer.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_local_candidate_scorer.py @@ -228,6 +228,25 @@ def test_local_scorer_reports_per_family_wins_and_the_measured_holdout() -> None assert result["incumbent_missing_areas"] == {} +@pytest.mark.parametrize("schema_version", [8, 9]) +def test_local_scorer_reads_either_hierarchy_schema(schema_version: int) -> None: + # The scorer reads names, metadata and final estimates, which schema 9 + # keeps on every row; only the hierarchy labels moved to catalogs. + scorer = _load_scorer() + registry, candidate, weights, metrics = _case() + candidate["schema_version"] = schema_version + + result = scorer.score_uk_local_candidate( + candidate_diagnostics=candidate, + incumbent_weights=weights, + incumbent_metrics=metrics, + target_registry=registry, + expected_reference_count=2, + ) + + assert result["rows_compared"] == 2 + + def test_local_scorer_refuses_a_non_frozen_surface() -> None: scorer = _load_scorer() registry, candidate, weights, metrics = _case() diff --git a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py index 4fa27b499..e5c3dfc8d 100644 --- a/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py +++ b/packages/microcosm-build/tests/engine_free/us/test_us_fiscal_refresh_builder.py @@ -5984,26 +5984,23 @@ def test_release_calibration_diagnostics_writes_nan_final_loss_as_null( written = _write_release_diagnostics_with_nan_final_loss(small_frame, tmp_path) diagnostics = json.loads(written.read_text()) - assert diagnostics["schema_version"] == 8 + # Published as schema 9: the labels sit in catalogs, the row refers to them. + assert diagnostics["schema_version"] == 9 assert diagnostics["targets"][0]["source"] == "fixture" assert diagnostics["targets"][0]["hierarchy"] == { - "provider": { - "id": "irs_soi", - "label": "IRS Statistics of Income", - }, - "category": { - "id": "irs_soi.income", - "label": "Income", - "provider_id": "irs_soi", - }, - "geography": { - "id": "0400000US06", - "label": "California", - "level": "state", - }, + "category_id": "irs_soi.income", + "geography_id": "0400000US06", "dimensions": [], "target": {"id": "income", "label": "Income"}, } + assert diagnostics["hierarchy_catalogs"] == { + "providers": [{"id": "irs_soi", "label": "IRS Statistics of Income"}], + "categories": [ + {"id": "irs_soi.income", "label": "Income", "provider_id": "irs_soi"} + ], + "geographies": [{"id": "0400000US06", "label": "California", "level": "state"}], + "dimensions": [], + } assert diagnostics["final_loss"] is None assert diagnostics["build"]["default_dataset"]["final_loss"] is None diff --git a/packages/microcosm-calibrate/README.md b/packages/microcosm-calibrate/README.md index 0ea58c348..f612653cb 100644 --- a/packages/microcosm-calibrate/README.md +++ b/packages/microcosm-calibrate/README.md @@ -85,9 +85,13 @@ label only when Chronicle has no label for that field. The dashboard consumes these identifiers and labels verbatim and does not infer schema-8 display text. Generic in-memory payloads whose targets carry no hierarchy retain schema 6 for historical readers; a payload cannot mix hierarchy-bearing and hierarchy-free -targets. Current release builders write only schema 8. They pass a complete -`TargetRegistry` to `write_calibration_diagnostics`, which validates the typed -document before an atomic write. The writer returns either `available` with the +targets. Current release builders assemble schema 8 and publish schema 9: they +pass a complete `TargetRegistry` to `write_calibration_diagnostics`, which +validates the typed document, declares each provider, category, geography and +dimension label once in `hierarchy_catalogs`, and writes it atomically. Rows +that give one identifier two labels are refused (`label_conflict`) unless the +registry country's reviewed choices in `microcosm.calibrate.hierarchy_labels` +name the label. The writer returns either `available` with the file digest or `failed` with a structured error. A failed diagnostics attempt does not invalidate the calibrated dataset. diff --git a/packages/microcosm-calibrate/src/microcosm/calibrate/__init__.py b/packages/microcosm-calibrate/src/microcosm/calibrate/__init__.py index bdab67f1c..c8043cc92 100644 --- a/packages/microcosm-calibrate/src/microcosm/calibrate/__init__.py +++ b/packages/microcosm-calibrate/src/microcosm/calibrate/__init__.py @@ -89,6 +89,7 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: TARGET_LOSS_BASIS_HASH_ALGORITHM, ) from microcosm.calibrate.diagnostics import ( # noqa: E402 - after the compat gate + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, diagnostics_payload, past_cap_census, @@ -185,6 +186,7 @@ def _assert_frame_compatible(version: str, required: tuple[int, int]) -> None: "score_predictions", "ssa_population_projection", "static_aging", + "CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION", "CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION", "DiagnosticsWriteFailure", "DiagnosticsWriteOutcome", diff --git a/packages/microcosm-calibrate/src/microcosm/calibrate/diagnostics.py b/packages/microcosm-calibrate/src/microcosm/calibrate/diagnostics.py index f6be6c598..7b711259e 100644 --- a/packages/microcosm-calibrate/src/microcosm/calibrate/diagnostics.py +++ b/packages/microcosm-calibrate/src/microcosm/calibrate/diagnostics.py @@ -33,11 +33,14 @@ TargetLossAttributionError, assemble_target_loss_attribution, ) +from microcosm.calibrate.hierarchy_labels import hierarchy_label_choices from microcosm.calibrate.solve import CalibrationResult from microcosm.diagnostics import ( + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, CalibrationDiagnosticsV8, DiagnosticsWriteOutcome, + HierarchyLabelChoices, failed_diagnostics_outcome, ) from microcosm.diagnostics import ( @@ -45,6 +48,7 @@ ) __all__ = [ + "CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION", "CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION", "diagnostics_payload", "past_cap_census", @@ -68,6 +72,9 @@ #: v8 replaces those parallel inferred fields with one complete, ordered #: hierarchy carried by each registry target: provider, category, geography, #: zero or more dimensions, and target. +#: v9 declares the provider, category, geography and dimension labels once, +#: in ``hierarchy_catalogs``, and the rows refer to them by ID. Payloads here +#: are still assembled as v8; the writer publishes v9. _HIERARCHY_FREE_DIAGNOSTICS_SCHEMA_VERSION = 6 _LOGGER = logging.getLogger(__name__) @@ -416,8 +423,10 @@ def diagnostics_payload( "A diagnostics payload cannot mix targets with and without calibration " f"hierarchies; missing {missing_hierarchy_names[:5]!r}." ) + # The in-memory document is schema 8, labels on every row; the writer + # publishes it as schema 9 with the labels in catalogs. schema_version = ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION if target_registry is not None or hierarchy_count else _HIERARCHY_FREE_DIAGNOSTICS_SCHEMA_VERSION ) @@ -498,8 +507,9 @@ def write_calibration_diagnostics( target_registry: object | None = None, build: dict[str, Any] | None = None, target_surface: Mapping[str, object] | None = None, + label_choices: HierarchyLabelChoices | None = None, ) -> DiagnosticsWriteOutcome: - """Construct, validate, and atomically write schema-8 diagnostics. + """Construct, validate, and atomically write schema-9 diagnostics. The conventional filename is ``calibration_diagnostics.json`` inside a release directory, alongside ``build_manifest.json``. @@ -507,6 +517,9 @@ def write_calibration_diagnostics( Args: result: The :func:`~microcosm.calibrate.solve.calibrate` output. path: Destination file path; parent directories must exist. + label_choices: Labels for identifiers the rows spell more than one + way; defaults to the registry country's + (:func:`~microcosm.calibrate.hierarchy_labels.hierarchy_label_choices`). A diagnostics failure is recorded in the returned outcome and logged, but does not raise through the dataset build. Current release artifacts require @@ -531,4 +544,10 @@ def write_calibration_diagnostics( ) except Exception as error: # diagnostics must never abort the dataset build return failed_diagnostics_outcome(error, path=path) - return write_typed_calibration_diagnostics(diagnostics, path) + if label_choices is None: + label_choices = hierarchy_label_choices( + getattr(target_registry, "country", None) + ) + return write_typed_calibration_diagnostics( + diagnostics, path, label_choices=label_choices + ) diff --git a/packages/microcosm-calibrate/src/microcosm/calibrate/hierarchy_labels.py b/packages/microcosm-calibrate/src/microcosm/calibrate/hierarchy_labels.py new file mode 100644 index 000000000..b41673143 --- /dev/null +++ b/packages/microcosm-calibrate/src/microcosm/calibrate/hierarchy_labels.py @@ -0,0 +1,151 @@ +"""Labels chosen for identifiers that a country's sources spell differently. + +Calibration diagnostics schema 9 declares each geography and dimension-value +label once, so the writer refuses target rows that label one identifier two +ways (``microcosm.diagnostics.catalog_calibration_diagnostics``), unless a +label is chosen here. A choice applies only where the rows disagree, so an +identifier that every row labels one way keeps that label. + +The choices cover the spellings found in published releases: + +- UK: Chronicle's local facts carry their area as a dimension named + ``geography`` whose value IDs are lower-case GSS codes, and each publisher + names the area its own way (HMRC "Hartlepool UA", ONS "Hartlepool"; 88 areas + in the October 2026 dense release, and "Yorkshire And The Humber" beside + "Yorkshire and The Humber" in the national releases). Such a value takes + its area's label: the payload's own geography label for that code, else + ``UK_GEOGRAPHY_ID_TO_LABEL``. The UK build's writer adds the ONS + local-authority names (``microcosm.build.uk_runtime.diagnostics``). +- US: IRS SOI and Census tables word three dimension values differently + (release ``populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z``, built + from the pinned feed at Chronicle ``51aa40f``, which still produces them). + Chronicle registers the two spellings of each as reviewed variants + (``REVIEWED_VALUE_LABEL_VARIANTS``, chronicle#377). One label for each was + chosen in review on 2026-10-21; a US feed re-pin brings the same labels from + Chronicle, so releases built after it carry one spelling and these choices + then apply only to older feeds. Each choice covers only the two reviewed + spellings, so a third spelling is still refused. +- Belgium: each Belgian Chronicle package states its national figure as the + group-by value ``measure`` = ``country_total`` and labels it with its own + measure name (SPF Finances, ONSS, ONEM and NBB; Chronicle registers the four + as reviewed variants, chronicle#265). The value is written as "Country + total", the label chosen in review on 2026-10-22 and the one Chronicle's + canonical register gives it (chronicle#377). Without it the transport + graph's diagnostics step would raise and stop a Belgian build. The choice + covers only the four package spellings, so another is still refused. + +Every disagreement a choice settles is logged as a warning. + +Any other spelling difference still stops the writer; correct the source or +add a reviewed choice here. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from types import MappingProxyType + +from microcosm.calibrate.geography_constants import UK_GEOGRAPHY_ID_TO_LABEL +from microcosm.diagnostics import HierarchyLabelChoices + +__all__ = [ + "BE_DIMENSION_VALUE_LABEL_CHOICES", + "BE_DIMENSION_VALUE_REVIEWED_SPELLINGS", + "BE_HIERARCHY_LABEL_CHOICES", + "UK_HIERARCHY_LABEL_CHOICES", + "US_DIMENSION_VALUE_LABEL_CHOICES", + "US_DIMENSION_VALUE_REVIEWED_SPELLINGS", + "US_HIERARCHY_LABEL_CHOICES", + "hierarchy_label_choices", +] + +_ADJUSTED_GROSS_INCOME = "us:statutes/26/62#adjusted_gross_income" + +#: ``(dimension ID, value ID)`` -> label, for the US values two sources word +#: differently. Counts are targets in release 0fb05b6. +US_DIMENSION_VALUE_LABEL_CHOICES: Mapping[tuple[str, str], str] = MappingProxyType( + { + # Seven SOI packages, including Table 1.1 (106 targets); the + # filing-season week-47 EITC table's column header adds "and deficit" + # (2 targets). chronicle#377 changes that package's label to this one; + # the variant leaves the feed at its next re-pin. + (_ADJUSTED_GROSS_INCOME, "under_1"): "No adjusted gross income", + # SOI Tables 1.2, 1.4, 2.1 and 4.3 (53 targets); Historic Table 2 and + # Table 1.1 write "All returns" (47). + (_ADJUSTED_GROSS_INCOME, "all"): "All returns, total", + # Census PEP state estimates by age (51 targets); the national table + # writes "Age 85 and over" (1 target). + ("age", "85_plus"): "Age 85 plus", + } +) +#: The spellings each US choice was reviewed for; another is refused. +US_DIMENSION_VALUE_REVIEWED_SPELLINGS: Mapping[tuple[str, str], frozenset[str]] = ( + MappingProxyType( + { + (_ADJUSTED_GROSS_INCOME, "under_1"): frozenset( + {"No adjusted gross income", "No adjusted gross income and deficit"} + ), + (_ADJUSTED_GROSS_INCOME, "all"): frozenset( + {"All returns", "All returns, total"} + ), + ("age", "85_plus"): frozenset({"Age 85 plus", "Age 85 and over"}), + } + ) +) + +_COUNTRY_TOTAL = ("measure", "country_total") + +#: ``(dimension ID, value ID)`` -> label, for the Belgian value each source +#: package labels its own way. +BE_DIMENSION_VALUE_LABEL_CHOICES: Mapping[tuple[str, str], str] = MappingProxyType( + {_COUNTRY_TOTAL: "Country total"} +) +#: The spellings the Belgian choice was reviewed for; another is refused. +BE_DIMENSION_VALUE_REVIEWED_SPELLINGS: Mapping[tuple[str, str], frozenset[str]] = ( + MappingProxyType( + { + _COUNTRY_TOTAL: frozenset( + { + # spf_finances/pit_2023 (calibration target + # spf_finances_pit_total) + "Belgium PIT federal and local tax before withholding", + # onss/contributions_2024 (onss_employee_contribution_total) + "Worker-borne personal social security contributions", + # onem_rva/unemployment_2024 (onem_unemployment_caseload) + "Complete unemployment benefit recipients", + # nbb/national_accounts_household_disposable_income_2024 + # (validation target nbb_household_disposable_income) + "Household disposable income gross", + } + ), + } + ) +) + +UK_HIERARCHY_LABEL_CHOICES = HierarchyLabelChoices( + geographies=UK_GEOGRAPHY_ID_TO_LABEL, + geography_dimensions=frozenset({"geography"}), +) +US_HIERARCHY_LABEL_CHOICES = HierarchyLabelChoices( + dimension_values=US_DIMENSION_VALUE_LABEL_CHOICES, + dimension_value_spellings=US_DIMENSION_VALUE_REVIEWED_SPELLINGS, +) +BE_HIERARCHY_LABEL_CHOICES = HierarchyLabelChoices( + dimension_values=BE_DIMENSION_VALUE_LABEL_CHOICES, + dimension_value_spellings=BE_DIMENSION_VALUE_REVIEWED_SPELLINGS, +) +_CHOICES_BY_COUNTRY: Mapping[str, HierarchyLabelChoices] = MappingProxyType( + { + "be": BE_HIERARCHY_LABEL_CHOICES, + "uk": UK_HIERARCHY_LABEL_CHOICES, + "us": US_HIERARCHY_LABEL_CHOICES, + } +) + + +def hierarchy_label_choices(country: str | None) -> HierarchyLabelChoices: + """The label choices for a registry's country; none for other countries.""" + + return _CHOICES_BY_COUNTRY.get( + (country or "").strip().lower(), HierarchyLabelChoices() + ) diff --git a/packages/microcosm-calibrate/tests/engine_free/shared/test_diagnostics.py b/packages/microcosm-calibrate/tests/engine_free/shared/test_diagnostics.py index 8e289cfa3..0e90773b4 100644 --- a/packages/microcosm-calibrate/tests/engine_free/shared/test_diagnostics.py +++ b/packages/microcosm-calibrate/tests/engine_free/shared/test_diagnostics.py @@ -18,7 +18,7 @@ import pytest from microcosm.calibrate import ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, TARGET_LOSS_ATTRIBUTION_WARNING_CODES, TARGET_LOSS_BASIS_HASH_ALGORITHM, CalibrationHierarchy, @@ -48,6 +48,11 @@ US_STATE_POSTAL_TO_NUMERIC_FIPS, ) from microcosm.calibrate.solve import zero_target_loss_census +from microcosm.diagnostics import ( + CalibrationDiagnosticsV8, + expand_calibration_diagnostics, + parse_calibration_diagnostics, +) from test_support.paths import paths_for _TEST_PATHS = paths_for("microcosm-calibrate") @@ -272,7 +277,9 @@ def test_payload_uses_compiled_target_hierarchy_without_registry( payload = diagnostics_payload(result) - assert payload["schema_version"] == CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + # Producers assemble schema 8, labels on every row; the writer publishes + # it as schema 9 with the labels in catalogs. + assert payload["schema_version"] == CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION assert payload["targets"][0]["hierarchy"] == { "provider": { "id": "census_pep", @@ -815,7 +822,11 @@ def test_payload_can_carry_target_registry_identity( ) assert outcome.status == "available" payload["target_surface"] = precomputed_target_surface - assert json.loads(output.read_text()) == payload + written = json.loads(output.read_text()) + assert written["schema_version"] == 9 + assert expand_calibration_diagnostics( + parse_calibration_diagnostics(written) + ) == CalibrationDiagnosticsV8.model_validate(payload) def test_registry_diagnostics_publish_uk_geography_and_all_ledger_dimensions( diff --git a/packages/microcosm-calibrate/tests/engine_free/shared/test_hierarchy_label_choices.py b/packages/microcosm-calibrate/tests/engine_free/shared/test_hierarchy_label_choices.py new file mode 100644 index 000000000..aeaf1a547 --- /dev/null +++ b/packages/microcosm-calibrate/tests/engine_free/shared/test_hierarchy_label_choices.py @@ -0,0 +1,376 @@ +"""Labels chosen for identifiers that a country's sources spell more than one way. + +Diagnostics schema 9 holds one label per identifier, so the writer refuses +rows that label one identifier two ways unless a label is chosen for it. The +choices below cover the spellings found in published releases: UK publishers' +names for one area (HMRC's "Hartlepool UA" beside ONS's "Hartlepool"), which +reach the rows as values of Chronicle's ``geography`` dimension, and three US +dimension values that two IRS SOI tables or two Census tables word +differently. They also cover the Belgian ``measure`` = ``country_total`` +value, which each Belgian Chronicle package labels its own way. +""" + +from __future__ import annotations + +import json +import logging +from pathlib import Path + +import pytest + +from microcosm.calibrate import ( + CalibrationHierarchy, + HierarchyCategory, + HierarchyDimension, + HierarchyGeography, + HierarchyNode, + TargetRegistry, + TargetSpec, + score_targets, + write_calibration_diagnostics, +) +from microcosm.calibrate.geography_constants import UK_GEOGRAPHY_ID_TO_LABEL +from microcosm.calibrate.hierarchy_labels import ( + US_DIMENSION_VALUE_LABEL_CHOICES, + hierarchy_label_choices, +) +from microcosm.diagnostics import HierarchyLabelChoices + +AGI = "us:statutes/26/62#adjusted_gross_income" +COUNTRY_TOTAL = ("measure", "country_total") + +#: Belgium's country-total targets (``be/target_references.json``), each read +#: from its own Chronicle package, which labels the group-by value ``measure`` +#: = ``country_total`` its own way (``source_package.yaml`` at Chronicle +#: ``6d74e42``, registered there as reviewed variants). The NBB target has +#: ``target_role`` "validation"; the other three are calibration targets. +BELGIAN_COUNTRY_TOTAL_TARGETS = ( + ( + "spf_finances_pit_total", + HierarchyNode("spf_finances", "FPS Finance"), + HierarchyCategory( + "spf_finances.income_tax", "Personal income tax", "spf_finances" + ), + "Belgium PIT federal and local tax before withholding", + ), + ( + "onss_employee_contribution_total", + HierarchyNode("onss", "National Social Security Office"), + HierarchyCategory( + "onss.social_security", "Social security contributions", "onss" + ), + "Worker-borne personal social security contributions", + ), + ( + "onem_unemployment_caseload", + HierarchyNode("onem", "National Employment Office"), + HierarchyCategory("onem.caseloads", "Benefit caseloads", "onem"), + "Complete unemployment benefit recipients", + ), + ( + "nbb_household_disposable_income", + HierarchyNode("nbb", "National Bank of Belgium"), + HierarchyCategory("nbb.national_accounts", "National accounts", "nbb"), + "Household disposable income gross", + ), +) +BELGIAN_COUNTRY_TOTAL_SPELLINGS = frozenset( + value_label for *_target, value_label in BELGIAN_COUNTRY_TOTAL_TARGETS +) + + +def test_uk_choices_name_areas_from_the_authoritative_catalog() -> None: + choices = hierarchy_label_choices("uk") + + assert dict(choices.geographies) == dict(UK_GEOGRAPHY_ID_TO_LABEL) + assert choices.geography_dimensions == frozenset({"geography"}) + assert dict(choices.dimension_values) == {} + assert hierarchy_label_choices("UK") == choices + + +def test_us_choices_name_the_three_values_us_sources_word_differently() -> None: + choices = hierarchy_label_choices("us") + + # The labels chosen in review on 2026-10-21, which the US feed re-pin also + # carries from Chronicle; older feeds and releases keep both spellings. + assert dict(choices.dimension_values) == { + (AGI, "under_1"): "No adjusted gross income", + (AGI, "all"): "All returns, total", + ("age", "85_plus"): "Age 85 plus", + } + assert dict(US_DIMENSION_VALUE_LABEL_CHOICES) == dict(choices.dimension_values) + # Each choice covers exactly the spellings found in release 0fb05b6 and + # registered by chronicle#377; a third spelling is still refused. + assert dict(choices.dimension_value_spellings) == { + (AGI, "under_1"): frozenset( + {"No adjusted gross income", "No adjusted gross income and deficit"} + ), + (AGI, "all"): frozenset({"All returns", "All returns, total"}), + ("age", "85_plus"): frozenset({"Age 85 plus", "Age 85 and over"}), + } + assert dict(choices.geographies) == {} + assert choices.geography_dimensions == frozenset() + + +def test_the_belgian_choice_covers_the_four_country_total_spellings() -> None: + choices = hierarchy_label_choices("be") + + # The label chosen in review on 2026-10-22, which is also the label + # Chronicle's canonical register gives this value (chronicle#377). + assert dict(choices.dimension_values) == {COUNTRY_TOTAL: "Country total"} + # The choice covers exactly the four package spellings Chronicle registers + # as reviewed variants; another spelling is still refused. + assert dict(choices.dimension_value_spellings) == { + COUNTRY_TOTAL: BELGIAN_COUNTRY_TOTAL_SPELLINGS + } + assert len(BELGIAN_COUNTRY_TOTAL_SPELLINGS) == 4 + assert dict(choices.geographies) == {} + assert choices.geography_dimensions == frozenset() + assert hierarchy_label_choices("BE") == choices + + +@pytest.mark.parametrize("country", ["nz", "toy", None]) +def test_other_countries_choose_no_labels(country: str | None) -> None: + assert hierarchy_label_choices(country) == HierarchyLabelChoices() + + +def _spec( + name: str, + value: float, + *, + geography: HierarchyGeography, + dimension: HierarchyDimension, + provider: HierarchyNode | None = None, + category: HierarchyCategory | None = None, +) -> TargetSpec: + return TargetSpec( + name=name, + entity="household", + measure="household_count", + value=value, + period=2025, + source="fixture | Fixture table | https://example.test/fixture", + family="fixture", + metadata={}, + hierarchy=CalibrationHierarchy( + provider=provider or HierarchyNode("fixture", "Fixture publisher"), + category=category + or HierarchyCategory("fixture.households", "Households", "fixture"), + geography=geography, + dimensions=(dimension,), + target=HierarchyNode(name, name.replace("_", " ")), + ), + ) + + +def _uk_area_registry(country: str, population: float) -> TargetRegistry: + hartlepool = HierarchyGeography("E06000001", "Hartlepool", level="local_authority") + return TargetRegistry( + ( + _spec( + "hmrc_households_e06000001", + population, + geography=hartlepool, + dimension=HierarchyDimension( + "geography", "Geography", "e06000001", "Hartlepool UA" + ), + ), + _spec( + "ons_households_e06000001", + population, + geography=hartlepool, + dimension=HierarchyDimension( + "geography", "Geography", "e06000001", "Hartlepool" + ), + ), + ), + country=country, + ) + + +def _us_band_registry(country: str, population: float) -> TargetRegistry: + nation = HierarchyGeography("0100000US", "United States", level="country") + return TargetRegistry( + ( + _spec( + "soi_returns_under_1", + population, + geography=nation, + dimension=HierarchyDimension( + AGI, "Adjusted gross income", "under_1", "No adjusted gross income" + ), + ), + _spec( + "eitc_returns_under_1", + population, + geography=nation, + dimension=HierarchyDimension( + AGI, + "Adjusted gross income", + "under_1", + "No adjusted gross income and deficit", + ), + ), + ), + country=country, + ) + + +def _belgian_country_total_registry( + population: float, + *, + extra_value_label: str | None = None, +) -> TargetRegistry: + """Belgium's country-total targets, each with its package's value label. + + ``extra_value_label`` adds a fifth target from another package whose + spelling of the value nobody has reviewed. + """ + + belgium = HierarchyGeography("BE", "Belgium", level="country") + targets = list(BELGIAN_COUNTRY_TOTAL_TARGETS) + if extra_value_label is not None: + targets.append( + ( + "statbel_country_total", + HierarchyNode("statbel", "Statbel"), + HierarchyCategory("statbel.demography", "Demography", "statbel"), + extra_value_label, + ) + ) + return TargetRegistry( + tuple( + _spec( + target_name, + population, + geography=belgium, + dimension=HierarchyDimension( + "measure", "Measure", "country_total", value_label + ), + provider=provider, + category=category, + ) + for target_name, provider, category, value_label in targets + ), + country="be", + ) + + +def _written_values(path: Path) -> list[dict]: + (dimension,) = json.loads(path.read_text())["hierarchy_catalogs"]["dimensions"] + return dimension["values"] + + +def test_the_writer_applies_the_registry_country_choices( + feasible_frame, + tmp_path: Path, +) -> None: + frame, truths = feasible_frame() + registry = _us_band_registry("us", truths["population"]) + result = score_targets(frame, registry.to_target_set()) + path = tmp_path / "calibration_diagnostics.json" + + outcome = write_calibration_diagnostics(result, path, target_registry=registry) + + assert outcome.status == "available" + assert _written_values(path) == [ + {"id": "under_1", "label": "No adjusted gross income"} + ] + + +def test_a_belgian_file_is_written_with_country_total_and_a_warning( + feasible_frame, + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + # Each Belgian package labels measure = country_total its own way. The + # transport graph's diagnostics step takes the same choices (by the + # registry's country) and raises on a refusal, so without a Belgian + # choice a Belgian build fails. + frame, truths = feasible_frame() + registry = _belgian_country_total_registry(truths["population"]) + result = score_targets(frame, registry.to_target_set()) + path = tmp_path / "calibration_diagnostics.json" + + with caplog.at_level("WARNING", logger="microcosm.diagnostics.catalogs"): + outcome = write_calibration_diagnostics(result, path, target_registry=registry) + + assert outcome.status == "available" + written = json.loads(path.read_text()) + assert written["schema_version"] == 9 + assert _written_values(path) == [{"id": "country_total", "label": "Country total"}] + (record,) = [ + record + for record in caplog.records + if record.name == "microcosm.diagnostics.catalogs" + ] + assert record.levelno == logging.WARNING + message = record.getMessage() + assert "label 1 identifier(s) more than one way" in message + assert "dimension measure value country_total: " in message + for value_label in BELGIAN_COUNTRY_TOTAL_SPELLINGS: + assert repr(value_label) in message + assert message.endswith("-> 'Country total'") + + +def test_a_belgian_spelling_nobody_reviewed_is_still_refused( + feasible_frame, + tmp_path: Path, +) -> None: + frame, truths = feasible_frame() + registry = _belgian_country_total_registry( + truths["population"], extra_value_label="Belgium country total" + ) + result = score_targets(frame, registry.to_target_set()) + path = tmp_path / "calibration_diagnostics.json" + + outcome = write_calibration_diagnostics(result, path, target_registry=registry) + + assert outcome.status == "failed" + assert outcome.error_code == "label_conflict" + assert ( + "dimension measure value country_total (a spelling its reviewed list " + "does not include)" + ) in outcome.message + assert not path.exists() + + +def test_the_uk_choices_give_an_area_its_own_geography_label( + feasible_frame, + tmp_path: Path, +) -> None: + # A UK registry needs the UK extension block, which the UK build's writer + # adds (microcosm.build.uk_runtime.diagnostics); the generic writer is + # given the UK choices explicitly here. + frame, truths = feasible_frame() + registry = _uk_area_registry("be", truths["population"]) + result = score_targets(frame, registry.to_target_set()) + path = tmp_path / "calibration_diagnostics.json" + + outcome = write_calibration_diagnostics( + result, + path, + target_registry=registry, + label_choices=hierarchy_label_choices("uk"), + ) + + assert outcome.status == "available" + assert _written_values(path) == [{"id": "e06000001", "label": "Hartlepool"}] + + +@pytest.mark.parametrize("registry_for", [_uk_area_registry, _us_band_registry]) +def test_a_country_without_choices_is_refused_the_same_spellings( + feasible_frame, + tmp_path: Path, + registry_for, +) -> None: + frame, truths = feasible_frame() + registry = registry_for("nz", truths["population"]) + result = score_targets(frame, registry.to_target_set()) + path = tmp_path / "calibration_diagnostics.json" + + outcome = write_calibration_diagnostics(result, path, target_registry=registry) + + assert outcome.status == "failed" + assert outcome.error_code == "label_conflict" + assert not path.exists() diff --git a/packages/microcosm-data/README.md b/packages/microcosm-data/README.md index 1347cf770..5561b14dd 100644 --- a/packages/microcosm-data/README.md +++ b/packages/microcosm-data/README.md @@ -72,7 +72,8 @@ Published releases live under `releases//` in the Hub dataset repo. Each release must include `build_manifest.json` and `release_manifest.json`; US releases must also include `us_source_coverage.json`. A current release manifest contains a typed `calibration_diagnostics` status. `available` binds a -schema-8 `calibration_diagnostics.json` file and its SHA-256. `failed` records +schema-9 `calibration_diagnostics.json` file (schema 8 in releases published +before it) and its SHA-256. `failed` records the expected schema, error code, and sanitized message and deliberately omits the file and artifact entry. Publication logs a warning but continues, because the diagnostics document reports a completed calibration and is not an input diff --git a/packages/microcosm-data/src/microcosm/data/contract.py b/packages/microcosm-data/src/microcosm/data/contract.py index 1ec393929..d63a35919 100644 --- a/packages/microcosm-data/src/microcosm/data/contract.py +++ b/packages/microcosm-data/src/microcosm/data/contract.py @@ -61,6 +61,7 @@ ) from microcosm.diagnostics import ( CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, UK_DIAGNOSTICS_SCHEMA_VERSION, UK_MODEL_FIGURE_FIELDS, @@ -226,6 +227,29 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: _SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS = ( SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS ) +#: Schema 8 (labels on every row; releases published before schema 9) and +#: schema 9 (labels in ``hierarchy_catalogs``; what the writer publishes). +#: Both carry the calibration hierarchy and are validated by the shared +#: models, so every check of a current document accepts either. +_HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS = ( + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS +) +_HIERARCHY_SCHEMA_VERSIONS_TEXT = " or ".join( + str(version) + for version in sorted(_HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS) +) + + +def _is_hierarchy_diagnostics_version(value: object) -> bool: + """Whether a declared diagnostics version is schema 8 or 9 (exact integers).""" + + return ( + isinstance(value, int) + and not isinstance(value, bool) + and value in _HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS + ) + + US_SOURCE_COVERAGE_DIAGNOSTICS_FILE = "us_source_coverage.json" SOURCE_COVERAGE_DIAGNOSTICS_SCHEMA_VERSION = 1 _SHA256_RE = re.compile(r"^[0-9a-f]{64}$") @@ -1583,10 +1607,10 @@ def _check_calibration_diagnostics_declaration( return status = declaration.get("status") if status == "available": - if declaration.get("schema_version") != CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: + if not _is_hierarchy_diagnostics_version(declaration.get("schema_version")): failures.append( "release_manifest.json available calibration diagnostics must " - f"declare schema_version {CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION}." + f"declare schema_version {_HIERARCHY_SCHEMA_VERSIONS_TEXT}." ) _check_sha256_field( filename="release_manifest.json", @@ -1612,12 +1636,13 @@ def _check_calibration_diagnostics_declaration( ) return if status == "failed": - if declaration.get("expected_schema_version") != ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + if not _is_hierarchy_diagnostics_version( + declaration.get("expected_schema_version") ): failures.append( "release_manifest.json failed calibration diagnostics must " - "declare the current expected_schema_version." + "declare the current expected_schema_version " + f"({_HIERARCHY_SCHEMA_VERSIONS_TEXT})." ) for field in ("error_code", "message"): if not isinstance(declaration.get(field), str) or not declaration[field]: @@ -3363,9 +3388,7 @@ def _check_calibration_diagnostics( """ schema_version = diagnostics.get("schema_version") - if not grandfathered_uk_june and schema_version == ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION - ): + if not grandfathered_uk_june and _is_hierarchy_diagnostics_version(schema_version): try: parse_calibration_diagnostics(diagnostics) except ValidationError as error: @@ -4465,10 +4488,10 @@ def _validate_local_area_release_dir(release_dir: Path, release_id: str) -> None ) else: _check_local_area_calibration_diagnostics(diagnostics, failures) - if ( - _is_uk_exact_k_release_id(release_id) - and diagnostics.get("schema_version") - != CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + if _is_uk_exact_k_release_id( + release_id + ) and not _is_hierarchy_diagnostics_version( + diagnostics.get("schema_version") ): _check_legacy_uk_calibration_diagnostics(diagnostics, failures) if _is_uk_exact_k_release_id(release_id): @@ -4595,9 +4618,7 @@ def _validate_uk_dense_release_dir(release_dir: Path, release_id: str) -> list[s if diagnostics_path.is_file(): diagnostics = _load_json(diagnostics_path, failures) if diagnostics is not None: - if diagnostics.get("schema_version") == ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION - ): + if _is_hierarchy_diagnostics_version(diagnostics.get("schema_version")): _check_calibration_diagnostics(diagnostics, failures) _check_local_area_calibration_diagnostics(diagnostics, failures) floor_warnings = _uk_brma_floor_verdict_warnings(diagnostics) @@ -6516,7 +6537,7 @@ def _check_uk_candidate_restated_losses( def _check_local_area_calibration_diagnostics( diagnostics: Mapping, failures: list[str] ) -> None: - if diagnostics.get("schema_version") == CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: + if _is_hierarchy_diagnostics_version(diagnostics.get("schema_version")): target_surface = diagnostics.get("target_surface") n_targets = ( target_surface.get("n_targets") @@ -6846,9 +6867,9 @@ def validate_release_dir( if ( _is_uk_exact_k_release_id(release_id) or release_id == _UK_NATIONAL_RELEASE_ID - ) and diagnostics.get( - "schema_version" - ) != CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: + ) and not _is_hierarchy_diagnostics_version( + diagnostics.get("schema_version") + ): _check_legacy_uk_calibration_diagnostics(diagnostics, failures) if _is_uk_exact_k_release_id(release_id): _check_uk_exact_k_diagnostics_identity( @@ -7967,8 +7988,8 @@ def _check_uk_dense_surface_files( ) original_diagnostics = loaded["source_calibration_diagnostics.json"] if shipped_diagnostics: - if original_diagnostics.get("schema_version") == ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + if _is_hierarchy_diagnostics_version( + original_diagnostics.get("schema_version") ): shipped_sha256 = _sha256(release_dir / "calibration_diagnostics.json") if shipped_sha256 != identity["candidate_diagnostics_sha256"]: diff --git a/packages/microcosm-data/tests/engine_free/shared/test_contract.py b/packages/microcosm-data/tests/engine_free/shared/test_contract.py index 42dfcdc7d..a33e0cd57 100644 --- a/packages/microcosm-data/tests/engine_free/shared/test_contract.py +++ b/packages/microcosm-data/tests/engine_free/shared/test_contract.py @@ -38,6 +38,10 @@ stored_input_evidence_manifest_block, ) from microcosm.data.stored_inputs import register_sha256 +from microcosm.diagnostics import ( + CalibrationDiagnosticsV8, + catalog_calibration_diagnostics, +) from test_support.paths import paths_for _TEST_PATHS = paths_for("microcosm-data") @@ -2569,7 +2573,7 @@ def test_legacy_diagnostics_exemption_is_scoped_to_the_exact_june_id( payload=diagnostics, ) - with pytest.raises(ReleaseContractError, match="publishes version 8"): + with pytest.raises(ReleaseContractError, match="publishes version 9"): validate_release_dir(directory) @@ -3887,6 +3891,95 @@ def test_schema_8_calibration_hierarchy_is_accepted(release_dir: Path) -> None: validate_release_dir(release_dir) +def _schema_9_calibration_diagnostics() -> dict: + """What the writer publishes for the schema-8 fixture: labels in catalogs.""" + + model = CalibrationDiagnosticsV8.model_validate(_schema_8_calibration_diagnostics()) + return json.loads( + catalog_calibration_diagnostics(model).model_dump_json(exclude_defaults=True) + ) + + +def test_schema_9_calibration_catalogs_are_accepted(release_dir: Path) -> None: + _write_json_and_refresh_manifest_hash( + release_dir, + filename="calibration_diagnostics.json", + artifact_key="calibration_diagnostics", + payload=_schema_9_calibration_diagnostics(), + ) + + validate_release_dir(release_dir) + + +def test_schema_9_always_uses_shared_validation(release_dir: Path) -> None: + diagnostics = _schema_9_calibration_diagnostics() + diagnostics["targets"][0]["hierarchy"]["geography_id"] = "0400000US06" + _write_json_and_refresh_manifest_hash( + release_dir, + filename="calibration_diagnostics.json", + artifact_key="calibration_diagnostics", + payload=diagnostics, + ) + + with pytest.raises(ReleaseContractError) as excinfo: + validate_release_dir(release_dir) + failures = "\n".join(excinfo.value.failures) + assert "does not satisfy the shared schema" in failures + assert "geography 0400000US06 is not in hierarchy_catalogs" in failures + + +@pytest.mark.parametrize("schema_version", [8, 9]) +def test_available_diagnostics_declare_either_hierarchy_schema( + release_dir: Path, schema_version: int +) -> None: + # A release published before schema 9 declares 8; the writer now + # publishes 9. + if schema_version == 9: + _write_json_and_refresh_manifest_hash( + release_dir, + filename="calibration_diagnostics.json", + artifact_key="calibration_diagnostics", + payload=_schema_9_calibration_diagnostics(), + ) + manifest_path = release_dir / "release_manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["calibration_diagnostics"] = { + "status": "available", + "schema_version": schema_version, + "sha256": manifest["artifacts"]["calibration_diagnostics"]["sha256"], + } + manifest_path.write_text(json.dumps(manifest, indent=1)) + + validate_release_dir(release_dir) + + manifest["calibration_diagnostics"]["schema_version"] = 7 + manifest_path.write_text(json.dumps(manifest, indent=1)) + with pytest.raises(ReleaseContractError) as excinfo: + validate_release_dir(release_dir) + assert ( + "release_manifest.json available calibration diagnostics must declare " + "schema_version 8 or 9." + ) in excinfo.value.failures + + +def test_failed_diagnostics_may_expect_the_published_schema_9( + release_dir: Path, +) -> None: + (release_dir / "calibration_diagnostics.json").unlink() + manifest_path = release_dir / "release_manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["artifacts"].pop("calibration_diagnostics") + manifest["calibration_diagnostics"] = { + "status": "failed", + "expected_schema_version": 9, + "error_code": "label_conflict", + "message": "Geography 0100000US has two labels.", + } + manifest_path.write_text(json.dumps(manifest, indent=1)) + + validate_release_dir(release_dir) + + def test_explicit_diagnostics_failure_does_not_block_national_release( release_dir: Path, ) -> None: diff --git a/packages/microcosm-diagnostics/README.md b/packages/microcosm-diagnostics/README.md index f7bfeec75..717462a34 100644 --- a/packages/microcosm-diagnostics/README.md +++ b/packages/microcosm-diagnostics/README.md @@ -1,14 +1,38 @@ # microcosm-diagnostics `microcosm-diagnostics` owns the versioned calibration-diagnostics models, -runtime validation, and the only atomic JSON writer for current schema-8 -documents. Solver adapters may assemble a typed model, but they do not write -the artifact directly. Release assemblers validate and copy current documents -without rewriting them; compatibility code may still adapt historical schema-6 -and schema-7 documents. This package deliberately does not depend on the -calibration solver or distribution packages. - -Schema 8 includes typed UK extension models for the shipped-weight summary, +runtime validation, and the only atomic JSON writer for current documents. +Solver adapters assemble a typed schema-8 model, but they do not write the +artifact directly: the writer publishes it as schema 9. Release assemblers +validate and copy current documents without rewriting them; compatibility code +may still adapt historical schema-6 and schema-7 documents. This package +deliberately does not depend on the calibration solver or distribution +packages. + +## Schemas 8 and 9 + +Producers assemble schema 8 (`CalibrationDiagnosticsV8`), in which every target +row carries its provider, category, geography and dimension labels. Schema 9 +(`CalibrationDiagnosticsV9`) is what the writer publishes: the same document +with each of those labels declared once, in a top-level `hierarchy_catalogs` +object, and referred to by ID from `targets[].hierarchy` (`category_id`, +`geography_id`, `dimensions[].id` and `value_id`; the target's own label stays +on the row). One identifier therefore cannot carry two labels in a published +file. Every other field is schema 8's. + +`catalog_calibration_diagnostics` builds the catalogs from the rows, listing +entries in the order the rows first use them. It refuses +(`HierarchyLabelConflictError`) a document whose rows give one identifier more +than one label, or one geography more than one level, unless a +`HierarchyLabelChoices` names the label: per geography ID, per dimension value, +or, for dimensions whose values are geography IDs (Chronicle's `geography`), +the label of the geography with that ID. A choice applies only where the rows +disagree. The writer reports a refusal as the failure code `label_conflict` +and writes nothing. `expand_calibration_diagnostics` rebuilds the schema-8 +document from a schema-9 one. Releases published before schema 9 keep their +schema-8 files, and `parse_calibration_diagnostics` reads both. + +Both schemas include typed UK extension models for the shipped-weight summary, zero-weight strata, geography-level pass rates, observation bases, family and area fit summaries, and measured or explicitly skipped rotated holdout runs. The models reject undeclared fields and reconcile counts, shares, ratios, fold @@ -17,33 +41,45 @@ target metadata, and build provenance remain explicitly extensible JSON mappings because their keys are producer-defined; every stable diagnostics record has a declared model. Historical schema-6 and schema-7 UK documents retain a read-only compatibility validator in `microcosm-data`; it is not used -for schema-8 production or validation. +for current production or validation. Every current model rejects `NaN`, positive infinity, and negative infinity at any nesting depth. Producers may convert a calculated value to `null` when `null` explicitly means that the measurement is unavailable, but values that -reach validation as non-finite numbers fail construction. The writer also uses -strict JSON serialization with non-finite output disabled, so a value introduced -after model construction returns a structured serialization failure instead of -silently becoming `null`. - -`calibration_diagnostics_v8_json_schema()` generates a Draft 2020-12 JSON -Schema for schema 8 from `CalibrationDiagnosticsV8`, for consumers that check -the file without importing these models. The models remain the authority. -Their cross-field reconciliations, finite-number and non-blank-text checks, -and strict integers cannot be written in a schema generated this way. The -schema's description lists each of these rules -(`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS`), and a test fails if a model +reach validation as non-finite numbers fail construction. The writer validates +the schema-9 document it builds, so a non-finite value introduced into a +schema-8 model after construction is refused there, and it serializes with +non-finite output disabled. + +## JSON Schemas + +`calibration_diagnostics_v9_json_schema()` and +`calibration_diagnostics_v8_json_schema()` generate Draft 2020-12 JSON Schemas +from `CalibrationDiagnosticsV9` and `CalibrationDiagnosticsV8`, for consumers +that check the file without importing these models. The models remain the +authority. Their cross-field reconciliations (including, for schema 9, that +every row refers to catalog entries that exist), finite-number and +non-blank-text checks, and strict integers cannot be written in a schema +generated this way. Each schema's description lists these rules +(`CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS`, +`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS`), and a test fails if a model validator is added without being listed. Complete validation still goes through `parse_calibration_diagnostics`. -The generated schema is committed, and shipped in the wheel, at +The generated schemas are committed, and shipped in the wheel, at + +`packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json` + +for the documents current builds publish, and at `packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v8.schema.json` -`load_calibration_diagnostics_v8_json_schema()` reads it. After changing a -schema-8 model, regenerate it with +for documents published before schema 9. +`load_calibration_diagnostics_v9_json_schema()` and +`load_calibration_diagnostics_v8_json_schema()` read them. After changing a +model, regenerate both with `uv run --no-sync python tools/generate_calibration_diagnostics_json_schema.py`; -`--check` reports a stale file without writing. A test fails while the -committed file differs from freshly generated output, and the US, UK and -transport build tests validate the diagnostics they write against it. +`--check` reports a stale file without writing, and `--schema-version` limits +the tool to one of them. A test fails while a committed file differs from +freshly generated output, and the US, UK and transport build tests validate the +diagnostics they write against the schema-9 file. diff --git a/packages/microcosm-diagnostics/pyproject.toml b/packages/microcosm-diagnostics/pyproject.toml index ff95ecc00..7c014b9fd 100644 --- a/packages/microcosm-diagnostics/pyproject.toml +++ b/packages/microcosm-diagnostics/pyproject.toml @@ -13,7 +13,7 @@ Homepage = "https://microcosm.institute" Repository = "https://github.com/PolicyEngine/microcosm" [dependency-groups] -# jsonschema checks documents against the generated schema-8 JSON Schema in +# jsonschema checks documents against the generated JSON Schemas in # tests only; the package itself needs nothing beyond pydantic. dev = ["pytest>=8", "jsonschema>=4.23,<5"] diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/__init__.py b/packages/microcosm-diagnostics/src/microcosm/diagnostics/__init__.py index 520a1a25e..01bcedfd3 100644 --- a/packages/microcosm-diagnostics/src/microcosm/diagnostics/__init__.py +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/__init__.py @@ -1,22 +1,40 @@ """Typed calibration diagnostics shared across the Microcosm stack.""" +from microcosm.diagnostics.catalogs import ( + HierarchyLabelChoices, + HierarchyLabelConflict, + HierarchyLabelConflictError, + catalog_calibration_diagnostics, + expand_calibration_diagnostics, +) from microcosm.diagnostics.json_schema import ( CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS, CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_ID, CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_LITERAL_NOTE, CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_RESOURCE, + CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS, + CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID, + CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_RESOURCE, JSON_SCHEMA_DRAFT_2020_12, JsonSchemaGap, assert_calibration_diagnostics_v8_json_schema_current, + assert_calibration_diagnostics_v9_json_schema_current, calibration_diagnostics_v8_json_schema, + calibration_diagnostics_v9_json_schema, committed_calibration_diagnostics_v8_json_schema_path, + committed_calibration_diagnostics_v9_json_schema_path, load_calibration_diagnostics_v8_json_schema, + load_calibration_diagnostics_v9_json_schema, render_calibration_diagnostics_v8_json_schema, + render_calibration_diagnostics_v9_json_schema, write_calibration_diagnostics_v8_json_schema, + write_calibration_diagnostics_v9_json_schema, ) from microcosm.diagnostics.schema import ( CALIBRATION_DIAGNOSTICS_ADAPTER, + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, UK_DIAGNOSTICS_SCHEMA_VERSION, UK_MODEL_FIGURE_FIELDS, @@ -26,6 +44,16 @@ UK_WITHHELD_PUBLISHED_OPTIONS, CalibrationDiagnostics, CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, + CatalogHierarchyReference, + HierarchyCatalogs, + HierarchyCategoryEntry, + HierarchyDimensionEntry, + HierarchyDimensionReference, + HierarchyDimensionValueEntry, + HierarchyGeographyEntry, + HierarchyProviderEntry, + TargetDiagnosticV9, UKAreaFit, UKConsumptionDrift, UKCountryFit, @@ -54,26 +82,38 @@ __all__ = [ "CALIBRATION_DIAGNOSTICS_ADAPTER", + "CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION", "CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION", "CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS", "CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_ID", "CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_LITERAL_NOTE", "CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_RESOURCE", - "JSON_SCHEMA_DRAFT_2020_12", - "SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS", + "CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS", + "CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID", + "CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_RESOURCE", "CalibrationDiagnostics", "CalibrationDiagnosticsV8", + "CalibrationDiagnosticsV9", + "CatalogHierarchyReference", "DiagnosticsFailureCode", "DiagnosticsWriteFailure", "DiagnosticsWriteOutcome", "DiagnosticsWriteSuccess", + "HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS", + "HierarchyCatalogs", + "HierarchyCategoryEntry", + "HierarchyDimensionEntry", + "HierarchyDimensionReference", + "HierarchyDimensionValueEntry", + "HierarchyGeographyEntry", + "HierarchyLabelChoices", + "HierarchyLabelConflict", + "HierarchyLabelConflictError", + "HierarchyProviderEntry", + "JSON_SCHEMA_DRAFT_2020_12", "JsonSchemaGap", - "UK_DIAGNOSTICS_SCHEMA_VERSION", - "UK_MODEL_FIGURE_FIELDS", - "UK_OPTIONAL_TARGET_GEOGRAPHY_LEVELS", - "UK_TARGET_GEOGRAPHY_LEVELS", - "UK_WITHHELD_PUBLISHED_ITERATE_FIELDS", - "UK_WITHHELD_PUBLISHED_OPTIONS", + "SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS", + "TargetDiagnosticV9", "UKAreaFit", "UKConsumptionDrift", "UKCountryFit", @@ -88,16 +128,30 @@ "UKWeightSummary", "UKWithheldModelFigures", "UKZeroWeightStratum", + "UK_DIAGNOSTICS_SCHEMA_VERSION", + "UK_MODEL_FIGURE_FIELDS", + "UK_OPTIONAL_TARGET_GEOGRAPHY_LEVELS", + "UK_TARGET_GEOGRAPHY_LEVELS", + "UK_WITHHELD_PUBLISHED_ITERATE_FIELDS", + "UK_WITHHELD_PUBLISHED_OPTIONS", "assert_calibration_diagnostics_v8_json_schema_current", + "assert_calibration_diagnostics_v9_json_schema_current", "calibration_diagnostics_json_schema", "calibration_diagnostics_v8_json_schema", + "calibration_diagnostics_v9_json_schema", + "catalog_calibration_diagnostics", "committed_calibration_diagnostics_v8_json_schema_path", + "committed_calibration_diagnostics_v9_json_schema_path", + "expand_calibration_diagnostics", "failed_diagnostics_outcome", "load_calibration_diagnostics_v8_json_schema", + "load_calibration_diagnostics_v9_json_schema", "parse_calibration_diagnostics", "render_calibration_diagnostics_v8_json_schema", + "render_calibration_diagnostics_v9_json_schema", "write_calibration_diagnostics", "write_calibration_diagnostics_v8_json_schema", + "write_calibration_diagnostics_v9_json_schema", ] __version__ = "0.1.0" diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/catalogs.py b/packages/microcosm-diagnostics/src/microcosm/diagnostics/catalogs.py new file mode 100644 index 000000000..fa2439060 --- /dev/null +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/catalogs.py @@ -0,0 +1,410 @@ +"""Hierarchy catalogs: schema 8 diagnostics to schema 9 and back. + +Producers assemble schema 8 (:class:`CalibrationDiagnosticsV8`), where every +target row repeats its provider, category, geography and dimension labels. +Microcosm checks that the facts inside one target agree, but nothing checks +across targets, so one geography or dimension value can arrive with two +labels from two publishers. Schema 9 (:class:`CalibrationDiagnosticsV9`) +declares each label once in ``hierarchy_catalogs`` and has the rows refer to +it by ID. + +:func:`catalog_calibration_diagnostics` builds the catalogs from the rows and +refuses (:class:`HierarchyLabelConflictError`) when the rows give one +identifier two labels, or one geography two levels, unless a +:class:`HierarchyLabelChoices` names the label to use. A choice applies only +where the rows disagree; an identifier the rows label one way keeps that +label. :func:`expand_calibration_diagnostics` rebuilds the schema 8 document. +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field +from types import MappingProxyType + +from microcosm.diagnostics.schema import ( + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, + CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, +) + +__all__ = [ + "HierarchyLabelChoices", + "HierarchyLabelConflict", + "HierarchyLabelConflictError", + "catalog_calibration_diagnostics", + "expand_calibration_diagnostics", +] + +_SHOWN_CONFLICTS = 10 +_LOGGER = logging.getLogger(__name__) + + +def _frozen_mapping(value: Mapping) -> Mapping: + return MappingProxyType(dict(value)) + + +@dataclass(frozen=True) +class HierarchyLabelChoices: + """Labels chosen for identifiers that a country's sources spell differently. + + ``geographies`` maps a geography ID to its label. ``dimension_values`` maps + a ``(dimension ID, value ID)`` pair to its label; when + ``dimension_value_spellings`` lists the spellings a value's choice was + reviewed for, the choice applies only while every spelling the rows use is + among them, so a new spelling is still refused (as Chronicle's + ``REVIEWED_VALUE_LABEL_VARIANTS`` does, chronicle#377). A dimension listed in + ``geography_dimensions`` has geography IDs as its values (Chronicle writes + them in lower case, for example ``e06000001``); when the rows label such a + value two ways, it takes the label of the geography with that ID, compared + without case: the payload's own geography label, else ``geographies``. + """ + + geographies: Mapping[str, str] = field(default_factory=dict) + dimension_values: Mapping[tuple[str, str], str] = field(default_factory=dict) + geography_dimensions: frozenset[str] = frozenset() + dimension_value_spellings: Mapping[tuple[str, str], frozenset[str]] = field( + default_factory=dict + ) + + def __post_init__(self) -> None: + object.__setattr__(self, "geographies", _frozen_mapping(self.geographies)) + object.__setattr__( + self, "dimension_values", _frozen_mapping(self.dimension_values) + ) + object.__setattr__( + self, "geography_dimensions", frozenset(self.geography_dimensions) + ) + object.__setattr__( + self, + "dimension_value_spellings", + _frozen_mapping( + { + key: frozenset(spellings) + for key, spellings in self.dimension_value_spellings.items() + } + ), + ) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, HierarchyLabelChoices): + return NotImplemented + return ( + dict(self.geographies) == dict(other.geographies) + and dict(self.dimension_values) == dict(other.dimension_values) + and self.geography_dimensions == other.geography_dimensions + and dict(self.dimension_value_spellings) + == dict(other.dimension_value_spellings) + ) + + __hash__ = None # type: ignore[assignment] + + def merged(self, other: HierarchyLabelChoices) -> HierarchyLabelChoices: + """These choices with ``other``'s added; ``other`` wins on a shared key.""" + + return HierarchyLabelChoices( + geographies={**self.geographies, **other.geographies}, + dimension_values={**self.dimension_values, **other.dimension_values}, + geography_dimensions=self.geography_dimensions | other.geography_dimensions, + dimension_value_spellings={ + **self.dimension_value_spellings, + **other.dimension_value_spellings, + }, + ) + + +@dataclass(frozen=True) +class HierarchyLabelConflict: + """One identifier the rows label (or place at a level) more than one way.""" + + #: What disagrees, for example ``geography 0400000US23`` or + #: ``dimension age value 85_plus``. + subject: str + #: Each distinct value, with the first target that carries it, in row order. + spellings: tuple[tuple[str, str], ...] + + def describe(self) -> str: + spellings = ", ".join( + f"{value!r} (first target {target})" for value, target in self.spellings + ) + return f"{self.subject}: {spellings}" + + +class HierarchyLabelConflictError(ValueError): + """The rows give identifiers more than one label, and no label is chosen.""" + + def __init__(self, conflicts: Iterable[HierarchyLabelConflict]) -> None: + self.conflicts = tuple(conflicts) + shown = "; ".join( + conflict.describe() for conflict in self.conflicts[:_SHOWN_CONFLICTS] + ) + more = len(self.conflicts) - _SHOWN_CONFLICTS + super().__init__( + "Calibration diagnostics schema " + f"{CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION} declares one label per " + f"identifier, but the target rows give {len(self.conflicts)} " + "identifier(s) more than one label or level, and no label is chosen " + f"for them: {shown}" + + (f"; and {more} more" if more > 0 else "") + + ". Make the sources agree, or choose the label in the country's " + "hierarchy label choices." + ) + + +class _Observed: + """Distinct values seen per key, each with the first target carrying it.""" + + def __init__(self) -> None: + self._values: dict[object, dict[str, str]] = {} + + def record(self, key: object, value: str, target: str) -> None: + self._values.setdefault(key, {}).setdefault(value, target) + + def keys(self) -> list[object]: + return list(self._values) + + def values(self, key: object) -> dict[str, str]: + return self._values[key] + + +def _single_or_conflict( + observed: _Observed, + key: object, + subject: str, + conflicts: list[HierarchyLabelConflict], + chosen: str | None = None, + resolved: list[tuple[HierarchyLabelConflict, str]] | None = None, +) -> str: + """The one value the rows give ``key``; else the chosen one; else a conflict. + + A disagreement a choice settles is appended to ``resolved``, so it is + logged rather than hidden. + """ + + values = observed.values(key) + if len(values) == 1: + return next(iter(values)) + if chosen is not None: + if resolved is not None: + resolved.append( + (HierarchyLabelConflict(subject, tuple(values.items())), chosen) + ) + return chosen + conflicts.append(HierarchyLabelConflict(subject, tuple(values.items()))) + # Any value keeps the catalog well formed; the conflict is raised below. + return next(iter(values)) + + +def catalog_calibration_diagnostics( + diagnostics: CalibrationDiagnosticsV8, + *, + label_choices: HierarchyLabelChoices | None = None, +) -> CalibrationDiagnosticsV9: + """Publish a schema 8 document as schema 9, one label per identifier. + + Catalog entries are listed in the order the rows first use them, and a + dimension's values in the order its rows first use them. Raises + :class:`HierarchyLabelConflictError` listing every identifier the rows + label (or, for a geography, place at a level) more than one way without + a chosen label. Provider, category and dimension labels, a category's + provider and a geography's level are never chosen. + """ + + choices = label_choices or HierarchyLabelChoices() + providers = _Observed() + categories = _Observed() + category_providers = _Observed() + geographies = _Observed() + geography_levels = _Observed() + dimensions = _Observed() + dimension_values = _Observed() + for row in diagnostics.targets: + hierarchy = row.hierarchy + providers.record(hierarchy.provider.id, hierarchy.provider.label, row.name) + categories.record(hierarchy.category.id, hierarchy.category.label, row.name) + category_providers.record( + hierarchy.category.id, hierarchy.category.provider_id, row.name + ) + geographies.record(hierarchy.geography.id, hierarchy.geography.label, row.name) + geography_levels.record( + hierarchy.geography.id, hierarchy.geography.level, row.name + ) + for dimension in hierarchy.dimensions: + dimensions.record(dimension.id, dimension.label, row.name) + dimension_values.record( + (dimension.id, dimension.value_id), dimension.value_label, row.name + ) + + conflicts: list[HierarchyLabelConflict] = [] + resolved: list[tuple[HierarchyLabelConflict, str]] = [] + provider_entries = [ + { + "id": provider_id, + "label": _single_or_conflict( + providers, provider_id, f"provider {provider_id}", conflicts + ), + } + for provider_id in providers.keys() + ] + category_entries = [ + { + "id": category_id, + "label": _single_or_conflict( + categories, category_id, f"category {category_id}", conflicts + ), + "provider_id": _single_or_conflict( + category_providers, + category_id, + f"category {category_id} provider", + conflicts, + ), + } + for category_id in categories.keys() + ] + geography_entries = [ + { + "id": geography_id, + "label": _single_or_conflict( + geographies, + geography_id, + f"geography {geography_id}", + conflicts, + chosen=choices.geographies.get(geography_id), + resolved=resolved, + ), + "level": _single_or_conflict( + geography_levels, + geography_id, + f"geography {geography_id} level", + conflicts, + ), + } + for geography_id in geographies.keys() + ] + area_labels = { + **{ + geography_id.casefold(): label + for geography_id, label in choices.geographies.items() + }, + **{entry["id"].casefold(): entry["label"] for entry in geography_entries}, + } + + def value_label(dimension_id: str, value_id: str) -> str: + key = (dimension_id, value_id) + subject = f"dimension {dimension_id} value {value_id}" + chosen = choices.dimension_values.get(key) + reviewed = choices.dimension_value_spellings.get(key) + if ( + chosen is not None + and reviewed is not None + and not set(dimension_values.values(key)) <= reviewed + ): + # A spelling nobody reviewed: refuse rather than relabel it. + chosen = None + subject += " (a spelling its reviewed list does not include)" + elif chosen is None and dimension_id in choices.geography_dimensions: + chosen = area_labels.get(value_id.casefold()) + return _single_or_conflict( + dimension_values, key, subject, conflicts, chosen=chosen, resolved=resolved + ) + + values_by_dimension: dict[str, list[dict[str, str]]] = {} + for dimension_id, value_id in dimension_values.keys(): + values_by_dimension.setdefault(dimension_id, []).append( + {"id": value_id, "label": value_label(dimension_id, value_id)} + ) + dimension_entries = [ + { + "id": dimension_id, + "label": _single_or_conflict( + dimensions, dimension_id, f"dimension {dimension_id}", conflicts + ), + "values": values_by_dimension[dimension_id], + } + for dimension_id in dimensions.keys() + ] + if conflicts: + raise HierarchyLabelConflictError(conflicts) + if resolved: + _LOGGER.warning( + "Calibration diagnostics: the target rows label %d identifier(s) more " + "than one way; the chosen label is written for each: %s", + len(resolved), + "; ".join( + f"{conflict.describe()} -> {label!r}" for conflict, label in resolved + ), + ) + + document = diagnostics.model_dump(mode="python") + for row in document["targets"]: + hierarchy = row["hierarchy"] + row["hierarchy"] = { + "category_id": hierarchy["category"]["id"], + "geography_id": hierarchy["geography"]["id"], + "dimensions": [ + {"id": dimension["id"], "value_id": dimension["value_id"]} + for dimension in hierarchy["dimensions"] + ], + "target": hierarchy["target"], + } + document["schema_version"] = CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION + document["hierarchy_catalogs"] = { + "providers": provider_entries, + "categories": category_entries, + "geographies": geography_entries, + "dimensions": dimension_entries, + } + return CalibrationDiagnosticsV9.model_validate(document) + + +def expand_calibration_diagnostics( + diagnostics: CalibrationDiagnosticsV9, +) -> CalibrationDiagnosticsV8: + """The schema 8 document a schema 9 document was published from.""" + + catalogs = diagnostics.hierarchy_catalogs + providers = {provider.id: provider for provider in catalogs.providers} + categories = {category.id: category for category in catalogs.categories} + geographies = {geography.id: geography for geography in catalogs.geographies} + dimensions = {dimension.id: dimension for dimension in catalogs.dimensions} + values = { + (dimension.id, value.id): value + for dimension in catalogs.dimensions + for value in dimension.values + } + document = diagnostics.model_dump(mode="python") + del document["hierarchy_catalogs"] + document["schema_version"] = CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION + for row in document["targets"]: + reference = row["hierarchy"] + category = categories[reference["category_id"]] + provider = providers[category.provider_id] + geography = geographies[reference["geography_id"]] + row["hierarchy"] = { + "provider": {"id": provider.id, "label": provider.label}, + "category": { + "id": category.id, + "label": category.label, + "provider_id": category.provider_id, + }, + "geography": { + "id": geography.id, + "label": geography.label, + "level": geography.level, + }, + "dimensions": [ + { + "id": dimension["id"], + "label": dimensions[dimension["id"]].label, + "value_id": dimension["value_id"], + "value_label": values[ + (dimension["id"], dimension["value_id"]) + ].label, + } + for dimension in reference["dimensions"] + ], + "target": reference["target"], + } + return CalibrationDiagnosticsV8.model_validate(document) diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/json_schema.py b/packages/microcosm-diagnostics/src/microcosm/diagnostics/json_schema.py index 78cdb76d6..8b3a11b3c 100644 --- a/packages/microcosm-diagnostics/src/microcosm/diagnostics/json_schema.py +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/json_schema.py @@ -1,10 +1,13 @@ -"""A Draft 2020-12 JSON Schema for calibration diagnostics schema 8. - -The schema is generated from :class:`CalibrationDiagnosticsV8`, so it cannot -drift from the models in field names, types, bounds, enumerations, required -fields or undeclared-field refusal. The models remain the authority: some of -their rules cannot be written in a schema generated this way, and the schema's -description lists each one (:data:`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS`). +"""Draft 2020-12 JSON Schemas for calibration diagnostics schemas 8 and 9. + +Each schema is generated from its model (:class:`CalibrationDiagnosticsV8`, +:class:`CalibrationDiagnosticsV9`), so it cannot drift from the models in field +names, types, bounds, enumerations, required fields or undeclared-field +refusal. The models remain the authority: some of their rules cannot be +written in a schema generated this way, and each schema's description lists +them (:data:`CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS`, +:data:`CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS`). Current builds publish +schema 9; the schema-8 file stays for documents published before it. """ from __future__ import annotations @@ -17,7 +20,10 @@ from pydantic import JsonValue -from microcosm.diagnostics.schema import CalibrationDiagnosticsV8 +from microcosm.diagnostics.schema import ( + CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, +) JSON_SCHEMA_DRAFT_2020_12: Final = "https://json-schema.org/draft/2020-12/schema" CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_ID: Final = ( @@ -345,3 +351,191 @@ def assert_calibration_diagnostics_v8_json_schema_current( f"{source} differs from the schema the models generate; " f"regenerate it with `{_REGENERATE_COMMAND}`." ) + + +# --------------------------------------------------------------------------- +# Schema 9: the schema current builds publish +# --------------------------------------------------------------------------- + +CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID: Final = ( + "calibration_diagnostics.v9.schema.json" +) +CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_RESOURCE: Final = ( + f"schemas/{CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID}" +) + +_V9_CATALOG_GAPS: Final[tuple[JsonSchemaGap, ...]] = ( + JsonSchemaGap( + key="hierarchy-catalog-identity", + rule=( + "Each hierarchy_catalogs array lists an ID once, each " + "hierarchy_catalogs.dimensions entry lists a value ID once, and " + "every category's provider_id is listed in " + "hierarchy_catalogs.providers." + ), + validators=( + "HierarchyCatalogs.identifiers_are_unique_and_providers_listed", + "HierarchyDimensionEntry.value_ids_are_unique", + ), + ), + JsonSchemaGap( + key="hierarchy-references", + rule=( + "Each targets[].hierarchy refers to a category_id, a geography_id " + "and dimension ids that hierarchy_catalogs lists, each value_id is " + "listed under its own dimension, and a row lists each dimension " + "once." + ), + validators=( + "CalibrationDiagnosticsV9.rows_refer_to_catalog_entries", + "CatalogHierarchyReference.dimension_ids_are_unique", + ), + ), +) + + +def _v9_gap(gap: JsonSchemaGap) -> JsonSchemaGap: + return JsonSchemaGap( + key=gap.key, + rule=gap.rule, + validators=tuple( + validator.replace( + "CalibrationDiagnosticsV8.", "CalibrationDiagnosticsV9." + ).replace("TargetDiagnosticV8.", "TargetDiagnosticV9.") + for validator in gap.validators + ), + ) + + +def _v9_gaps() -> tuple[JsonSchemaGap, ...]: + gaps: list[JsonSchemaGap] = [] + for gap in CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS: + if gap.key == "hierarchy-identity": + # Schema 9 rows hold IDs; the catalogs hold the labels. + gaps.extend(_V9_CATALOG_GAPS) + else: + gaps.append(_v9_gap(gap)) + return tuple(gaps) + + +#: The rules the schema-9 models enforce that its generated schema leaves out: +#: schema 8's, with the row-hierarchy rule replaced by the catalog rules. +CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS: Final[tuple[JsonSchemaGap, ...]] = ( + _v9_gaps() +) + +_V9_TITLE: Final = "Microcosm calibration diagnostics, schema 9" +_V9_COMMENT: Final = ( + "Generated from microcosm.diagnostics.CalibrationDiagnosticsV9. Do not " + f"edit it; regenerate it with `{_REGENERATE_COMMAND}`." +) + + +def _v9_description() -> str: + gap_lines = "\n".join( + f"- {gap.rule}" for gap in CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS + ) + return ( + "The calibration_diagnostics.json document, schema 9, that Microcosm " + "builds write beside a calibrated dataset. It is schema 8 with each " + "provider, category, geography and dimension label declared once, in " + "hierarchy_catalogs, and referred to by ID from targets[].hierarchy, so " + "one identifier cannot carry two labels. The pydantic models it is " + "generated from remain the authority. A document that satisfies this " + "schema can still break one of the rules below, which the models " + "enforce and this schema does not express, so validate with " + "microcosm.diagnostics.parse_calibration_diagnostics for the complete " + "contract. Equalities between floating-point values hold within the " + "models' tolerances.\n\n" + "Rules the models enforce that this schema does not express:\n" + f"{gap_lines}\n\n" + "Where this schema is stricter than the models' parser:\n" + f"- {CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_LITERAL_NOTE}" + ) + + +def calibration_diagnostics_v9_json_schema() -> dict[str, JsonValue]: + """Return the schema-9 JSON Schema generated from the typed models.""" + + generated = CalibrationDiagnosticsV9.model_json_schema(mode="validation") + generated.pop("title", None) + generated.pop("description", None) + return { + "$schema": JSON_SCHEMA_DRAFT_2020_12, + "$id": CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID, + "$comment": _V9_COMMENT, + "title": _V9_TITLE, + "description": _v9_description(), + **generated, + } + + +def render_calibration_diagnostics_v9_json_schema() -> str: + """Return the schema-9 schema as the exact text committed to the repository.""" + + return ( + json.dumps( + calibration_diagnostics_v9_json_schema(), + indent=2, + ensure_ascii=False, + ) + + "\n" + ) + + +def committed_calibration_diagnostics_v9_json_schema_path() -> Path: + """Return the committed schema-9 file, shipped inside the package.""" + + return Path( + str( + files("microcosm.diagnostics").joinpath( + CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_RESOURCE + ) + ) + ) + + +def load_calibration_diagnostics_v9_json_schema( + path: str | Path | None = None, +) -> dict[str, JsonValue]: + """Read the committed schema-9 schema, or the schema file at ``path``.""" + + source = ( + committed_calibration_diagnostics_v9_json_schema_path() + if path is None + else Path(path) + ) + return json.loads(source.read_bytes()) + + +def write_calibration_diagnostics_v9_json_schema( + path: str | Path | None = None, +) -> Path: + """Write freshly generated schema-9 text to the committed file or ``path``.""" + + output = ( + committed_calibration_diagnostics_v9_json_schema_path() + if path is None + else Path(path) + ) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_bytes(render_calibration_diagnostics_v9_json_schema().encode("utf-8")) + return output + + +def assert_calibration_diagnostics_v9_json_schema_current( + path: str | Path | None = None, +) -> None: + """Raise unless the file is byte-identical to freshly generated schema-9 text.""" + + source = ( + committed_calibration_diagnostics_v9_json_schema_path() + if path is None + else Path(path) + ) + expected = render_calibration_diagnostics_v9_json_schema().encode("utf-8") + if source.read_bytes() != expected: + raise ValueError( + f"{source} differs from the schema the models generate; " + f"regenerate it with `{_REGENERATE_COMMAND}`." + ) diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/schema.py b/packages/microcosm-diagnostics/src/microcosm/diagnostics/schema.py index d504047ff..e5efa8e24 100644 --- a/packages/microcosm-diagnostics/src/microcosm/diagnostics/schema.py +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/schema.py @@ -17,8 +17,16 @@ model_validator, ) -CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: Final[Literal[8]] = 8 -SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS = frozenset({6, 7, 8}) +#: The schema current builds publish: 9, with hierarchy labels in catalogs. +CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: Final[Literal[9]] = 9 +#: The schema producers assemble before writing: 8, with every hierarchy +#: label on its row. The writer turns it into schema 9 +#: (``catalog_calibration_diagnostics``). +CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION: Final[Literal[8]] = 8 +SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS = frozenset({6, 7, 8, 9}) +#: The versions whose rows carry the calibration hierarchy: on each row in +#: schema 8, by reference to ``hierarchy_catalogs`` in schema 9. +HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS = frozenset({8, 9}) UK_DIAGNOSTICS_SCHEMA_VERSION: Final[Literal[1]] = 1 @@ -701,8 +709,79 @@ def reconcile_uk_diagnostics(self) -> UKDiagnosticsV1: return self +def _reconcile_diagnostics_counts( + diagnostics: CalibrationDiagnosticsV8 | CalibrationDiagnosticsV9, +) -> None: + """Counts that schema 8 and schema 9 documents must both reconcile.""" + + if diagnostics.target_surface.n_targets != len(diagnostics.targets): + raise ValueError("target_surface.n_targets must equal len(targets)") + if diagnostics.target_registry.n_specs < len(diagnostics.targets): + raise ValueError("target_registry.n_specs cannot be less than len(targets)") + if diagnostics.target_surface.n_records != diagnostics.n_records: + raise ValueError("target_surface.n_records must equal n_records") + if diagnostics.n_nonzero > diagnostics.n_records: + raise ValueError("n_nonzero cannot exceed n_records") + if diagnostics.target_registry.country == "uk": + if diagnostics.uk_diagnostics is None: + raise ValueError( + "UK calibration diagnostics require a uk_diagnostics block" + ) + uk = diagnostics.uk_diagnostics + if uk.weights.n_records != diagnostics.n_records: + raise ValueError("UK weights.n_records must match top-level n_records") + if diagnostics.effective_sample_size is None or not math.isclose( + uk.weights.effective_sample_size, + diagnostics.effective_sample_size, + rel_tol=1e-12, + abs_tol=1e-12, + ): + raise ValueError( + "UK weights.effective_sample_size must match the top-level value" + ) + if diagnostics.top_1pct_weight_share is None or not math.isclose( + uk.weights.top_1pct_weight_share, + diagnostics.top_1pct_weight_share, + rel_tol=1e-12, + abs_tol=1e-12, + ): + raise ValueError( + "UK weights.top_1pct_weight_share must match the top-level value" + ) + total_targets = sum( + row.n_targets for row in uk.target_pass_rates_by_geography_level + ) + total_scored = sum( + row.n_scored for row in uk.target_pass_rates_by_geography_level + ) + total_skipped = sum( + row.n_skipped for row in uk.target_pass_rates_by_geography_level + ) + if total_targets != diagnostics.target_registry.n_specs: + raise ValueError( + "UK geography target counts must match target_registry.n_specs" + ) + if total_scored != len(diagnostics.targets): + raise ValueError( + "UK geography scored counts must match the target row count" + ) + if total_skipped != len(diagnostics.skipped): + raise ValueError( + "UK geography skipped counts must match the skipped row count" + ) + elif diagnostics.uk_diagnostics is not None: + raise ValueError( + "uk_diagnostics is only valid when target_registry.country is 'uk'" + ) + + class CalibrationDiagnosticsV8(DiagnosticsModel): - """The only calibration-diagnostics schema emitted by current builds.""" + """Schema 8: every target row carries its complete hierarchy, labels included. + + Producers assemble this document; the writer publishes it as schema 9 + (``CalibrationDiagnosticsV9``), and releases published before schema 9 + carry it as written. + """ schema_version: Literal[8] weight_entity: NonBlankText @@ -744,65 +823,229 @@ def finite_optional_scalar(cls, value: float | None) -> float | None: @model_validator(mode="after") def reconcile_counts(self) -> CalibrationDiagnosticsV8: - if self.target_surface.n_targets != len(self.targets): - raise ValueError("target_surface.n_targets must equal len(targets)") - if self.target_registry.n_specs < len(self.targets): - raise ValueError("target_registry.n_specs cannot be less than len(targets)") - if self.target_surface.n_records != self.n_records: - raise ValueError("target_surface.n_records must equal n_records") - if self.n_nonzero > self.n_records: - raise ValueError("n_nonzero cannot exceed n_records") - if self.target_registry.country == "uk": - if self.uk_diagnostics is None: - raise ValueError( - "UK calibration diagnostics require a uk_diagnostics block" - ) - uk = self.uk_diagnostics - if uk.weights.n_records != self.n_records: - raise ValueError("UK weights.n_records must match top-level n_records") - if self.effective_sample_size is None or not math.isclose( - uk.weights.effective_sample_size, - self.effective_sample_size, - rel_tol=1e-12, - abs_tol=1e-12, - ): - raise ValueError( - "UK weights.effective_sample_size must match the top-level value" - ) - if self.top_1pct_weight_share is None or not math.isclose( - uk.weights.top_1pct_weight_share, - self.top_1pct_weight_share, - rel_tol=1e-12, - abs_tol=1e-12, - ): + _reconcile_diagnostics_counts(self) + return self + + +class HierarchyProviderEntry(DiagnosticsModel): + id: NonBlankText + label: NonBlankText + + +class HierarchyCategoryEntry(DiagnosticsModel): + id: NonBlankText + label: NonBlankText + provider_id: NonBlankText + + +class HierarchyGeographyEntry(DiagnosticsModel): + id: NonBlankText + label: NonBlankText + level: NonBlankText + + +class HierarchyDimensionValueEntry(DiagnosticsModel): + id: NonBlankText + label: NonBlankText + + +class HierarchyDimensionEntry(DiagnosticsModel): + id: NonBlankText + label: NonBlankText + values: list[HierarchyDimensionValueEntry] + + @model_validator(mode="after") + def value_ids_are_unique(self) -> HierarchyDimensionEntry: + seen: set[str] = set() + for value in self.values: + if value.id in seen: raise ValueError( - "UK weights.top_1pct_weight_share must match the top-level value" + f"hierarchy_catalogs dimension {self.id} lists value " + f"{value.id} twice" ) - total_targets = sum( - row.n_targets for row in uk.target_pass_rates_by_geography_level - ) - total_scored = sum( - row.n_scored for row in uk.target_pass_rates_by_geography_level - ) - total_skipped = sum( - row.n_skipped for row in uk.target_pass_rates_by_geography_level - ) - if total_targets != self.target_registry.n_specs: + seen.add(value.id) + return self + + +class HierarchyCatalogs(DiagnosticsModel): + """Each hierarchy label, declared once (schema 9). + + A category names its provider; a dimension lists its values. Entries are + arrays, so a repeated ID stays visible to any reader. + """ + + providers: list[HierarchyProviderEntry] + categories: list[HierarchyCategoryEntry] + geographies: list[HierarchyGeographyEntry] + dimensions: list[HierarchyDimensionEntry] + + @model_validator(mode="after") + def identifiers_are_unique_and_providers_listed(self) -> HierarchyCatalogs: + for name in ("providers", "categories", "geographies", "dimensions"): + seen: set[str] = set() + for entry in getattr(self, name): + if entry.id in seen: + raise ValueError( + f"hierarchy_catalogs.{name} lists {entry.id} twice" + ) + seen.add(entry.id) + providers = {provider.id for provider in self.providers} + for category in self.categories: + if category.provider_id not in providers: raise ValueError( - "UK geography target counts must match target_registry.n_specs" + f"category {category.id} refers to provider " + f"{category.provider_id}, which hierarchy_catalogs.providers " + "does not list" ) - if total_scored != len(self.targets): + return self + + +class HierarchyDimensionReference(DiagnosticsModel): + id: NonBlankText + value_id: NonBlankText + + +class CatalogHierarchyReference(DiagnosticsModel): + """A schema 9 row's hierarchy: catalog IDs, and the target's own label. + + The provider is the category's. ``dimensions`` keep producer order; each + ``value_id`` is listed under its own dimension in the catalog. + """ + + category_id: NonBlankText + geography_id: NonBlankText + dimensions: list[HierarchyDimensionReference] + target: HierarchyNode + + @model_validator(mode="after") + def dimension_ids_are_unique(self) -> CatalogHierarchyReference: + dimension_ids = [dimension.id for dimension in self.dimensions] + if len(dimension_ids) != len(set(dimension_ids)): + raise ValueError("hierarchy dimension ids must be unique") + return self + + +class TargetDiagnosticV9(DiagnosticsModel): + """A schema 8 target row whose hierarchy refers to the catalogs. + + The fields are schema 8's, in the same order; only ``hierarchy`` differs. + """ + + name: NonBlankText + target_name: NonBlankText + period: int | NonBlankText + entity: NonBlankText + measure: Selector + filter: Selector | None + source: NonBlankText + metadata: dict[str, JsonValue] + target: float | None + compiled_target: float | None + initial_estimate: float | None + final_estimate: float | None + relative_error: float | None + within_tolerance: bool | None + hierarchy: CatalogHierarchyReference + registry: RegistryTargetMetadata | None = None + target_loss_weight: float | None = None + target_loss_weight_share: float | None = None + target_loss_scale: float | None = None + final_capped_scaled_error: float | None = None + final_loss_contribution: float | None = None + + @model_validator(mode="after") + def hierarchy_target_matches_row(self) -> TargetDiagnosticV9: + if self.hierarchy.target.id != self.target_name: + raise ValueError("hierarchy target id must match target_name") + return self + + +class CalibrationDiagnosticsV9(DiagnosticsModel): + """The calibration-diagnostics schema current builds publish. + + Schema 8 with each provider, category, geography and dimension label + declared once in ``hierarchy_catalogs`` and referred to by ID from the + rows, so one identifier cannot carry two labels. Every other field is + schema 8's. + """ + + schema_version: Literal[9] + weight_entity: NonBlankText + options: dict[str, JsonValue] + target_surface: TargetSurface + target_registry: TargetRegistryRef + l0_lambda: float | None + n_nonzero: int = Field(ge=0) + n_records: int = Field(gt=0) + initial_loss: float | None + final_loss: float | None + fraction_within_10pct: float | None + effective_sample_size: float | None + realized_max_weight_ratio: float | None + top_1pct_weight_share: float | None + loss_trajectory: list[float | None] + skipped: list[SkippedTarget] + past_cap_census: PastCapCensus | None + diagnostic_warnings: list[DiagnosticsWarning] + hierarchy_catalogs: HierarchyCatalogs + targets: list[TargetDiagnosticV9] + target_loss_basis: TargetLossBasis | None = None + build: dict[str, JsonValue] | None = None + uk_diagnostics: UKDiagnosticsV1 | None = None + + @field_validator( + "l0_lambda", + "initial_loss", + "final_loss", + "fraction_within_10pct", + "effective_sample_size", + "realized_max_weight_ratio", + "top_1pct_weight_share", + ) + @classmethod + def finite_optional_scalar(cls, value: float | None) -> float | None: + if value is not None and not math.isfinite(value): + raise ValueError("diagnostic scalars must be finite or null") + return value + + @model_validator(mode="after") + def reconcile_counts(self) -> CalibrationDiagnosticsV9: + _reconcile_diagnostics_counts(self) + return self + + @model_validator(mode="after") + def rows_refer_to_catalog_entries(self) -> CalibrationDiagnosticsV9: + catalogs = self.hierarchy_catalogs + categories = {category.id for category in catalogs.categories} + geographies = {geography.id for geography in catalogs.geographies} + dimension_values = { + dimension.id: {value.id for value in dimension.values} + for dimension in catalogs.dimensions + } + for row in self.targets: + reference = row.hierarchy + if reference.category_id not in categories: raise ValueError( - "UK geography scored counts must match the target row count" + f"target {row.name}: category {reference.category_id} is not " + "in hierarchy_catalogs" ) - if total_skipped != len(self.skipped): + if reference.geography_id not in geographies: raise ValueError( - "UK geography skipped counts must match the skipped row count" + f"target {row.name}: geography {reference.geography_id} is not " + "in hierarchy_catalogs" ) - elif self.uk_diagnostics is not None: - raise ValueError( - "uk_diagnostics is only valid when target_registry.country is 'uk'" - ) + for dimension in reference.dimensions: + values = dimension_values.get(dimension.id) + if values is None: + raise ValueError( + f"target {row.name}: dimension {dimension.id} is not in " + "hierarchy_catalogs" + ) + if dimension.value_id not in values: + raise ValueError( + f"target {row.name}: value {dimension.value_id} is not " + f"listed under dimension {dimension.id}" + ) return self @@ -822,7 +1065,7 @@ class LegacyCalibrationDiagnostics(BaseModel): CalibrationDiagnostics = Annotated[ - CalibrationDiagnosticsV8 | LegacyCalibrationDiagnostics, + CalibrationDiagnosticsV9 | CalibrationDiagnosticsV8 | LegacyCalibrationDiagnostics, Field(discriminator="schema_version"), ] CALIBRATION_DIAGNOSTICS_ADAPTER = TypeAdapter(CalibrationDiagnostics) diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json b/packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json new file mode 100644 index 000000000..d5cff94cf --- /dev/null +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/calibration_diagnostics.v9.schema.json @@ -0,0 +1,1891 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "calibration_diagnostics.v9.schema.json", + "$comment": "Generated from microcosm.diagnostics.CalibrationDiagnosticsV9. Do not edit it; regenerate it with `uv run --no-sync python tools/generate_calibration_diagnostics_json_schema.py`.", + "title": "Microcosm calibration diagnostics, schema 9", + "description": "The calibration_diagnostics.json document, schema 9, that Microcosm builds write beside a calibrated dataset. It is schema 8 with each provider, category, geography and dimension label declared once, in hierarchy_catalogs, and referred to by ID from targets[].hierarchy, so one identifier cannot carry two labels. The pydantic models it is generated from remain the authority. A document that satisfies this schema can still break one of the rules below, which the models enforce and this schema does not express, so validate with microcosm.diagnostics.parse_calibration_diagnostics for the complete contract. Equalities between floating-point values hold within the models' tolerances.\n\nRules the models enforce that this schema does not express:\n- Every number is finite, at any depth, including inside options, targets[].metadata and build. JSON has no NaN or Infinity, but lenient parsers such as Python's json module accept those tokens, and a JSON Schema number cannot exclude the values they produce, so parse submissions with a parser that refuses them.\n- Integer fields hold integers written without a fraction or exponent. JSON Schema counts 2.0 and 1e3 as integers; the strict models refuse them, because Python's json module decodes them as floats.\n- Strings the models type as non-blank contain at least one character that is not whitespace. These are most names, identifiers, labels, sources, entities, reasons, codes and messages, and the keys of uk_diagnostics.target_observation_basis, the zero-weight strata and the consumption-drift maps.\n- Digests match ^[0-9a-f]{64}$ with nothing after them. In the ECMA-262 regular expressions JSON Schema specifies, $ matches only at the end of the string, but validators that evaluate pattern with Python's re module also let $ match before a final newline, so they accept a digest followed by a newline.\n- Each hierarchy_catalogs array lists an ID once, each hierarchy_catalogs.dimensions entry lists a value ID once, and every category's provider_id is listed in hierarchy_catalogs.providers.\n- Each targets[].hierarchy refers to a category_id, a geography_id and dimension ids that hierarchy_catalogs lists, each value_id is listed under its own dimension, and a row lists each dimension once.\n- Each targets[].hierarchy.target.id equals that row's target_name.\n- target_surface.constraint_matrix has n_targets rows and n_records columns, and its nnz is at most rows times columns.\n- target_surface.n_targets equals the number of targets, target_registry.n_specs is at least the number of targets, target_surface.n_records equals n_records, and n_nonzero is at most n_records.\n- A UK registry (target_registry.country \"uk\") requires uk_diagnostics; any other registry requires it to be null or absent. In a UK document, uk_diagnostics.weights.n_records, effective_sample_size and top_1pct_weight_share equal the top-level n_records, effective_sample_size and top_1pct_weight_share. The n_targets, n_scored and n_skipped values in uk_diagnostics.target_pass_rates_by_geography_level sum to target_registry.n_specs, the number of targets and the number of skipped targets.\n- In uk_diagnostics.weights, positive_weight_records plus zero_weight_records equals n_records, effective_sample_size is at most n_records, and ess_fraction equals effective_sample_size divided by n_records. median_positive_weight and max_to_median_positive_weight are null exactly when positive_weight_records is 0; otherwise max_to_median_positive_weight equals max_weight divided by median_positive_weight.\n- In each uk_diagnostics.zero_weight_rows_by_stratum row, positive_weight_rows plus zero_weight_rows equals rows. Across the strata, rows, positive_weight_rows and zero_weight_rows sum to weights.n_records, weights.positive_weight_records and weights.zero_weight_records.\n- uk_diagnostics.target_pass_rates_by_geography_level has exactly one row for each of national, region, country, local_authority and constituency. In each row, n_scored plus n_skipped equals n_targets, n_within_10pct is at most n_scored, and pass_rate equals n_within_10pct divided by n_targets, or is null when n_targets is 0.\n- uk_diagnostics.weakest_families and weakest_areas_by_fit are both present or both absent. In weakest_families, weakest_areas_by_fit.bottom_by_fit and weakest_areas_by_fit.countries, n_within_10pct is at most n_targets and pass_rate equals n_within_10pct divided by n_targets. bottom_by_fit has at most limit rows and at most n_areas_scored rows.\n- A measured uk_diagnostics.rotated_holdout has unique grains and n_folds entries in both fold_losses and folds. Its fold ids cover 0 to n_folds - 1, each fold's holdout_loss equals its fold_losses entry, mean_holdout_loss is the mean of fold_losses and worst_holdout_loss is their maximum. Each fold has n_holdout_targets unique holdout_target_indices.\n- Each uk_diagnostics.consumption_drift value has change equal to final minus design, and its relative_change is null exactly when design is 0.\n\nWhere this schema is stricter than the models' parser:\n- Literal fields hold exactly their declared JSON value. The models compare literals with Python equality, so their parser also accepts 1 for true and true for 1, for example in rotated_holdout.skipped or uk_diagnostics.schema_version. The canonical writer always emits the declared value, which this schema requires.", + "$defs": { + "CatalogHierarchyReference": { + "additionalProperties": false, + "description": "A schema 9 row's hierarchy: catalog IDs, and the target's own label.\n\nThe provider is the category's. ``dimensions`` keep producer order; each\n``value_id`` is listed under its own dimension in the catalog.", + "properties": { + "category_id": { + "title": "Category Id", + "type": "string" + }, + "geography_id": { + "title": "Geography Id", + "type": "string" + }, + "dimensions": { + "items": { + "$ref": "#/$defs/HierarchyDimensionReference" + }, + "title": "Dimensions", + "type": "array" + }, + "target": { + "$ref": "#/$defs/HierarchyNode" + } + }, + "required": [ + "category_id", + "geography_id", + "dimensions", + "target" + ], + "title": "CatalogHierarchyReference", + "type": "object" + }, + "DiagnosticsWarning": { + "additionalProperties": false, + "properties": { + "code": { + "title": "Code", + "type": "string" + }, + "severity": { + "const": "warning", + "title": "Severity", + "type": "string" + }, + "message": { + "title": "Message", + "type": "string" + } + }, + "required": [ + "code", + "severity", + "message" + ], + "title": "DiagnosticsWarning", + "type": "object" + }, + "HierarchyCatalogs": { + "additionalProperties": false, + "description": "Each hierarchy label, declared once (schema 9).\n\nA category names its provider; a dimension lists its values. Entries are\narrays, so a repeated ID stays visible to any reader.", + "properties": { + "providers": { + "items": { + "$ref": "#/$defs/HierarchyProviderEntry" + }, + "title": "Providers", + "type": "array" + }, + "categories": { + "items": { + "$ref": "#/$defs/HierarchyCategoryEntry" + }, + "title": "Categories", + "type": "array" + }, + "geographies": { + "items": { + "$ref": "#/$defs/HierarchyGeographyEntry" + }, + "title": "Geographies", + "type": "array" + }, + "dimensions": { + "items": { + "$ref": "#/$defs/HierarchyDimensionEntry" + }, + "title": "Dimensions", + "type": "array" + } + }, + "required": [ + "providers", + "categories", + "geographies", + "dimensions" + ], + "title": "HierarchyCatalogs", + "type": "object" + }, + "HierarchyCategoryEntry": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + }, + "provider_id": { + "title": "Provider Id", + "type": "string" + } + }, + "required": [ + "id", + "label", + "provider_id" + ], + "title": "HierarchyCategoryEntry", + "type": "object" + }, + "HierarchyDimensionEntry": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + }, + "values": { + "items": { + "$ref": "#/$defs/HierarchyDimensionValueEntry" + }, + "title": "Values", + "type": "array" + } + }, + "required": [ + "id", + "label", + "values" + ], + "title": "HierarchyDimensionEntry", + "type": "object" + }, + "HierarchyDimensionReference": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "value_id": { + "title": "Value Id", + "type": "string" + } + }, + "required": [ + "id", + "value_id" + ], + "title": "HierarchyDimensionReference", + "type": "object" + }, + "HierarchyDimensionValueEntry": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + } + }, + "required": [ + "id", + "label" + ], + "title": "HierarchyDimensionValueEntry", + "type": "object" + }, + "HierarchyGeographyEntry": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + }, + "level": { + "title": "Level", + "type": "string" + } + }, + "required": [ + "id", + "label", + "level" + ], + "title": "HierarchyGeographyEntry", + "type": "object" + }, + "HierarchyNode": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + } + }, + "required": [ + "id", + "label" + ], + "title": "HierarchyNode", + "type": "object" + }, + "HierarchyProviderEntry": { + "additionalProperties": false, + "properties": { + "id": { + "title": "Id", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + } + }, + "required": [ + "id", + "label" + ], + "title": "HierarchyProviderEntry", + "type": "object" + }, + "JsonValue": {}, + "PastCapCensus": { + "additionalProperties": false, + "properties": { + "cap": { + "exclusiveMinimum": 0, + "title": "Cap", + "type": "number" + }, + "scale_basis": { + "title": "Scale Basis", + "type": "string" + }, + "n_targets": { + "minimum": 0, + "title": "N Targets", + "type": "integer" + }, + "initial_past_cap": { + "minimum": 0, + "title": "Initial Past Cap", + "type": "integer" + }, + "final_past_cap": { + "minimum": 0, + "title": "Final Past Cap", + "type": "integer" + }, + "escaped": { + "minimum": 0, + "title": "Escaped", + "type": "integer" + }, + "frozen": { + "minimum": 0, + "title": "Frozen", + "type": "integer" + }, + "pushed_out": { + "minimum": 0, + "title": "Pushed Out", + "type": "integer" + }, + "pushed_out_rows": { + "items": { + "$ref": "#/$defs/PushedOutTarget" + }, + "title": "Pushed Out Rows", + "type": "array" + } + }, + "required": [ + "cap", + "scale_basis", + "n_targets", + "initial_past_cap", + "final_past_cap", + "escaped", + "frozen", + "pushed_out", + "pushed_out_rows" + ], + "title": "PastCapCensus", + "type": "object" + }, + "PushedOutTarget": { + "additionalProperties": false, + "properties": { + "name": { + "title": "Name", + "type": "string" + }, + "init_rel": { + "title": "Init Rel", + "type": "number" + }, + "final_rel": { + "title": "Final Rel", + "type": "number" + } + }, + "required": [ + "name", + "init_rel", + "final_rel" + ], + "title": "PushedOutTarget", + "type": "object" + }, + "RegistryTargetMetadata": { + "additionalProperties": false, + "properties": { + "family": { + "title": "Family", + "type": "string" + }, + "se": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Se" + }, + "signed": { + "title": "Signed", + "type": "boolean" + }, + "notes": { + "title": "Notes", + "type": "string" + } + }, + "required": [ + "family", + "se", + "signed", + "notes" + ], + "title": "RegistryTargetMetadata", + "type": "object" + }, + "Selector": { + "additionalProperties": false, + "properties": { + "kind": { + "enum": [ + "column", + "callable" + ], + "title": "Kind", + "type": "string" + }, + "name": { + "title": "Name", + "type": "string" + } + }, + "required": [ + "kind", + "name" + ], + "title": "Selector", + "type": "object" + }, + "SkippedTarget": { + "additionalProperties": false, + "properties": { + "name": { + "title": "Name", + "type": "string" + }, + "reason": { + "title": "Reason", + "type": "string" + } + }, + "required": [ + "name", + "reason" + ], + "title": "SkippedTarget", + "type": "object" + }, + "TargetDiagnosticV9": { + "additionalProperties": false, + "description": "A schema 8 target row whose hierarchy refers to the catalogs.\n\nThe fields are schema 8's, in the same order; only ``hierarchy`` differs.", + "properties": { + "name": { + "title": "Name", + "type": "string" + }, + "target_name": { + "title": "Target Name", + "type": "string" + }, + "period": { + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ], + "title": "Period" + }, + "entity": { + "title": "Entity", + "type": "string" + }, + "measure": { + "$ref": "#/$defs/Selector" + }, + "filter": { + "anyOf": [ + { + "$ref": "#/$defs/Selector" + }, + { + "type": "null" + } + ] + }, + "source": { + "title": "Source", + "type": "string" + }, + "metadata": { + "additionalProperties": { + "$ref": "#/$defs/JsonValue" + }, + "title": "Metadata", + "type": "object" + }, + "target": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Target" + }, + "compiled_target": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Compiled Target" + }, + "initial_estimate": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Initial Estimate" + }, + "final_estimate": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Final Estimate" + }, + "relative_error": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Relative Error" + }, + "within_tolerance": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "title": "Within Tolerance" + }, + "hierarchy": { + "$ref": "#/$defs/CatalogHierarchyReference" + }, + "registry": { + "anyOf": [ + { + "$ref": "#/$defs/RegistryTargetMetadata" + }, + { + "type": "null" + } + ], + "default": null + }, + "target_loss_weight": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Target Loss Weight" + }, + "target_loss_weight_share": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Target Loss Weight Share" + }, + "target_loss_scale": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Target Loss Scale" + }, + "final_capped_scaled_error": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Final Capped Scaled Error" + }, + "final_loss_contribution": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Final Loss Contribution" + } + }, + "required": [ + "name", + "target_name", + "period", + "entity", + "measure", + "filter", + "source", + "metadata", + "target", + "compiled_target", + "initial_estimate", + "final_estimate", + "relative_error", + "within_tolerance", + "hierarchy" + ], + "title": "TargetDiagnosticV9", + "type": "object" + }, + "TargetLossBasis": { + "additionalProperties": false, + "properties": { + "formula": { + "title": "Formula", + "type": "string" + }, + "cap": { + "exclusiveMinimum": 0, + "title": "Cap", + "type": "number" + }, + "target_count": { + "minimum": 0, + "title": "Target Count", + "type": "integer" + }, + "total_target_weight": { + "exclusiveMinimum": 0, + "title": "Total Target Weight", + "type": "number" + }, + "weight_kind": { + "title": "Weight Kind", + "type": "string" + }, + "scale_kind": { + "title": "Scale Kind", + "type": "string" + }, + "hash_algorithm": { + "title": "Hash Algorithm", + "type": "string" + }, + "sha256": { + "pattern": "^[0-9a-f]{64}$", + "title": "Sha256", + "type": "string" + } + }, + "required": [ + "formula", + "cap", + "target_count", + "total_target_weight", + "weight_kind", + "scale_kind", + "hash_algorithm", + "sha256" + ], + "title": "TargetLossBasis", + "type": "object" + }, + "TargetRegistryRef": { + "additionalProperties": false, + "properties": { + "country": { + "title": "Country", + "type": "string" + }, + "version": { + "title": "Version", + "type": "string" + }, + "n_specs": { + "exclusiveMinimum": 0, + "title": "N Specs", + "type": "integer" + } + }, + "required": [ + "country", + "version", + "n_specs" + ], + "title": "TargetRegistryRef", + "type": "object" + }, + "TargetSurface": { + "additionalProperties": false, + "properties": { + "schema_version": { + "const": 1, + "title": "Schema Version", + "type": "integer" + }, + "weight_entity": { + "title": "Weight Entity", + "type": "string" + }, + "n_targets": { + "exclusiveMinimum": 0, + "title": "N Targets", + "type": "integer" + }, + "n_records": { + "exclusiveMinimum": 0, + "title": "N Records", + "type": "integer" + }, + "constraint_matrix": { + "$ref": "#/$defs/TargetSurfaceMatrix" + }, + "sha256": { + "pattern": "^[0-9a-f]{64}$", + "title": "Sha256", + "type": "string" + }, + "names_sha256": { + "pattern": "^[0-9a-f]{64}$", + "title": "Names Sha256", + "type": "string" + }, + "values_sha256": { + "pattern": "^[0-9a-f]{64}$", + "title": "Values Sha256", + "type": "string" + } + }, + "required": [ + "schema_version", + "weight_entity", + "n_targets", + "n_records", + "constraint_matrix", + "sha256", + "names_sha256", + "values_sha256" + ], + "title": "TargetSurface", + "type": "object" + }, + "TargetSurfaceMatrix": { + "additionalProperties": false, + "properties": { + "rows": { + "minimum": 0, + "title": "Rows", + "type": "integer" + }, + "columns": { + "minimum": 0, + "title": "Columns", + "type": "integer" + }, + "nnz": { + "minimum": 0, + "title": "Nnz", + "type": "integer" + } + }, + "required": [ + "rows", + "columns", + "nnz" + ], + "title": "TargetSurfaceMatrix", + "type": "object" + }, + "UKAreaFit": { + "additionalProperties": false, + "properties": { + "geography_level": { + "enum": [ + "local_authority", + "constituency" + ], + "title": "Geography Level", + "type": "string" + }, + "area_code": { + "title": "Area Code", + "type": "string" + }, + "country": { + "enum": [ + "England", + "Northern Ireland", + "Scotland", + "Wales" + ], + "title": "Country", + "type": "string" + }, + "n_targets": { + "exclusiveMinimum": 0, + "title": "N Targets", + "type": "integer" + }, + "n_within_10pct": { + "minimum": 0, + "title": "N Within 10Pct", + "type": "integer" + }, + "pass_rate": { + "maximum": 1, + "minimum": 0, + "title": "Pass Rate", + "type": "number" + }, + "worst_target": { + "title": "Worst Target", + "type": "string" + }, + "worst_abs_relative_error": { + "minimum": 0, + "title": "Worst Abs Relative Error", + "type": "number" + }, + "loss_contribution": { + "minimum": 0, + "title": "Loss Contribution", + "type": "number" + }, + "support_below_floor": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Support Below Floor" + }, + "nonzero_households": { + "anyOf": [ + { + "minimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Nonzero Households" + }, + "nonzero_source_households": { + "anyOf": [ + { + "minimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Nonzero Source Households" + }, + "effective_sample_size": { + "anyOf": [ + { + "minimum": 0, + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Effective Sample Size" + } + }, + "required": [ + "geography_level", + "area_code", + "country", + "n_targets", + "n_within_10pct", + "pass_rate", + "worst_target", + "worst_abs_relative_error", + "loss_contribution" + ], + "title": "UKAreaFit", + "type": "object" + }, + "UKConsumptionDrift": { + "additionalProperties": false, + "description": "Stage-levelled spend and its composition, design against final weights.\n\nReport only (microcosm#1113): the stage sets these levels and shares, so\ncalibration should barely move them; a large move means a bound row is\npulling weight onto the columns instead.", + "properties": { + "report_only": { + "const": true, + "title": "Report Only", + "type": "boolean" + }, + "totals": { + "additionalProperties": { + "$ref": "#/$defs/UKDriftValue" + }, + "minProperties": 1, + "title": "Totals", + "type": "object" + }, + "shares": { + "additionalProperties": { + "$ref": "#/$defs/UKDriftValue" + }, + "minProperties": 1, + "title": "Shares", + "type": "object" + } + }, + "required": [ + "report_only", + "totals", + "shares" + ], + "title": "UKConsumptionDrift", + "type": "object" + }, + "UKCountryFit": { + "additionalProperties": false, + "properties": { + "country": { + "enum": [ + "England", + "Northern Ireland", + "Scotland", + "Wales" + ], + "title": "Country", + "type": "string" + }, + "geography_level": { + "enum": [ + "local_authority", + "constituency" + ], + "title": "Geography Level", + "type": "string" + }, + "n_areas": { + "exclusiveMinimum": 0, + "title": "N Areas", + "type": "integer" + }, + "n_targets": { + "exclusiveMinimum": 0, + "title": "N Targets", + "type": "integer" + }, + "n_within_10pct": { + "minimum": 0, + "title": "N Within 10Pct", + "type": "integer" + }, + "pass_rate": { + "maximum": 1, + "minimum": 0, + "title": "Pass Rate", + "type": "number" + }, + "worst_target": { + "title": "Worst Target", + "type": "string" + }, + "worst_abs_relative_error": { + "minimum": 0, + "title": "Worst Abs Relative Error", + "type": "number" + }, + "loss_contribution": { + "minimum": 0, + "title": "Loss Contribution", + "type": "number" + } + }, + "required": [ + "country", + "geography_level", + "n_areas", + "n_targets", + "n_within_10pct", + "pass_rate", + "worst_target", + "worst_abs_relative_error", + "loss_contribution" + ], + "title": "UKCountryFit", + "type": "object" + }, + "UKDiagnosticsV1": { + "additionalProperties": false, + "properties": { + "schema_version": { + "const": 1, + "title": "Schema Version", + "type": "integer" + }, + "weights": { + "$ref": "#/$defs/UKWeightSummary" + }, + "zero_weight_rows_by_stratum": { + "items": { + "$ref": "#/$defs/UKZeroWeightStratum" + }, + "minItems": 1, + "title": "Zero Weight Rows By Stratum", + "type": "array" + }, + "target_pass_rates_by_geography_level": { + "items": { + "$ref": "#/$defs/UKGeographyPassRate" + }, + "title": "Target Pass Rates By Geography Level", + "type": "array" + }, + "target_observation_basis": { + "additionalProperties": { + "type": "string" + }, + "title": "Target Observation Basis", + "type": "object" + }, + "weakest_families": { + "anyOf": [ + { + "items": { + "$ref": "#/$defs/UKWeakestFamily" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Weakest Families" + }, + "weakest_areas_by_fit": { + "anyOf": [ + { + "$ref": "#/$defs/UKWeakestAreasByFit" + }, + { + "type": "null" + } + ], + "default": null + }, + "rotated_holdout": { + "anyOf": [ + { + "$ref": "#/$defs/UKMeasuredRotatedHoldout" + }, + { + "$ref": "#/$defs/UKSkippedRotatedHoldout" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Rotated Holdout" + }, + "consumption_drift": { + "anyOf": [ + { + "$ref": "#/$defs/UKConsumptionDrift" + }, + { + "type": "null" + } + ], + "default": null + }, + "model_figures_withheld": { + "anyOf": [ + { + "$ref": "#/$defs/UKWithheldModelFigures" + }, + { + "type": "null" + } + ], + "default": null + } + }, + "required": [ + "schema_version", + "weights", + "zero_weight_rows_by_stratum", + "target_pass_rates_by_geography_level", + "target_observation_basis" + ], + "title": "UKDiagnosticsV1", + "type": "object" + }, + "UKDriftValue": { + "additionalProperties": false, + "description": "One quantity at the design weights and at the final weights.", + "properties": { + "design": { + "title": "Design", + "type": "number" + }, + "final": { + "title": "Final", + "type": "number" + }, + "change": { + "title": "Change", + "type": "number" + }, + "relative_change": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Relative Change" + } + }, + "required": [ + "design", + "final", + "change", + "relative_change" + ], + "title": "UKDriftValue", + "type": "object" + }, + "UKGeographyPassRate": { + "additionalProperties": false, + "properties": { + "geography_level": { + "enum": [ + "national", + "region", + "country", + "local_authority", + "constituency", + "brma" + ], + "title": "Geography Level", + "type": "string" + }, + "n_targets": { + "minimum": 0, + "title": "N Targets", + "type": "integer" + }, + "n_scored": { + "minimum": 0, + "title": "N Scored", + "type": "integer" + }, + "n_skipped": { + "minimum": 0, + "title": "N Skipped", + "type": "integer" + }, + "n_within_10pct": { + "minimum": 0, + "title": "N Within 10Pct", + "type": "integer" + }, + "pass_rate": { + "anyOf": [ + { + "maximum": 1, + "minimum": 0, + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Pass Rate" + } + }, + "required": [ + "geography_level", + "n_targets", + "n_scored", + "n_skipped", + "n_within_10pct", + "pass_rate" + ], + "title": "UKGeographyPassRate", + "type": "object" + }, + "UKMeasuredRotatedHoldout": { + "additionalProperties": false, + "properties": { + "report_only": { + "const": true, + "title": "Report Only", + "type": "boolean" + }, + "method": { + "const": "rotated_folds", + "title": "Method", + "type": "string" + }, + "target_loss_cap": { + "exclusiveMinimum": 0, + "title": "Target Loss Cap", + "type": "number" + }, + "loss_weight_scale": { + "const": "held_local_grains_only", + "title": "Loss Weight Scale", + "type": "string" + }, + "target_weight_rule": { + "enum": [ + "uniform", + "grain_equal", + "grain_family_equal", + "grain_family_equal_sqrt_count", + "nation_grain_family_equal", + "nation_grain_family_equal_sqrt_count" + ], + "title": "Target Weight Rule", + "type": "string" + }, + "population": { + "const": "held_out_local_targets", + "title": "Population", + "type": "string" + }, + "grains": { + "items": { + "enum": [ + "constituency", + "local_authority", + "la" + ], + "type": "string" + }, + "minItems": 1, + "title": "Grains", + "type": "array" + }, + "n_folds": { + "minimum": 2, + "title": "N Folds", + "type": "integer" + }, + "seed": { + "title": "Seed", + "type": "integer" + }, + "solve_seed": { + "title": "Solve Seed", + "type": "integer" + }, + "mean_holdout_loss": { + "minimum": 0, + "title": "Mean Holdout Loss", + "type": "number" + }, + "worst_holdout_loss": { + "minimum": 0, + "title": "Worst Holdout Loss", + "type": "number" + }, + "fold_losses": { + "items": { + "minimum": 0, + "type": "number" + }, + "title": "Fold Losses", + "type": "array" + }, + "folds": { + "items": { + "$ref": "#/$defs/UKRotatedHoldoutFold" + }, + "title": "Folds", + "type": "array" + } + }, + "required": [ + "report_only", + "method", + "target_loss_cap", + "loss_weight_scale", + "target_weight_rule", + "population", + "grains", + "n_folds", + "seed", + "solve_seed", + "mean_holdout_loss", + "worst_holdout_loss", + "fold_losses", + "folds" + ], + "title": "UKMeasuredRotatedHoldout", + "type": "object" + }, + "UKRotatedHoldoutFold": { + "additionalProperties": false, + "properties": { + "fold": { + "minimum": 0, + "title": "Fold", + "type": "integer" + }, + "n_train_targets": { + "minimum": 0, + "title": "N Train Targets", + "type": "integer" + }, + "n_holdout_targets": { + "exclusiveMinimum": 0, + "title": "N Holdout Targets", + "type": "integer" + }, + "holdout_target_indices": { + "items": { + "minimum": 0, + "type": "integer" + }, + "title": "Holdout Target Indices", + "type": "array" + }, + "training_national_rows": { + "minimum": 0, + "title": "Training National Rows", + "type": "integer" + }, + "holdout_loss": { + "minimum": 0, + "title": "Holdout Loss", + "type": "number" + } + }, + "required": [ + "fold", + "n_train_targets", + "n_holdout_targets", + "holdout_target_indices", + "training_national_rows", + "holdout_loss" + ], + "title": "UKRotatedHoldoutFold", + "type": "object" + }, + "UKSkippedRotatedHoldout": { + "additionalProperties": false, + "properties": { + "skipped": { + "const": true, + "title": "Skipped", + "type": "boolean" + } + }, + "required": [ + "skipped" + ], + "title": "UKSkippedRotatedHoldout", + "type": "object" + }, + "UKWeakestAreasByFit": { + "additionalProperties": false, + "properties": { + "limit": { + "exclusiveMinimum": 0, + "title": "Limit", + "type": "integer" + }, + "n_areas_scored": { + "minimum": 0, + "title": "N Areas Scored", + "type": "integer" + }, + "bottom_by_fit": { + "items": { + "$ref": "#/$defs/UKAreaFit" + }, + "title": "Bottom By Fit", + "type": "array" + }, + "countries": { + "items": { + "$ref": "#/$defs/UKCountryFit" + }, + "title": "Countries", + "type": "array" + } + }, + "required": [ + "limit", + "n_areas_scored", + "bottom_by_fit", + "countries" + ], + "title": "UKWeakestAreasByFit", + "type": "object" + }, + "UKWeakestFamily": { + "additionalProperties": false, + "properties": { + "family": { + "title": "Family", + "type": "string" + }, + "n_targets": { + "exclusiveMinimum": 0, + "title": "N Targets", + "type": "integer" + }, + "n_within_10pct": { + "minimum": 0, + "title": "N Within 10Pct", + "type": "integer" + }, + "pass_rate": { + "maximum": 1, + "minimum": 0, + "title": "Pass Rate", + "type": "number" + }, + "worst_target": { + "title": "Worst Target", + "type": "string" + }, + "worst_abs_relative_error": { + "minimum": 0, + "title": "Worst Abs Relative Error", + "type": "number" + }, + "loss_contribution": { + "minimum": 0, + "title": "Loss Contribution", + "type": "number" + }, + "loss_share": { + "maximum": 1, + "minimum": 0, + "title": "Loss Share", + "type": "number" + } + }, + "required": [ + "family", + "n_targets", + "n_within_10pct", + "pass_rate", + "worst_target", + "worst_abs_relative_error", + "loss_contribution", + "loss_share" + ], + "title": "UKWeakestFamily", + "type": "object" + }, + "UKWeightSummary": { + "additionalProperties": false, + "properties": { + "n_records": { + "exclusiveMinimum": 0, + "title": "N Records", + "type": "integer" + }, + "positive_weight_records": { + "minimum": 0, + "title": "Positive Weight Records", + "type": "integer" + }, + "zero_weight_records": { + "minimum": 0, + "title": "Zero Weight Records", + "type": "integer" + }, + "total_weight": { + "minimum": 0, + "title": "Total Weight", + "type": "number" + }, + "effective_sample_size": { + "minimum": 0, + "title": "Effective Sample Size", + "type": "number" + }, + "ess_fraction": { + "maximum": 1, + "minimum": 0, + "title": "Ess Fraction", + "type": "number" + }, + "median_positive_weight": { + "anyOf": [ + { + "exclusiveMinimum": 0, + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Median Positive Weight" + }, + "max_weight": { + "minimum": 0, + "title": "Max Weight", + "type": "number" + }, + "max_to_median_positive_weight": { + "anyOf": [ + { + "minimum": 1, + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Max To Median Positive Weight" + }, + "top_1pct_weight_share": { + "maximum": 1, + "minimum": 0, + "title": "Top 1Pct Weight Share", + "type": "number" + } + }, + "required": [ + "n_records", + "positive_weight_records", + "zero_weight_records", + "total_weight", + "effective_sample_size", + "ess_fraction", + "median_positive_weight", + "max_weight", + "max_to_median_positive_weight", + "top_1pct_weight_share" + ], + "title": "UKWeightSummary", + "type": "object" + }, + "UKWithheldModelFigures": { + "additionalProperties": false, + "description": "Target rows whose model figures stay in the build (microcosm#1262).\n\nA row resting on fewer than ``min_source_households`` distinct source\nhouseholds keeps its place without its model figures. While any does, the\npayload's totals are restated over the published rows (``totals_basis``)\nso that none can be differenced against them, and its loss trajectory is\nempty.", + "properties": { + "targets": { + "exclusiveMinimum": 0, + "title": "Targets", + "type": "integer" + }, + "min_source_households": { + "exclusiveMinimum": 0, + "title": "Min Source Households", + "type": "integer" + }, + "totals_basis": { + "const": "published_rows", + "title": "Totals Basis", + "type": "string" + }, + "note": { + "title": "Note", + "type": "string" + } + }, + "required": [ + "targets", + "min_source_households", + "totals_basis", + "note" + ], + "title": "UKWithheldModelFigures", + "type": "object" + }, + "UKZeroWeightStratum": { + "additionalProperties": false, + "properties": { + "stratum": { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "boolean" + }, + { + "type": "integer" + }, + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "minProperties": 1, + "title": "Stratum", + "type": "object" + }, + "rows": { + "minimum": 0, + "title": "Rows", + "type": "integer" + }, + "positive_weight_rows": { + "minimum": 0, + "title": "Positive Weight Rows", + "type": "integer" + }, + "zero_weight_rows": { + "minimum": 0, + "title": "Zero Weight Rows", + "type": "integer" + }, + "weight_sum": { + "minimum": 0, + "title": "Weight Sum", + "type": "number" + } + }, + "required": [ + "stratum", + "rows", + "positive_weight_rows", + "zero_weight_rows", + "weight_sum" + ], + "title": "UKZeroWeightStratum", + "type": "object" + } + }, + "additionalProperties": false, + "properties": { + "schema_version": { + "const": 9, + "title": "Schema Version", + "type": "integer" + }, + "weight_entity": { + "title": "Weight Entity", + "type": "string" + }, + "options": { + "additionalProperties": { + "$ref": "#/$defs/JsonValue" + }, + "title": "Options", + "type": "object" + }, + "target_surface": { + "$ref": "#/$defs/TargetSurface" + }, + "target_registry": { + "$ref": "#/$defs/TargetRegistryRef" + }, + "l0_lambda": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "L0 Lambda" + }, + "n_nonzero": { + "minimum": 0, + "title": "N Nonzero", + "type": "integer" + }, + "n_records": { + "exclusiveMinimum": 0, + "title": "N Records", + "type": "integer" + }, + "initial_loss": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Initial Loss" + }, + "final_loss": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Final Loss" + }, + "fraction_within_10pct": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Fraction Within 10Pct" + }, + "effective_sample_size": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Effective Sample Size" + }, + "realized_max_weight_ratio": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Realized Max Weight Ratio" + }, + "top_1pct_weight_share": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "title": "Top 1Pct Weight Share" + }, + "loss_trajectory": { + "items": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "title": "Loss Trajectory", + "type": "array" + }, + "skipped": { + "items": { + "$ref": "#/$defs/SkippedTarget" + }, + "title": "Skipped", + "type": "array" + }, + "past_cap_census": { + "anyOf": [ + { + "$ref": "#/$defs/PastCapCensus" + }, + { + "type": "null" + } + ] + }, + "diagnostic_warnings": { + "items": { + "$ref": "#/$defs/DiagnosticsWarning" + }, + "title": "Diagnostic Warnings", + "type": "array" + }, + "hierarchy_catalogs": { + "$ref": "#/$defs/HierarchyCatalogs" + }, + "targets": { + "items": { + "$ref": "#/$defs/TargetDiagnosticV9" + }, + "title": "Targets", + "type": "array" + }, + "target_loss_basis": { + "anyOf": [ + { + "$ref": "#/$defs/TargetLossBasis" + }, + { + "type": "null" + } + ], + "default": null + }, + "build": { + "anyOf": [ + { + "additionalProperties": { + "$ref": "#/$defs/JsonValue" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Build" + }, + "uk_diagnostics": { + "anyOf": [ + { + "$ref": "#/$defs/UKDiagnosticsV1" + }, + { + "type": "null" + } + ], + "default": null + } + }, + "required": [ + "schema_version", + "weight_entity", + "options", + "target_surface", + "target_registry", + "l0_lambda", + "n_nonzero", + "n_records", + "initial_loss", + "final_loss", + "fraction_within_10pct", + "effective_sample_size", + "realized_max_weight_ratio", + "top_1pct_weight_share", + "loss_trajectory", + "skipped", + "past_cap_census", + "diagnostic_warnings", + "hierarchy_catalogs", + "targets" + ], + "type": "object" +} diff --git a/packages/microcosm-diagnostics/src/microcosm/diagnostics/writer.py b/packages/microcosm-diagnostics/src/microcosm/diagnostics/writer.py index 09c6795e8..5ea5e828d 100644 --- a/packages/microcosm-diagnostics/src/microcosm/diagnostics/writer.py +++ b/packages/microcosm-diagnostics/src/microcosm/diagnostics/writer.py @@ -13,9 +13,15 @@ from pydantic import ValidationError +from microcosm.diagnostics.catalogs import ( + HierarchyLabelChoices, + HierarchyLabelConflictError, + catalog_calibration_diagnostics, +) from microcosm.diagnostics.schema import ( CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, ) _LOGGER = logging.getLogger(__name__) @@ -23,6 +29,9 @@ DiagnosticsFailureCode = Literal[ "construction_error", "validation_error", + # The rows give one identifier two labels and no label is chosen + # (HierarchyLabelConflictError); schema 9 holds one label per identifier. + "label_conflict", "serialization_error", "io_error", "unexpected_error", @@ -33,14 +42,14 @@ class DiagnosticsWriteSuccess: status: Literal["available"] path: Path - schema_version: Literal[8] + schema_version: Literal[9] sha256: str @dataclass(frozen=True) class DiagnosticsWriteFailure: status: Literal["failed"] - expected_schema_version: Literal[8] + expected_schema_version: Literal[9] error_code: DiagnosticsFailureCode message: str @@ -92,15 +101,30 @@ def failed_diagnostics_outcome( def write_calibration_diagnostics( - diagnostics: CalibrationDiagnosticsV8, + diagnostics: CalibrationDiagnosticsV8 | CalibrationDiagnosticsV9, path: Path | str, + *, + label_choices: HierarchyLabelChoices | None = None, ) -> DiagnosticsWriteOutcome: - """Validate and atomically write diagnostics without failing its dataset build.""" + """Validate and atomically write schema 9 diagnostics without failing a build. + + A schema 8 document, as producers assemble it, is published as schema 9: + its hierarchy labels move into ``hierarchy_catalogs`` + (:func:`catalog_calibration_diagnostics`, with ``label_choices``). When the + rows give one identifier two labels and no label is chosen, nothing is + written and the outcome's error code is ``label_conflict``. + """ output = Path(path) temporary = output.with_name(f".{output.name}.{uuid.uuid4().hex}.tmp") try: - validated = CalibrationDiagnosticsV8.model_validate(diagnostics) + if isinstance(diagnostics, CalibrationDiagnosticsV9): + validated = CalibrationDiagnosticsV9.model_validate(diagnostics) + else: + validated = catalog_calibration_diagnostics( + CalibrationDiagnosticsV8.model_validate(diagnostics), + label_choices=label_choices, + ) payload = validated.model_dump(mode="python", exclude_defaults=True) encoded = json.dumps( payload, @@ -110,6 +134,14 @@ def write_calibration_diagnostics( ).encode("utf-8") temporary.write_bytes(encoded) os.replace(temporary, output) + except HierarchyLabelConflictError as error: + # The outcome's message is shortened; log every conflicting identifier. + _LOGGER.warning("Calibration diagnostics label conflicts: %s", error) + return failed_diagnostics_outcome( + error, + error_code="label_conflict", + path=output, + ) except ValidationError as error: return failed_diagnostics_outcome( error, diff --git a/packages/microcosm-diagnostics/tests/engine_free/shared/test_hierarchy_catalogs.py b/packages/microcosm-diagnostics/tests/engine_free/shared/test_hierarchy_catalogs.py new file mode 100644 index 000000000..7151d5e95 --- /dev/null +++ b/packages/microcosm-diagnostics/tests/engine_free/shared/test_hierarchy_catalogs.py @@ -0,0 +1,587 @@ +"""Diagnostics schema 9: hierarchy labels declared once, in catalogs. + +Schema 8 copies every provider, category, geography and dimension label onto +every target row, so one geography or one dimension value could be labelled +two ways in one file. Schema 9 declares each label once in +``hierarchy_catalogs`` and rows refer to it by ID. The writer builds the +catalogs from the rows and refuses a document whose rows give one identifier +two labels, unless an explicit label choice resolves it. +""" + +from __future__ import annotations + +import copy +import json +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from microcosm.diagnostics import ( + CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION, + CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, + SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, + CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, + HierarchyLabelChoices, + HierarchyLabelConflictError, + catalog_calibration_diagnostics, + expand_calibration_diagnostics, + parse_calibration_diagnostics, + write_calibration_diagnostics, +) +from test_support.microcosm_diagnostics.payloads import schema_8_payload + +AGI = "us:statutes/26/62#adjusted_gross_income" + + +def _target( + name: str, + *, + geography: dict, + dimensions: list[dict], + provider: dict | None = None, + category: dict | None = None, +) -> dict: + row = copy.deepcopy(schema_8_payload()["targets"][0]) + provider = provider or {"id": "irs_soi", "label": "IRS Statistics of Income"} + category = category or { + "id": "irs_soi.eitc", + "label": "Earned income tax credit", + "provider_id": provider["id"], + } + row.update(name=f"{name}@2024", target_name=name) + row["hierarchy"] = { + "provider": provider, + "category": category, + "geography": geography, + "dimensions": dimensions, + "target": {"id": name, "label": name.replace(".", " ")}, + } + return row + + +def _payload(targets: list[dict]) -> dict: + payload = schema_8_payload() + payload["targets"] = targets + payload["target_surface"]["n_targets"] = len(targets) + payload["target_surface"]["constraint_matrix"]["rows"] = len(targets) + payload["target_registry"]["n_specs"] = len(targets) + return payload + + +UNITED_STATES = {"id": "0100000US", "label": "United States", "level": "country"} +MAINE = {"id": "0400000US23", "label": "Maine", "level": "state"} + + +def _band(value_id: str, value_label: str) -> dict: + return { + "id": AGI, + "label": "Adjusted gross income", + "value_id": value_id, + "value_label": value_label, + } + + +def _status(value_id: str, value_label: str) -> dict: + return { + "id": "filing_status", + "label": "Filing status", + "value_id": value_id, + "value_label": value_label, + } + + +def _us_payload() -> dict: + """Three rows; both dimensions have a value ``all``.""" + + return _payload( + [ + _target( + "irs_soi.eitc.under_1.all", + geography=UNITED_STATES, + dimensions=[ + _band("under_1", "No adjusted gross income"), + _status("all", "All filing statuses"), + ], + ), + _target( + "irs_soi.eitc.all.joint", + geography=UNITED_STATES, + dimensions=[ + _band("all", "All returns"), + _status("joint", "Married filing jointly"), + ], + ), + _target("irs_soi.eitc.maine", geography=MAINE, dimensions=[]), + ] + ) + + +def _model(payload: dict) -> CalibrationDiagnosticsV8: + return CalibrationDiagnosticsV8.model_validate(payload) + + +def test_the_published_schema_is_9_and_producers_still_assemble_schema_8() -> None: + assert CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION == 9 + assert CALIBRATION_DIAGNOSTICS_ASSEMBLY_SCHEMA_VERSION == 8 + assert HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS == frozenset({8, 9}) + assert SUPPORTED_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS == frozenset({6, 7, 8, 9}) + + +def test_catalogs_declare_each_label_once_and_rows_refer_by_id() -> None: + catalogued = catalog_calibration_diagnostics(_model(_us_payload())) + document = json.loads(catalogued.model_dump_json(exclude_defaults=True)) + + assert document["schema_version"] == 9 + # Catalog entries are listed in the order the rows first use them. + assert document["hierarchy_catalogs"] == { + "providers": [{"id": "irs_soi", "label": "IRS Statistics of Income"}], + "categories": [ + { + "id": "irs_soi.eitc", + "label": "Earned income tax credit", + "provider_id": "irs_soi", + } + ], + "geographies": [UNITED_STATES, MAINE], + "dimensions": [ + { + "id": AGI, + "label": "Adjusted gross income", + "values": [ + {"id": "under_1", "label": "No adjusted gross income"}, + {"id": "all", "label": "All returns"}, + ], + }, + { + "id": "filing_status", + "label": "Filing status", + "values": [ + {"id": "all", "label": "All filing statuses"}, + {"id": "joint", "label": "Married filing jointly"}, + ], + }, + ], + } + assert [row["hierarchy"] for row in document["targets"]] == [ + { + "category_id": "irs_soi.eitc", + "geography_id": "0100000US", + "dimensions": [ + {"id": AGI, "value_id": "under_1"}, + {"id": "filing_status", "value_id": "all"}, + ], + "target": { + "id": "irs_soi.eitc.under_1.all", + "label": "irs_soi eitc under_1 all", + }, + }, + { + "category_id": "irs_soi.eitc", + "geography_id": "0100000US", + "dimensions": [ + {"id": AGI, "value_id": "all"}, + {"id": "filing_status", "value_id": "joint"}, + ], + "target": { + "id": "irs_soi.eitc.all.joint", + "label": "irs_soi eitc all joint", + }, + }, + { + "category_id": "irs_soi.eitc", + "geography_id": "0400000US23", + "dimensions": [], + "target": {"id": "irs_soi.eitc.maine", "label": "irs_soi eitc maine"}, + }, + ] + # Every other field is schema 8's. + for written, assembled in zip( + document["targets"], _us_payload()["targets"], strict=True + ): + assert {key: value for key, value in written.items() if key != "hierarchy"} == { + key: value for key, value in assembled.items() if key != "hierarchy" + } + + +def test_expanding_schema_9_gives_back_the_schema_8_document() -> None: + model = _model(_us_payload()) + + assert expand_calibration_diagnostics(catalog_calibration_diagnostics(model)) == ( + model + ) + + +def test_the_writer_writes_schema_9_and_reports_it(tmp_path: Path) -> None: + model = _model(_us_payload()) + path = tmp_path / "calibration_diagnostics.json" + + outcome = write_calibration_diagnostics(model, path) + + assert outcome.status == "available" + assert outcome.schema_version == 9 + written = json.loads(path.read_text()) + assert written == json.loads( + catalog_calibration_diagnostics(model).model_dump_json(exclude_defaults=True) + ) + parsed = parse_calibration_diagnostics(written) + assert isinstance(parsed, CalibrationDiagnosticsV9) + assert expand_calibration_diagnostics(parsed) == model + + +def _conflict_payload(kind: str) -> dict: + payload = _us_payload() + first, second, third = payload["targets"] + if kind == "geography label": + third["hierarchy"]["geography"] = {**UNITED_STATES, "label": "USA"} + elif kind == "geography level": + third["hierarchy"]["geography"] = {**UNITED_STATES, "level": "nation"} + elif kind == "dimension value label": + second["hierarchy"]["dimensions"][0] = _band( + "under_1", "No adjusted gross income and deficit" + ) + elif kind == "dimension label": + second["hierarchy"]["dimensions"][1]["label"] = "Filing status of return" + elif kind == "provider label": + third["hierarchy"]["provider"] = {"id": "irs_soi", "label": "IRS SOI"} + third["hierarchy"]["category"]["provider_id"] = "irs_soi" + elif kind == "category label": + third["hierarchy"]["category"] = { + **first["hierarchy"]["category"], + "label": "EITC", + } + elif kind == "category provider": + third["hierarchy"]["provider"] = {"id": "irs", "label": "IRS"} + third["hierarchy"]["category"] = { + **first["hierarchy"]["category"], + "provider_id": "irs", + } + return payload + + +@pytest.mark.parametrize( + ("kind", "message"), + [ + ( + "geography label", + "geography 0100000US: 'United States' (first target " + "irs_soi.eitc.under_1.all@2024), 'USA' (first target " + "irs_soi.eitc.maine@2024)", + ), + ( + "geography level", + "geography 0100000US level: 'country' (first target " + "irs_soi.eitc.under_1.all@2024), 'nation' (first target " + "irs_soi.eitc.maine@2024)", + ), + ( + "dimension value label", + f"dimension {AGI} value under_1: 'No adjusted gross income' (first " + "target irs_soi.eitc.under_1.all@2024), 'No adjusted gross income and " + "deficit' (first target irs_soi.eitc.all.joint@2024)", + ), + ( + "dimension label", + "dimension filing_status: 'Filing status' (first target " + "irs_soi.eitc.under_1.all@2024), 'Filing status of return' (first " + "target irs_soi.eitc.all.joint@2024)", + ), + ( + "provider label", + "provider irs_soi: 'IRS Statistics of Income' (first target " + "irs_soi.eitc.under_1.all@2024), 'IRS SOI' (first target " + "irs_soi.eitc.maine@2024)", + ), + ( + "category label", + "category irs_soi.eitc: 'Earned income tax credit' (first target " + "irs_soi.eitc.under_1.all@2024), 'EITC' (first target " + "irs_soi.eitc.maine@2024)", + ), + ( + "category provider", + "category irs_soi.eitc provider: 'irs_soi' (first target " + "irs_soi.eitc.under_1.all@2024), 'irs' (first target " + "irs_soi.eitc.maine@2024)", + ), + ], +) +def test_two_labels_for_one_identifier_are_refused(kind: str, message: str) -> None: + with pytest.raises(HierarchyLabelConflictError) as raised: + catalog_calibration_diagnostics(_model(_conflict_payload(kind))) + + assert message in str(raised.value) + assert len(raised.value.conflicts) == 1 + + +def test_the_writer_refuses_a_conflict_and_writes_nothing(tmp_path: Path) -> None: + path = tmp_path / "calibration_diagnostics.json" + path.write_text("stale") + + outcome = write_calibration_diagnostics( + _model(_conflict_payload("dimension value label")), path + ) + + assert outcome.status == "failed" + assert outcome.error_code == "label_conflict" + assert outcome.expected_schema_version == 9 + assert "No adjusted gross income and deficit" in outcome.message + assert not path.exists() + assert list(tmp_path.glob(f".{path.name}.*.tmp")) == [] + + +def test_a_label_choice_resolves_a_geography_conflict_only() -> None: + payload = _us_payload() + payload["targets"][1]["hierarchy"]["geography"] = {**UNITED_STATES, "label": "USA"} + # A chosen label applies only where the rows disagree: Maine has one label + # among the rows, so it keeps it although the choices name it otherwise. + choices = HierarchyLabelChoices( + geographies={"0100000US": "United States", "0400000US23": "State of Maine"} + ) + + catalogued = catalog_calibration_diagnostics(_model(payload), label_choices=choices) + + assert [ + (entry.id, entry.label) for entry in catalogued.hierarchy_catalogs.geographies + ] == [("0100000US", "United States"), ("0400000US23", "Maine")] + + +def test_a_label_choice_resolves_a_dimension_value_conflict() -> None: + choices = HierarchyLabelChoices( + dimension_values={(AGI, "under_1"): "No adjusted gross income"} + ) + + catalogued = catalog_calibration_diagnostics( + _model(_conflict_payload("dimension value label")), label_choices=choices + ) + + # The second row now names band ``under_1`` too, so ``all`` is unused. + (band, _status_dimension) = catalogued.hierarchy_catalogs.dimensions + assert [(value.id, value.label) for value in band.values] == [ + ("under_1", "No adjusted gross income"), + ] + + +def test_a_dimension_value_choice_covers_only_its_reviewed_spellings() -> None: + # Like Chronicle's REVIEWED_VALUE_LABEL_VARIANTS (chronicle#377): a choice + # resolves the spellings it was reviewed for, and a new one is refused. + reviewed = HierarchyLabelChoices( + dimension_values={(AGI, "under_1"): "No adjusted gross income"}, + dimension_value_spellings={ + (AGI, "under_1"): frozenset( + {"No adjusted gross income", "No adjusted gross income and deficit"} + ) + }, + ) + catalogued = catalog_calibration_diagnostics( + _model(_conflict_payload("dimension value label")), label_choices=reviewed + ) + (band, _status_dimension) = catalogued.hierarchy_catalogs.dimensions + assert [(value.id, value.label) for value in band.values] == [ + ("under_1", "No adjusted gross income") + ] + + payload = _conflict_payload("dimension value label") + payload["targets"][1]["hierarchy"]["dimensions"][0]["value_label"] = "No AGI" + with pytest.raises(HierarchyLabelConflictError) as raised: + catalog_calibration_diagnostics(_model(payload), label_choices=reviewed) + assert ( + f"dimension {AGI} value under_1 (a spelling its reviewed list does not " + "include): 'No adjusted gross income' (first target " + "irs_soi.eitc.under_1.all@2024), 'No AGI'" + ) in str(raised.value) + + +def test_every_disagreement_a_choice_resolves_is_logged_as_a_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + # A choice never hides a disagreement: the build log names the identifier, + # each spelling with its first target, and the label written. + choices = HierarchyLabelChoices( + dimension_values={(AGI, "under_1"): "No adjusted gross income"} + ) + + with caplog.at_level("WARNING", logger="microcosm.diagnostics.catalogs"): + catalog_calibration_diagnostics( + _model(_conflict_payload("dimension value label")), label_choices=choices + ) + + (record,) = caplog.records + assert record.levelname == "WARNING" + assert record.getMessage() == ( + "Calibration diagnostics: the target rows label 1 identifier(s) more " + "than one way; the chosen label is written for each: " + f"dimension {AGI} value under_1: 'No adjusted gross income' (first " + "target irs_soi.eitc.under_1.all@2024), 'No adjusted gross income and " + "deficit' (first target irs_soi.eitc.all.joint@2024) -> 'No adjusted " + "gross income'" + ) + + +def test_a_payload_whose_rows_agree_logs_nothing( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level("WARNING", logger="microcosm.diagnostics.catalogs"): + catalog_calibration_diagnostics(_model(_us_payload())) + + assert caplog.records == [] + + +def test_a_label_choice_does_not_resolve_a_level_conflict() -> None: + choices = HierarchyLabelChoices(geographies={"0100000US": "United States"}) + + with pytest.raises(HierarchyLabelConflictError, match="0100000US level"): + catalog_calibration_diagnostics( + _model(_conflict_payload("geography level")), label_choices=choices + ) + + +# The UK pattern found in the published UK trees: Chronicle's local facts +# carry the area as a dimension named ``geography`` whose value ID is the +# lower-case GSS code, labelled with each publisher's own name ("Hartlepool +# UA" from HMRC, "Hartlepool" from ONS), while the row's geography already has +# one label. +HARTLEPOOL = {"id": "E06000001", "label": "Hartlepool", "level": "local_authority"} + + +def _area(value_label: str, value_id: str = "e06000001") -> dict: + return { + "id": "geography", + "label": "Geography", + "value_id": value_id, + "value_label": value_label, + } + + +def _uk_area_payload() -> dict: + return _payload( + [ + _target( + "hmrc.income.e06000001", + geography=HARTLEPOOL, + dimensions=[_area("Hartlepool UA")], + ), + _target( + "ons.population.e06000001", + geography=HARTLEPOOL, + dimensions=[_area("Hartlepool")], + ), + ] + ) + + +def test_a_geography_dimension_value_takes_its_area_label_from_the_payload() -> None: + model = _model(_uk_area_payload()) + with pytest.raises(HierarchyLabelConflictError, match="value e06000001"): + catalog_calibration_diagnostics(model) + + catalogued = catalog_calibration_diagnostics( + model, + label_choices=HierarchyLabelChoices( + geography_dimensions=frozenset({"geography"}) + ), + ) + + (area,) = catalogued.hierarchy_catalogs.dimensions + assert [(value.id, value.label) for value in area.values] == [ + ("e06000001", "Hartlepool") + ] + + +def test_a_geography_dimension_value_outside_the_payload_uses_the_chosen_label() -> ( + None +): + payload = _uk_area_payload() + # Neither row is at Middlesbrough, so the payload's geographies do not + # name it; the chosen geography labels do, by upper-case GSS code. + for row, label in zip( + payload["targets"], ("Middlesbrough UA", "Middlesbrough"), strict=True + ): + row["hierarchy"]["dimensions"] = [_area(label, "e06000002")] + choices = HierarchyLabelChoices( + geographies={"E06000002": "Middlesbrough"}, + geography_dimensions=frozenset({"geography"}), + ) + + catalogued = catalog_calibration_diagnostics(_model(payload), label_choices=choices) + + (area,) = catalogued.hierarchy_catalogs.dimensions + assert [(value.id, value.label) for value in area.values] == [ + ("e06000002", "Middlesbrough") + ] + + +def _schema_9_document() -> dict: + return json.loads( + catalog_calibration_diagnostics(_model(_us_payload())).model_dump_json( + exclude_defaults=True + ) + ) + + +def _corrupt(document: dict, change: str) -> None: + catalogs = document["hierarchy_catalogs"] + first_row = document["targets"][0]["hierarchy"] + if change == "unknown category": + first_row["category_id"] = "irs_soi.ctc" + elif change == "unknown geography": + first_row["geography_id"] = "0400000US06" + elif change == "unknown dimension": + first_row["dimensions"][0]["id"] = "income_range" + elif change == "value under the wrong dimension": + first_row["dimensions"][1]["value_id"] = "under_1" + elif change == "duplicate geography": + catalogs["geographies"].append({**catalogs["geographies"][0], "label": "USA"}) + elif change == "duplicate dimension value": + catalogs["dimensions"][0]["values"].append({"id": "all", "label": "All"}) + elif change == "category with an unknown provider": + catalogs["categories"][0]["provider_id"] = "irs" + elif change == "dimension listed twice on a row": + first_row["dimensions"][1] = dict(first_row["dimensions"][0]) + elif change == "hierarchy target of another row": + first_row["target"]["id"] = "irs_soi.eitc.all.joint" + + +@pytest.mark.parametrize( + ("change", "message"), + [ + ("unknown category", "category irs_soi.ctc is not in hierarchy_catalogs"), + ("unknown geography", "geography 0400000US06 is not in hierarchy_catalogs"), + ("unknown dimension", "dimension income_range is not in hierarchy_catalogs"), + ( + "value under the wrong dimension", + "value under_1 is not listed under dimension filing_status", + ), + ("duplicate geography", "hierarchy_catalogs.geographies lists 0100000US twice"), + ( + "duplicate dimension value", + f"hierarchy_catalogs dimension {AGI} lists value all twice", + ), + ( + "category with an unknown provider", + "category irs_soi.eitc refers to provider irs, which " + "hierarchy_catalogs.providers does not list", + ), + ("dimension listed twice on a row", "hierarchy dimension ids must be unique"), + ("hierarchy target of another row", "hierarchy target id must match"), + ], +) +def test_schema_9_refuses_a_structurally_inconsistent_document( + change: str, message: str +) -> None: + document = _schema_9_document() + _corrupt(document, change) + + with pytest.raises(ValidationError, match=message): + parse_calibration_diagnostics(document) + + +def test_both_hierarchy_schemas_parse_to_their_own_models() -> None: + assert isinstance( + parse_calibration_diagnostics(_us_payload()), CalibrationDiagnosticsV8 + ) + assert isinstance( + parse_calibration_diagnostics(_schema_9_document()), CalibrationDiagnosticsV9 + ) diff --git a/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema.py b/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema.py index 2a437107d..d5e1f076e 100644 --- a/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema.py +++ b/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema.py @@ -676,11 +676,11 @@ def test_generator_tool_writes_and_checks_the_committed_schema( written = tmp_path / committed.name assert tool.main(["--check"]) == 0 - assert tool.main(["--out", str(written)]) == 0 + assert tool.main(["--schema-version", "8", "--out", str(written)]) == 0 assert written.read_bytes() == committed.read_bytes() written.write_text("{}\n", encoding="utf-8") - assert tool.main(["--check", "--out", str(written)]) == 1 + assert tool.main(["--check", "--schema-version", "8", "--out", str(written)]) == 1 assert "regenerate it with" in capsys.readouterr().err with pytest.raises(ValueError, match="differs from the schema the models generate"): diagnostics.assert_calibration_diagnostics_v8_json_schema_current(written) diff --git a/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema_v9.py b/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema_v9.py new file mode 100644 index 000000000..472f1b316 --- /dev/null +++ b/packages/microcosm-diagnostics/tests/engine_free/shared/test_json_schema_v9.py @@ -0,0 +1,319 @@ +"""The JSON Schema generated for calibration diagnostics schema 9. + +Schema 9 is what the writer publishes. Its JSON Schema is generated from +``CalibrationDiagnosticsV9`` the way schema 8's is from +``CalibrationDiagnosticsV8``; the schema-8 file stays committed for documents +already published. The models remain the authority, so every model rule the +generated schema cannot express is listed, with a document that shows it. +""" + +from __future__ import annotations + +import copy +import importlib.util +import json +from collections.abc import Callable +from functools import cache +from pathlib import Path + +import pytest +from jsonschema import Draft202012Validator +from pydantic import ValidationError + +from microcosm import diagnostics +from test_support.microcosm_diagnostics.json_schema import ( + json_schema_errors, + load_strict_json, + published_json_schema_errors, +) +from test_support.microcosm_diagnostics.payloads import ( + schema_8_payload, + uk_schema_8_payload, +) +from test_support.paths import paths_for + +_TEST_PATHS = paths_for("microcosm-diagnostics") +DRAFT_2020_12 = "https://json-schema.org/draft/2020-12/schema" +PUBLISHED_V9_SCHEMA_PATH = ( + "packages/microcosm-diagnostics/src/microcosm/diagnostics/schemas/" + "calibration_diagnostics.v9.schema.json" +) +#: Gaps that exist only because schema 9 refers to catalogs by ID. +CATALOG_GAP_KEYS = frozenset( + {"hierarchy-catalog-identity", "hierarchy-references", "target-hierarchy"} +) + +DocumentMutation = Callable[[dict], None] + + +@cache +def _generated_schema_text() -> str: + return json.dumps(diagnostics.calibration_diagnostics_v9_json_schema()) + + +def _generated_schema() -> dict: + return json.loads(_generated_schema_text()) + + +def _schema_9(payload: dict) -> dict: + """The document the writer publishes for a schema-8 payload.""" + + model = diagnostics.CalibrationDiagnosticsV8.model_validate(payload) + return json.loads( + diagnostics.catalog_calibration_diagnostics(model).model_dump_json( + exclude_defaults=True + ) + ) + + +def _with_dimension(payload: dict) -> dict: + payload = copy.deepcopy(payload) + payload["targets"][0]["hierarchy"]["dimensions"] = [ + { + "id": "age_band", + "label": "Age band", + "value_id": "0_17", + "value_label": "Under 18", + } + ] + return payload + + +def test_v9_schema_declares_draft_2020_12_and_passes_its_metaschema() -> None: + schema = _generated_schema() + + assert schema["$schema"] == DRAFT_2020_12 + assert schema["$id"] == diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID + Draft202012Validator.check_schema(schema) + + +def test_v9_schema_pins_version_9_requires_catalogs_and_forbids_undeclared_fields() -> ( + None +): + schema = _generated_schema() + + assert schema["title"] == "Microcosm calibration diagnostics, schema 9" + assert schema["properties"]["schema_version"]["const"] == 9 + assert "hierarchy_catalogs" in schema["required"] + assert schema["additionalProperties"] is False + assert all( + definition.get("additionalProperties") is False + for definition in schema["$defs"].values() + if definition.get("type") == "object" + ) + + +@pytest.mark.parametrize( + "payload", + [ + pytest.param(schema_8_payload(), id="us"), + pytest.param(_with_dimension(schema_8_payload()), id="us-with-dimension"), + pytest.param(uk_schema_8_payload(), id="uk"), + ], +) +def test_v9_schema_accepts_the_documents_the_writer_publishes(payload: dict) -> None: + document = load_strict_json(json.dumps(_schema_9(payload), allow_nan=False)) + + assert json_schema_errors(document, _generated_schema()) == [] + assert published_json_schema_errors(document) == [] + + +def test_each_schema_refuses_the_other_version() -> None: + schema_8 = schema_8_payload() + schema_9 = _schema_9(schema_8) + + assert json_schema_errors(schema_8, _generated_schema()) + assert json_schema_errors( + schema_9, diagnostics.calibration_diagnostics_v8_json_schema() + ) + + +def _row_hierarchy(document: dict) -> dict: + return document["targets"][0]["hierarchy"] + + +def _set_category(document: dict) -> None: + _row_hierarchy(document)["category_id"] = "census.households" + + +def _duplicate_geography(document: dict) -> None: + geographies = document["hierarchy_catalogs"]["geographies"] + geographies.append({**geographies[0], "label": "USA"}) + + +def _misplace_value(document: dict) -> None: + document["hierarchy_catalogs"]["dimensions"].append( + {"id": "sex", "label": "Sex", "values": [{"id": "female", "label": "Female"}]} + ) + _row_hierarchy(document)["dimensions"] = [{"id": "age_band", "value_id": "female"}] + + +def _repeat_row_dimension(document: dict) -> None: + dimensions = _row_hierarchy(document)["dimensions"] + dimensions.append(dict(dimensions[0])) + + +def _point_at_another_target(document: dict) -> None: + _row_hierarchy(document)["target"]["id"] = "households" + + +GAP_CASES = [ + pytest.param( + "hierarchy-references", + _set_category, + "category census.households is not in hierarchy_catalogs", + id="unknown-category", + ), + pytest.param( + "hierarchy-references", + _misplace_value, + "value female is not listed under dimension age_band", + id="value-under-the-wrong-dimension", + ), + pytest.param( + "hierarchy-references", + _repeat_row_dimension, + "hierarchy dimension ids must be unique", + id="dimension-twice-on-a-row", + ), + pytest.param( + "hierarchy-catalog-identity", + _duplicate_geography, + "hierarchy_catalogs.geographies lists 0100000US twice", + id="duplicate-geography", + ), + pytest.param( + "target-hierarchy", + _point_at_another_target, + "hierarchy target id must match", + id="hierarchy-target-of-another-row", + ), +] + + +@pytest.mark.parametrize(("gap_key", "mutation", "model_error"), GAP_CASES) +def test_v9_schema_accepts_documents_only_the_models_reject( + gap_key: str, + mutation: DocumentMutation, + model_error: str, +) -> None: + document = _schema_9(_with_dimension(schema_8_payload())) + mutation(document) + + assert gap_key in { + gap.key for gap in diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS + } + assert json_schema_errors(document, _generated_schema()) == [] + with pytest.raises(ValidationError, match=model_error): + diagnostics.CalibrationDiagnosticsV9.model_validate(document) + + +def test_v9_gaps_are_schema_8_gaps_plus_the_catalog_rules() -> None: + v8_keys = { + gap.key for gap in diagnostics.CALIBRATION_DIAGNOSTICS_V8_JSON_SCHEMA_GAPS + } + v9_keys = [ + gap.key for gap in diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS + ] + + assert len(v9_keys) == len(set(v9_keys)) + # Schema 8's row-hierarchy rule becomes the catalog rules. + assert set(v9_keys) == (v8_keys - {"hierarchy-identity"}) | CATALOG_GAP_KEYS + assert CATALOG_GAP_KEYS == {case.values[0] for case in GAP_CASES} + + +def _validator_qualified_names(core_schema: object) -> set[str]: + names: set[str] = set() + pending = [core_schema] + visited: set[int] = set() + while pending: + node = pending.pop() + if id(node) in visited: + continue + visited.add(id(node)) + if isinstance(node, dict): + if str(node.get("type", "")).startswith("function-"): + names.add(node["function"]["function"].__qualname__) + pending.extend(node.values()) + elif isinstance(node, (list, tuple)): + pending.extend(node) + return names + + +def test_every_v9_model_validator_is_documented_as_a_schema_gap() -> None: + model_validators = _validator_qualified_names( + diagnostics.CalibrationDiagnosticsV9.__pydantic_core_schema__ + ) + documented_validators = { + validator + for gap in diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS + for validator in gap.validators + } + + assert "CalibrationDiagnosticsV9.reconcile_counts" in model_validators + assert documented_validators == model_validators + + +def test_v9_description_lists_every_gap() -> None: + description = _generated_schema()["description"] + + assert "schema 9" in description + for gap in diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_GAPS: + assert gap.rule in description, gap.key + + +def test_committed_v9_schema_matches_the_generated_schema() -> None: + committed = diagnostics.committed_calibration_diagnostics_v9_json_schema_path() + + assert committed.read_bytes() == ( + diagnostics.render_calibration_diagnostics_v9_json_schema().encode("utf-8") + ) + assert diagnostics.load_calibration_diagnostics_v9_json_schema() == ( + diagnostics.calibration_diagnostics_v9_json_schema() + ) + diagnostics.assert_calibration_diagnostics_v9_json_schema_current() + + +def test_committed_v9_schema_lives_at_its_documented_path() -> None: + committed = diagnostics.committed_calibration_diagnostics_v9_json_schema_path() + package_readme = (_TEST_PATHS.package / "README.md").read_text(encoding="utf-8") + + assert committed.resolve() == ( + (_TEST_PATHS.repository / PUBLISHED_V9_SCHEMA_PATH).resolve() + ) + assert committed.name == diagnostics.CALIBRATION_DIAGNOSTICS_V9_JSON_SCHEMA_ID + assert f"`{PUBLISHED_V9_SCHEMA_PATH}`" in package_readme + + +def _load_generator_tool(): + path = ( + _TEST_PATHS.repository + / "tools" + / "generate_calibration_diagnostics_json_schema.py" + ) + spec = importlib.util.spec_from_file_location( + "generate_calibration_diagnostics_json_schema", path + ) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def test_generator_tool_writes_and_checks_the_committed_v9_schema( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + tool = _load_generator_tool() + committed = diagnostics.committed_calibration_diagnostics_v9_json_schema_path() + written = tmp_path / committed.name + + assert tool.main(["--check"]) == 0 + assert tool.main(["--schema-version", "9", "--out", str(written)]) == 0 + assert written.read_bytes() == committed.read_bytes() + + written.write_text("{}\n", encoding="utf-8") + assert tool.main(["--check", "--schema-version", "9", "--out", str(written)]) == 1 + assert "regenerate it with" in capsys.readouterr().err + with pytest.raises(ValueError, match="differs from the schema the models generate"): + diagnostics.assert_calibration_diagnostics_v9_json_schema_current(written) diff --git a/packages/microcosm-diagnostics/tests/engine_free/shared/test_schema.py b/packages/microcosm-diagnostics/tests/engine_free/shared/test_schema.py index 864afdf3f..56f077b25 100644 --- a/packages/microcosm-diagnostics/tests/engine_free/shared/test_schema.py +++ b/packages/microcosm-diagnostics/tests/engine_free/shared/test_schema.py @@ -8,7 +8,9 @@ from microcosm.diagnostics import ( CalibrationDiagnosticsV8, + CalibrationDiagnosticsV9, calibration_diagnostics_json_schema, + expand_calibration_diagnostics, parse_calibration_diagnostics, write_calibration_diagnostics, ) @@ -332,8 +334,10 @@ def test_writer_strict_serialization_rejects_a_non_finite_mutated_mapping( outcome = write_calibration_diagnostics(model, path) + # The writer validates the schema 9 document it builds from the model, so + # the non-finite value is refused there, before serialization. assert outcome.status == "failed" - assert outcome.error_code == "serialization_error" + assert outcome.error_code == "validation_error" assert not path.exists() @@ -380,7 +384,10 @@ def test_writer_round_trips_valid_output(tmp_path: Path) -> None: outcome = write_calibration_diagnostics(model, path) assert outcome.status == "available" - assert parse_calibration_diagnostics(json.loads(path.read_text())) == model + assert outcome.schema_version == 9 + written = parse_calibration_diagnostics(json.loads(path.read_text())) + assert isinstance(written, CalibrationDiagnosticsV9) + assert expand_calibration_diagnostics(written) == model @pytest.mark.parametrize( diff --git a/test_support/microcosm_diagnostics/json_schema.py b/test_support/microcosm_diagnostics/json_schema.py index 307dd193c..ed23497e2 100644 --- a/test_support/microcosm_diagnostics/json_schema.py +++ b/test_support/microcosm_diagnostics/json_schema.py @@ -41,8 +41,19 @@ def json_schema_errors(document: object, schema: Mapping[str, object]) -> list[s def published_json_schema_errors(document: object) -> list[str]: - """Return the errors against the committed schema-8 file consumers read.""" + """Return the errors against the committed schema file consumers read. - return json_schema_errors( - document, diagnostics.load_calibration_diagnostics_v8_json_schema() + A document declaring ``schema_version`` 9 is checked against the schema-9 + file the writer's output must satisfy; any other document against the + schema-8 file, so a schema-8 document is still checked as before. + """ + + schema_version = ( + document.get("schema_version") if isinstance(document, Mapping) else None + ) + schema = ( + diagnostics.load_calibration_diagnostics_v9_json_schema() + if schema_version == 9 + else diagnostics.load_calibration_diagnostics_v8_json_schema() ) + return json_schema_errors(document, schema) diff --git a/tools/assemble_uk_dense_release_dir.py b/tools/assemble_uk_dense_release_dir.py index 9b20ac52e..e04d03bd3 100644 --- a/tools/assemble_uk_dense_release_dir.py +++ b/tools/assemble_uk_dense_release_dir.py @@ -53,7 +53,7 @@ validate_release_dir, ) from microcosm.diagnostics import ( - CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION, + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS, parse_calibration_diagnostics, ) @@ -135,7 +135,10 @@ def _write_release_diagnostics( ) -> None: """Copy current diagnostics exactly; adapt only legacy candidate evidence.""" - if diagnostics.get("schema_version") == CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: + # Schema 8 and schema 9 (labels in catalogs) are both current documents. + if diagnostics.get("schema_version") in ( + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS + ): try: parse_calibration_diagnostics(diagnostics) except ValidationError as error: diff --git a/tools/generate_calibration_diagnostics_json_schema.py b/tools/generate_calibration_diagnostics_json_schema.py index cd44c9ef7..25317f291 100644 --- a/tools/generate_calibration_diagnostics_json_schema.py +++ b/tools/generate_calibration_diagnostics_json_schema.py @@ -1,20 +1,57 @@ -"""Regenerate or check the committed calibration-diagnostics schema-8 JSON Schema.""" +"""Regenerate or check the committed calibration-diagnostics JSON Schemas. + +Without ``--schema-version`` the tool writes or checks both committed files: +schema 9, which current builds publish, and schema 8, for documents published +before it. ``--out`` writes or checks one schema at another path and needs +``--schema-version``. +""" from __future__ import annotations import argparse import sys +from collections.abc import Callable from pathlib import Path from microcosm.diagnostics import ( assert_calibration_diagnostics_v8_json_schema_current, + assert_calibration_diagnostics_v9_json_schema_current, committed_calibration_diagnostics_v8_json_schema_path, + committed_calibration_diagnostics_v9_json_schema_path, write_calibration_diagnostics_v8_json_schema, + write_calibration_diagnostics_v9_json_schema, ) +#: For each schema version: its committed path, its check and its writer. +_SCHEMAS: dict[ + int, + tuple[ + Callable[[], Path], + Callable[[str | Path | None], None], + Callable[[str | Path | None], Path], + ], +] = { + 9: ( + committed_calibration_diagnostics_v9_json_schema_path, + assert_calibration_diagnostics_v9_json_schema_current, + write_calibration_diagnostics_v9_json_schema, + ), + 8: ( + committed_calibration_diagnostics_v8_json_schema_path, + assert_calibration_diagnostics_v8_json_schema_current, + write_calibration_diagnostics_v8_json_schema, + ), +} + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--schema-version", + type=int, + choices=sorted(_SCHEMAS), + help="the diagnostics schema to write or check (default: both)", + ) parser.add_argument( "--out", type=Path, @@ -26,17 +63,24 @@ def main(argv: list[str] | None = None) -> int: help="exit 1 unless the file equals freshly generated output", ) args = parser.parse_args(argv) - path = args.out or committed_calibration_diagnostics_v8_json_schema_path() - if args.check: - try: - assert_calibration_diagnostics_v8_json_schema_current(path) - except (OSError, ValueError) as error: - print(f"stale: {error}", file=sys.stderr) - return 1 - print(f"current: {path}") - return 0 - print(write_calibration_diagnostics_v8_json_schema(path)) - return 0 + if args.out is not None and args.schema_version is None: + parser.error("--out needs --schema-version") + versions = [args.schema_version] if args.schema_version else list(_SCHEMAS) + exit_code = 0 + for version in versions: + committed_path, assert_current, write = _SCHEMAS[version] + schema_file = args.out or committed_path() + if args.check: + try: + assert_current(schema_file) + except (OSError, ValueError) as error: + print(f"stale: {error}", file=sys.stderr) + exit_code = 1 + continue + print(f"current: {schema_file}") + continue + print(write(schema_file)) + return exit_code if __name__ == "__main__": diff --git a/tools/score_uk_local_candidate.py b/tools/score_uk_local_candidate.py index 136c88e8d..ed18038a6 100644 --- a/tools/score_uk_local_candidate.py +++ b/tools/score_uk_local_candidate.py @@ -34,6 +34,7 @@ default_target_loss_scales, relative_error_loss, ) +from microcosm.diagnostics import HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS #: ``statistical_scope`` (the rowwise ``brma`` grain) is Scotland's PIPR BRMA #: rows, which the incumbent never fitted: they score candidate-only, as @@ -104,9 +105,13 @@ def _candidate_estimates( A withheld row (microcosm#1262) carries the note in its metadata and no final estimate; it is neither scored nor missing. """ - if diagnostics.get("schema_version") != CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION: + # Schema 9 moved only the hierarchy labels into catalogs; the names, + # metadata and final estimates read here are on every row in 8 and 9. + if diagnostics.get("schema_version") not in ( + HIERARCHY_CALIBRATION_DIAGNOSTICS_SCHEMA_VERSIONS + ): raise ValueError( - "UK local scoring requires calibration diagnostics schema " + "UK local scoring requires calibration diagnostics schema 8 or " f"{CALIBRATION_DIAGNOSTICS_SCHEMA_VERSION}." ) rows = diagnostics.get("targets")