diff --git a/README.md b/README.md index d8b49ac01f..273abd388b 100644 --- a/README.md +++ b/README.md @@ -356,6 +356,10 @@ including Hypothesis properties) run in the normal `uv run pytest` suite. ## Releasing & alerts +[The release contract](docs/release-contract.md) (C1 to C5) says what the +publisher enforces today and what version 1, which the ARCH-1 plan adds, asks +of every release. + The [native SPM role source-enrichment lane](docs/us-native-spm-role-source-enrichment.md) creates a new US H5 from the exact reviewed BuildP parent, preserves its original variables and schema-5 calibration evidence, and requires fresh country/wrapper diff --git a/changelog.d/1285-release-contract.added.md b/changelog.d/1285-release-contract.added.md new file mode 100644 index 0000000000..c97c172264 --- /dev/null +++ b/changelog.d/1285-release-contract.added.md @@ -0,0 +1,3 @@ +`docs/release-contract.md` puts the release contract on paper (ARCH-1, cos d1160): the end state, the releases in service on 2026-10-15, and contracts C1 to C5 (release manifest and certification, loader compatibility, staging and run outcome, the Logbook, publish and registry), each as the code enforces it today and as the plan's version 1 would, with how far each milestone had got on 2026-11-03. It says how C1 version 1 keys by declared country instead of release-id strings, records where the recertification lane and C2 version 1 disagree, carries microcosm#378's parity rule, and replaces the branch-only `us-uk-release-path.md`. It changes nothing that publishes. + +`tools/generate_release_contract_fixtures.py` writes version 1 of the contract's golden fixtures under `packages/microcosm-data/tests/fixtures/release_contract/v1`, following the staging-fixture pattern: the UK national line's manifests and an inventory of what the contract requires of that line (C1), the `latest.json` and `latest-national.json` payloads and a registry snapshot (C5), with `SHA256SUMS` and `--check`. The UK national assembler's manifest construction moves into `uk_national_release_manifests`, a pure function the generator also calls; its output is unchanged. diff --git a/changelog.d/1285-uk-exact-k-id-minting.removed.md b/changelog.d/1285-uk-exact-k-id-minting.removed.md new file mode 100644 index 0000000000..df7c0d7f18 --- /dev/null +++ b/changelog.d/1285-uk-exact-k-id-minting.removed.md @@ -0,0 +1 @@ +Retire `format_uk_release_id`, `UKReleaseIdentity` and `apply_uk_release_identity` from `microcosm.build.uk_runtime`: they minted `populace-uk---k` ids and had no caller. Version 1 of the release contract (`docs/release-contract.md`, C1, planned for M3) derives a new release's id from its declared identity instead. The UK line ids and the tier vocabulary stay, and the release contract still validates existing exact-k ids. diff --git a/docs/agent-guide.md b/docs/agent-guide.md index 49121f3a5f..61eda737f0 100644 --- a/docs/agent-guide.md +++ b/docs/agent-guide.md @@ -218,6 +218,12 @@ HF_STAGING_READ_TOKEN= bash tools/run_integration_tests.sh ## The PR-CI / certification boundary +[The release contract](release-contract.md) (C1 to C5) states what the +publisher enforces today and what version 1, which the ARCH-1 plan adds, asks +of every country's releases. Change it, its golden fixtures +(`tools/generate_release_contract_fixtures.py`) and the code that enforces it in +the same pull request. + PR CI never receives external writer credentials or touches restricted microdata. Same-repository runs may receive the optional read-only `HF_STAGING_READ_TOKEN` in the `integration-uk` job; fork pull requests receive diff --git a/docs/release-contract.md b/docs/release-contract.md new file mode 100644 index 0000000000..3999ee2c10 --- /dev/null +++ b/docs/release-contract.md @@ -0,0 +1,655 @@ +# The release contract + +What a Microcosm release must be before anyone may serve it, for every +country. Max ruled ARCH-1 yes on 2026-10-15 (cos d1160), adopting the end state +below, in which publishing refuses non-conforming releases from 2026-11-06. +This document states that end state and the five contracts, C1 to C5: each as +the code enforces it today, and each one's version 1, the requirement the +2026-10-09 plan adds. + +Nothing here refuses anything. This document, its golden fixtures and their +tests describe and freeze; they do not change what publishes. Each "today" +statement below comes from reading the code at `origin/main` 4a36edcf3 +(2026-11-03), or from executing against it, and cites the function that does the +work; statements about policyengine.py and the Hub carry the date of their +reading. Each version-1 statement is a requirement the code does not enforce on +2026-11-03. Beside each, this document names the milestone the plan assigns it +to, and the [enforcement timeline](#enforcement-timeline) records how far each +milestone had got by that date. When refusal starts is Max's call. + +This document replaces `docs/us-uk-release-path.md`, a planning snapshot that +never reached main. Its last version (363a9033b, 15 September) lives on the +graph-native branches, which cos d1161 closed on 2026-10-15. Two of +its rules carry over: record every reference period explicitly instead of +inferring it from a filename (C1's declared identity below), and make every +dashboard and release verdict refer to the same candidate bytes (C1's digest +bindings). + +## The end state + +- **One producer per country.** Each country builds one population on + `run_graph` and derives every published size from it by selection on one + target register. +- **One release contract.** Every release passes C1 to C5, frozen at the shape + the UK national line already ships. +- **One registry.** `microcosm.data.registry` names every file any consumer + serves, and policyengine.py's bundle mirrors it. +- **Publishing refuses** any release that does not conform (from 2026-11-06, in + the ruling). +- **Retired:** the incumbent UK data package (uk-data), the Build P patch lanes, the second US + calibration driver and staging version 1. + +The plan that gets there, with owners and dates, is the 2026-10-09 systems +report that cos d1160 cites (Max's planning record, not in this repository; +milestones M0 to M7). This document is M0's deliverable. +[Enforcement timeline](#enforcement-timeline) lists the milestones that change +what the contract enforces. + +## Releases in service on 2026-10-15 + +Read on 2026-10-15 from policyengine.py's bundle 6.4.3 +(`src/policyengine/data/bundle/manifest.json` at d74ddc713), the public +`policyengine/populace-us` Hub repository and `microcosm.data.registry`. + +| Grain | Served release | Producer | Meets the contract? | +| --- | --- | --- | --- | +| US national (policyengine.py and `latest.json`) | `populace-us-2024-spm-20260915` | A source enrichment of a parent release (its manifest declares `release_type: source_enrichment` and ships the parent's manifests); the 10-09 report traces the parent to Build P | No: a separate source-enrichment contract, no certification, no Logbook row | +| US state and district (policyengine.py `regional_release_manifest_uri`) | `populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z` | `tools/build_us_acs_local_release.py` | No: not in the registry, no staging, no Logbook row | +| US candidate (tag only) | `populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z` (Route A) | `tools/route_a/` then the fiscal-refresh tool | No: 16 gates in an ad hoc `gates` dict, staging `{"enabled": false, "reason": "--no-staging"}`, no certification | +| UK, PolicyEngine app | uk-data 1.58.1, `enhanced_frs_2024_25` | the incumbent UK data package (uk-data) | Not a Microcosm release; certified for policyengine-uk 2.130.1 on the `legacy_compatible_model_package` basis | +| UK, `microcosm.data.resolve("uk")` | `populace_uk_2023.h5` (the June 2023 file) | none current | Grandfathered id | +| UK national line | `latest-national.json` names `microcosm-uk-2024-25-national` at a per-cut tag | `microcosm-build-uk` on `run_graph`, then `tools/certify_uk_release_cut.py` and `tools/assemble_uk_release_dir.py` | The assembler refuses to write a cut that fails today's C1, and line promotion runs C5's guards. It lacks what no producer writes yet: C1's declared identity, C3's run-outcome binding and C4's Logbook binding | + +So three producers served three datasets, and microcosm's own registry pointed +the UK default at a fourth file. + +This document has not re-read the table since 2026-10-15, except for one change +on main: on 2026-11-02 microcosm#1432 made the UK 2024-25 national line the UK +2025 default load. Its +registry entry is now `("uk", 2025, "compact")`, followed through +`latest-national.json`, so `microcosm.data.resolve("uk")` resolves to the +national line, and the June 2023 file stays at `("uk", 2023, "compact")`. + +## The five contracts + +| | Contract | Freezes | Enforced today by | Golden fixture | +| --- | --- | --- | --- | --- | +| C1 | Release manifest and certification | The release directory's files, both manifests, and a certification composed from gate-battery reports | `validate_release_dir` (`microcosm/data/contract.py`) | `v1/c1/` | +| C2 | Loader compatibility | One rule admitting a model and Core version against a release | `microcosm.data.loader` and policyengine.py, separately | none | +| C3 | Staging telemetry and run outcome | How a build records and delivers its end | `publish_cli._staging_undelivered` | the staging fixtures | +| C4 | Logbook | One durable row per terminal build attempt | nothing at publication | none | +| C5 | Publish and registry | Pointers, the id grammar, and the registry | `prepare_release` / `publish_release` (`microcosm/data/release.py`) and `microcosm.data.registry` | `v1/c5/` | + +### C1: the release manifest and certification + +#### Today + +`validate_release_dir(release_dir)` is the one validator every publish path runs +before uploading anything (`prepare_release` calls it, or its evidence-tier +sibling). It dispatches in three steps: + +1. **Release type.** `release_manifest.json`'s `release_type`. Two types hand + off to separate contracts and return. `source_enrichment` goes to + `validate_source_enrichment_candidate`, the Build P inheritance path; the + served US default on 2026-10-15 was one of these. + `compatibility_recertification` goes to `validate_recertification_candidate` + (`microcosm/data/compatibility_recertification.py`, merged 2026-10-23), the + contract for a release that republishes a published US calibration + release's bytes under a newer policyengine-us. An absent `release_type`, or + `calibration`, means a calibration release; the validator refuses any other + value. When the manifest declares an annual-projection extension, + `validate_annual_projection_extension` checks it next. +2. **Dataset role.** `dataset_role` is `national_default` (also when absent) or + `non_default_local_area`. A local-area release goes to + `_validate_local_area_release_dir`, or to `_validate_uk_dense_release_dir` when + its id is the UK dense line's. +3. **The release id string.** Everything else keys on the directory name: + - `required_release_files(release_id)` adds `us_source_coverage.json` for a + `populace-us-` prefix, `terminal_gates.json` for a UK exact-k id + (`populace-uk---k`), and `release_certification.json` for the + literal `microcosm-uk-2024-25-national`. Any other id gets the three base + files: `build_manifest.json`, `release_manifest.json` and + `calibration_diagnostics.json`. Whatever the id, the validator skips + `calibration_diagnostics.json` when the release manifest records failed + diagnostics generation (`calibration_diagnostics.status: failed`, read by + `_calibration_diagnostics_required`), and the pointer payloads then omit its + path. + - `_expected_model_package(release_id)` names `policyengine-us` or + `policyengine-uk` from the same prefixes. For any other id it returns + `None`. The checks then skip the build manifest's model-package runtime + pin and the `compatible_model_packages` list entirely, and they no longer + check which package `build.built_with_model_package` names (it still needs + a non-empty name and a resolved version). + - The US critical-target fit, stored-input evidence and source-coverage + checks fire on the `populace-us-` prefix; the UK identity checks fire on + `populace-uk-` and the exact-k pattern. + +Whatever the id, a `calibration` release in the national-default role must carry: + +- **`build_manifest.json`**: `build_id` equal to the directory name; `code` + with `repository`, a 40-character `git_commit` and `git_dirty: false`; + `runtime` with resolved `python`, `policyengine-core` and, when the id names + a model package, that package's versions; `dataset` and `calibration` with filenames and + SHA-256s, the calibration naming its target surface and registry; and a + `gates` object (`_check_build_manifest`). +- **`release_manifest.json`**: `schema_version` 1 + (`RELEASE_MANIFEST_SCHEMA_VERSION`); `build.build_id` equal to the directory + name; `build.built_with_core_package` and `build.built_with_model_package`; + a non-empty `compatible_core_packages` list that includes the built Core + version and, when the id names a model package, a non-empty + `compatible_model_packages` list that includes the built model version; + `data_package`; `default_datasets.national` + naming a `microdata` artifact; and `artifacts`, each with `kind`, `path`, + `repo_id`, `revision` and `sha256`, all pinned to one revision + (`_check_release_manifest`). + +The **certification** is UK-only. For `microcosm-uk-2024-25-national`, +`release_certification.json` is a required file. The validator refuses any +directory that carries a `release_cut_gates.json`, or a `terminal_gates.json` +whose `posture` is `calibration_seam`, without one, whatever its id. +`_check_uk_release_certification` then requires `kind: +uk_release_certification`, `country: uk`, the release id, and exactly three +parts: `spine`, `calibration_seam` and `release_cut`. Each part's scoped spec +digests must match the pins in `contract.py`, and every gate in its scope must +have passed: its `statuses` must equal `{"passed": }`. The +document must also say `shippable: true` and carry, in `spec`, the full gate +spec's digests, which must match the committed full-manifest pins. One +HMAC-SHA256 attestation covers the whole document, under the key in +`MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY`; no part carries its own signature. +`_check_uk_certification_evidence_binding` binds each part's report file in the +release directory (`spine_gates.json`, `terminal_gates.json`, +`release_cut_gates.json`) and the score receipt to the signed digests. On the +national line, the certification is the only place a shippability verdict +lives. + +#### The id-string keying, and how C1 v1 keys by declared country instead + +Every country-specific check hangs off the release id. Two consequences follow, +both executed in `test_release_contract_keying.py`: + +- **A new country fails open.** An NZ, BE or AM release, or a future + `microcosm-us-*` one, gets only the three base files and no model-package + expectation: no certification, no source coverage, no + `compatible_model_packages` check. It validates against the weakest contract + by default. +- **No other country can promote a line.** `line_for_release_id` knows only + `microcosm-uk-2024-25-national` and `microcosm-uk-2024-25-local-k`. The + publisher's `--promote-line` and the loader's line reader both call it, so no + other release can reach a `latest-.json` pointer. + +The registry also disagrees with the id about the year. The registry files the +UK national line under `("uk", 2025, "compact")`, the UK 2025 default load since +microcosm#1432 (2026-11-02), while its id says `2024-25`. +`test_release_contract_keying.py` pins the mismatch. Until microcosm#1432, the +live-loader test in `test_loader.py` required every resolved release id to start +with `microcosm---` (or the `populace-` form), which the national +line's id cannot match; microcosm#1144 reported the failure from a live run. It +now asks only for `microcosm--` of a release reached through a line +pointer. + +#### Version 1, planned for M3: dispatch on the declared country + +C1 v1 dispatches on what the manifest declares instead: + +1. **Release manifest schema 2** adds a declared identity, required on every new + release: + + ```json + { + "schema_version": 2, + "country": "uk", + "vintage": "2024-25", + "line": "national", + "dataset_role": "national_default", + "producer": {"tool": "tools/assemble_uk_release_dir.py", "commit": "<40-hex>"} + } + ``` + + `dataset_role` stops being optional. `producer` closes the gap that left Build + P's producer an inference: a manifest has no tool field today. +2. **One country table in the data shard** replaces the prefix tests. Each + registered country declares its model package, the files its releases add to + the C1 base set (the US adds `us_source_coverage.json` and the stored-input + evidence; every country adds its certification part reports), its + country-specific checks (the US critical-target fit, stored-input, + source-coverage and split-microdata checks; the UK identity, diagnostics and + gate-report checks), the lines it may publish, and its Hub repository. Every + check that keys on an id string today reads the table through the declared + country instead: `required_release_files`, `_expected_model_package` and + `line_for_release_id`, and also each `startswith("populace-us-")`, + exact-k and constant-id test in `contract.py`, `release.py` and `loader.py` + (about 35 sites). A schema-2 US release then cannot skip the US checks by + choosing another id. +3. **Unknown countries fail closed.** The validator refuses them instead of + falling back to the base set. The plan registers NZ before the switch, + because cos d962's late-November deliverable depends on it. +4. **The validator derives the id instead of parsing it.** A schema-2 release + id must equal `microcosm---` rendered from the + declared fields, and + the directory name must equal the id. Nothing reads meaning out of the id + string any more. The pointer readers validate a line before they fetch the + manifest, so schema-2 pointers also carry `country`, `vintage` and `line`. + The reader checks them against the registry entry, then against the manifest + it fetches. +5. **One vintage string.** The registry entry carries the vintage string, + and the loader compares the manifest's `vintage` with it exactly, which + removes the `2025` versus `2024-25` mismatch. +6. **Grandfathering is a frozen list of published revisions.** Schema-1 + manifests keep today's id-keyed dispatch only for the (release id, revision) + pairs published before the switch: the `populace-*` release ids and the + per-cut tags of the UK national and dense lines. The list cannot hold release + ids alone, because the UK lines keep one constant id across cuts, so an + id-only list would grandfather every future cut. The publisher refuses a new + schema-1 cut. + +#### The frozen shape + +C1 freezes what the UK national line ships today. A conforming national or local +release directory carries: + +- `build_manifest.json` and `release_manifest.json` in the shape of + [`v1/c1/microcosm-uk-2024-25-national/`](#golden-fixtures), which + `uk_national_release_manifests` in `tools/assemble_uk_release_dir.py` writes; +- `calibration_diagnostics.json`; +- `release_certification.json`, which the gate battery composes from the + country's part reports, with every part's report file beside it, bound by + digest, and the score receipt that decided shippability. + +The 10-09 report's plan brings the US to this shape by adapter at M2: the +fiscal-refresh tool declares its 16 gates in `us/gates.json`, projects its own +verdicts through the gate battery under `BlockingMode.MARKS_ARTIFACT`, and +composes the certification from the projected reports. It projects; it does not +re-evaluate. Neither `us/gates.json` nor that adapter exists on 2026-11-03, and +no US release conforms, so there is deliberately no US C1 golden. +`test_support/microcosm_data/release.py` and the fiscal-refresh builder tests +pin the US national shape the publisher accepts today. + +### C2: loader compatibility + +#### Today: two rules + +**microcosm.data's loader.** `_package_certification` reads +`compatible_model_packages` and `compatible_core_packages` from the release +manifest. Each must hold at least one entry for the package with a valid +PEP 440 specifier, and the built version must satisfy one of them. At load, +`_enforce_package_compatibility` refuses unless the installed model package and +the installed `policyengine-core` each satisfy one of their specifiers. + +**The release validator.** For `calibration` releases, +`_check_compatible_package_entries` checks the same lists at publication. An +entry may declare `basis: publisher_claim` with a `declared_by` naming who made +the claim; an entry without a basis records what the build measured. It does +not require an entry wider than the built version to declare that basis. For +`source_enrichment` releases (the served US default on 2026-10-15), +`validate_source_enrichment_candidate` is already stricter. Each list must be +the exact pin `=={version}` unless the certified report declares a publisher +claim. The validator accepts a claim only as `compatibility_claim_entry` builds +it: it names its declarer, contains the tested version, and has bounds above and +below. Core always takes an exact pin ("no producer can declare a Core range", +`source_enrichment.py`). For a `compatibility_recertification` child, +`validate_recertification_candidate` requires both lists to equal what +`claim_entries` builds: every entry of the parent, plus one model claim, built +by the same `compatibility_claim_entry`, for the policyengine-us version the +child tested, plus, when no parent entry covers the tested Core, an exact pin of +that Core with `basis: publisher_claim`. `_check_recertification_claim` narrows +the model claim to `==T`, the tested version, or to `>=L,_packages` entries for that package. There must be at least + one, and each must be a valid PEP 440 specifier. +2. **Exact pins and claims.** An entry whose specifier is exactly `==V` or + `===V`, with no wildcard, where `Version(V) == Version(built)` (PEP 440 + equality, any local label included, as in rule 4), is an exact pin. Every + other entry is a claim. It must declare `basis: publisher_claim` and a + `declared_by`, must contain the built version, and must have bounds above and + below. `compatibility_claim_entry` applies the same tests today, except that + it asks for the tested version, which on a source enrichment is the built + one. +3. **Core.** `policyengine-core` takes exact pins only. Claims apply to the model + package alone, as on the source-enrichment path. +4. **Admission.** A loader admits the installed version when + `Version(installed) == Version(built)`, PEP 440 equality with any local label + included, or when it satisfies a claim. Containment in an exact pin alone is + not enough, because `2.0.0+local` satisfies `==2.0.0`. Otherwise the loader + refuses the pair. +5. **Basis.** The basis is `built_with_model_package` for the first case and + `publisher_claim` for the second, attributed to the first matching claim in + manifest order. +6. **Legacy manifests.** Both loaders keep today's rule for a schema-1 manifest, + which every release published before C1 v1 carries, and apply rules 1 to 5 + to schema-2 manifests. Adopting C2 v1 therefore changes no load of an already + published release, including the registry's UK default and the releases + policyengine.py's bundle pins, whatever their entries say. + +Rules 2 and 3 tighten the validator for `calibration` releases, which today +accepts a wider entry without a basis and a Core range. Rule 4 tightens both +loaders, which today admit any installed version an entry contains. + +**Open: the recertification lane.** The `compatibility_recertification` lane +(microcosm#1366, merged 2026-10-23) writes two entries that rules 2 and 3, as +written, refuse: a model claim that does not contain the built version, since +the parent's own entries already cover that one, and an exact pin of the tested +Core with `basis: publisher_claim`. Before a loader adopts C2 v1, either the +rule admits both or the lane changes; the plan does not yet say which. + +Under the plan, microcosm.data's loader and policyengine.py's bundle +certification both implement this rule at M2. policyengine.py's +`legacy_compatible_model_package` label becomes `publisher_claim`, and it gains +the Core check. Owner: anth-volk. On 2026-11-03 microcosm.data's loader still +applies today's rule; `loader.py` has not changed since 2026-10-15. +policyengine.py keeps its current rule for uk-data releases only while it still +serves one. + +### C3: staging telemetry and run outcome + +#### Today + +- **The delivery record.** `build_manifest.json` may carry a `staging` block. In + version 2 it is exactly `contract_version` (2), `enabled`, `mode`, `run_id`, + `configured_repository`, `upload_attempts`, `upload_successes`, `read_back`, + `last_error_code` and `opt_out_reason`. The delivery record stays at version 2 + while the staging documents moved to version 3 (`STAGING_CONTRACT_VERSION` and + `STAGING_DOCUMENT_VERSION` in `staging_v2.py`). +- **The documents.** Staging version 3 adds a `blocked` lifecycle status beside + `running`, `completed` and `failed`. The UK producers write it through + `StagingRunBundleWriterV2`. The US fiscal-refresh tool still writes version 1 + through `microcosm.build.staging.StagingRunBundleWriter`, the only non-test + importer of version 1. The ACS-local tool writes none. Since microcosm#1410 + (merged 2026-11-02), both writers hand the run's files to the build's emitter, + which commits them to the staging repository with the build's own Hugging Face + credential. +- **The run outcome.** `microcosm.build.run_outcome` (microcosm#1147) decides how + a build ended once (`completed`, `blocked`, `failed`, `interrupted`, + `terminated`, `aborted`) and derives the staging status and the Logbook + disposition from it. The UK full build, spine build and row-wise staging use + it; no US tool does. +- **The emitter.** microcosm#1153 moved the local emitter out of this repository + into the external `microcosm-emitter` package on 2026-10-13, and Max accepted + the extraction on 2026-10-15 (cos d1162's note). Since microcosm#1410, + microcosm-build depends only on `microcosm-emitter-interface` (`>=0.7.0,<1`). + Each machine installs the `microcosm-emitter` command on its own (for example + with `uv tool install microcosm-emitter`), and the adapter, + `microcosm.build.telemetry_emitter`, starts it as the build's child process + with three modules: telemetry, Orrery graph publication and the Hugging Face + hand-over of staging run files. The adapter supplies build identity and never + starts an emitter older than 0.6.0 (`MINIMUM_EMITTER_VERSION`). It is best + effort: a missing, slow, refusing or outdated emitter leaves the build running + without it. The fiscal-refresh tool, the UK spine build and the UK row-wise + staging path start it. +- **The publish gate.** `publish_cli._staging_undelivered` refuses a release + whose manifest records staging the build intended but never delivered: an empty + block, a version-2 record in `local_only` mode or with a failed read-back, or + an enabled run with no successful upload. A disabled record with a non-empty + reason is a legal opt-out, and so is a manifest with no `staging` key. + `--allow-missing-staging` overrides the gate. The Route A candidate in the + table above carries the version-1 opt-out. + +#### Version 1 (planned) + +- A `national_default` or local-area release may not opt out of staging, and + `--allow-missing-staging` no longer applies to those roles. +- Remote staging delivery stays required, as it is today. It is the + pre-publication review: the build appears on the staging dashboard before + anyone serves it. That is the deliberate difference from telemetry and the + Logbook, whose remote halves never gate. A staging outage must not force a + rebuild, though. M3 adds a command that re-delivers the release's own staging + bundle and reads it back. The command writes its receipt, + `staging_redelivery.json`, into the release directory: the run id, the + repository, the files committed, the read-back result, and the SHA-256 of the + `progress.json` it re-delivered. It never rewrites the build manifest. When + `build_manifest.json`'s `staging` block records an undelivered run, the + receipt joins the release's C1 file set, and the publisher accepts the release + only if the receipt's read-back passed and its digest matches the shipped + `progress.json`. M3 also moves the delivery gate from `publish_cli` into + `prepare_release`, so every publish path runs it. +- The release directory ships the build's staging version-3 terminal document + (its `progress.json`), which `build_manifest.json` binds by digest and the + country's C1 file set lists. The build manifest records the `run_outcome` it + says, and the publisher refuses any outcome other than `completed`. Builds run + on Modal and other hosts while publishing runs elsewhere, so the record + travels with the release, and the publisher never reads it from the build + host. +- Telemetry delivery is never a precondition, so a collector outage cannot + block a release. +- The plan deletes staging version 1, its re-export from `microcosm.build` and + the version-1 branch of `publish_cli` at M2, once the US writes version 3 + (microcosm#933). On 2026-11-03 the fiscal-refresh tool still writes version 1. + The dashboard keeps its version-1 reader for history. + +### C4: the Logbook + +#### Today + +`microcosm.build.logbook.record_build_attempt` validates a row of 17 fields +(`LOGBOOK_ROW_FIELDS`), writes it as `.json` under the caller's +spool directory (file fsync, atomic rename, directory fsync), and only then +attempts a remote insert. Local failures raise; it returns remote failures and +never raises them. Rows chain: `row_digest` is the SHA-256 of the canonical JSON +of every non-chain field followed by `prev_row_digest`. The disposition is one of +`BUILD_DISPOSITIONS`, and `run_outcome.logbook_disposition` maps each outcome +onto one. + +The UK full build, spine build, row-wise drivers and release-cut certifier write +rows. On the US side only `tools/build_us_multispine_pool.py` writes one; +`tools/us_bundle_generation/core.py` reads `logbook/us.jsonl` and verifies a +row's digest. Neither the fiscal-refresh tool nor the ACS-local tool writes +one. `logbook/us.jsonl` holds 28 rows, the last from +2026-08-09. No release directory binds a Logbook row today: the UK national +manifests carry the attempt id and cut tag, not a row digest. + +#### Version 1 (planned) + +- Every terminal build attempt writes one row through `record_build_attempt`, + with its disposition from `logbook_disposition(run_outcome)`. +- The producer copies its terminal attempt's row into the release directory, as + `.json`, when `record_build_attempt` returns, not later from the + spool, because `reconcile_spool` deletes a row once the server acknowledges + it. `build_manifest.json` records `logbook.row_digest`. +- The publisher refuses a `national_default` or local-area release whose shipped + row is missing, does not validate as a Logbook row, or does not hash to that + digest. The publisher does not require the remote insert. +- Whether microcosm#739's Logbook family model folds into `run_outcome` is + anth-volk's call. + +### C5: publish and registry + +#### Today + +- **Order.** `prepare_release` runs the release contract (`validate_release_dir`) + and its pointer and tag guards before any upload; `publish_cli` runs + its reform-validation and staging gates before it calls `prepare_release`. + `publish_release` then uploads the release files, creates the immutable tag, + and writes the pointer last, so a reader never sees a pointer before its + release. For a compatibility recertification it first calls + `require_parent_publication`, which refuses unless the parent's tag on the + Hub serves byte for byte the parent manifests the child carries, and every + parent artifact at its pinned digest. +- **Pointers.** Each is a schema-1 JSON document at the repository root: + - `latest.json`, the repository-global pointer: `schema_version`, + `release_id`, `updated_at`, and `paths` to the contract files + (`latest_pointer_payload`). Only a `national_default` release whose tag is + its own id may move it, and never a reported-receipt source enrichment or a + compatibility recertification. + - `latest-.json`, a line pointer: the same fields plus `line` and + `revision`, where the revision may be a per-cut tag + (`line_pointer_payload`). The line must match `^[a-z]+(-k[1-9][0-9]*)?$`, and + the release's declared role must be the line's role + (`dataset_role_for_line`). + - `latest-evidence.json`, the evidence tier's pointer. The 10-09 report lists + the evidence tier for retirement (see the retire list). +- **The registry.** `microcosm.data.registry.REGISTRY` maps `(country, year, + variant)` to a `DatasetSpec` naming the Hub repository, file, engine class and + pointer path. The loader follows registry, then pointer, then the manifest at + the pointer's revision, then the artifact, and refuses an artifact whose + repository or path differs from the registry's. It holds four entries: + `("uk", 2023, "compact")`, the June file; `("uk", 2025, "compact")`, the + national line, followed through `latest-national.json` and the UK 2025 + default load since microcosm#1432 (2026-11-02); `("uk", 2025, "dense")`, tag + only; and `("us", 2024, "compact")`. +- **The gaps.** The US state and district file that policyengine.py served on + 2026-10-15 has no registry entry. On 2026-10-15 the UK default disagreed three + ways: the registry's default was the June file, policyengine.py served + uk-data, and the microcosm national line had its own pointer. Since + microcosm#1432 the registry's UK default follows the national line's pointer; + this document has not re-read policyengine.py's UK default since 2026-10-15. + +#### Version 1 (planned) + +- **One id grammar** for new releases: `microcosm---`, + rendered from C1's declared identity. Legacy ids stay on C1's frozen list. +- **Every line publishes through `latest-.json`**, for every country. + `latest.json` stays as the frozen pointer of the existing US national line + until that line moves to `latest-national.json`. +- **Every file any consumer serves has a registry entry**, including the US + state and district file (M2), and each entry declares its vintage. +- **policyengine.py mirrors the registry.** A CI check there asserts that every + bundle `data_releases` entry resolves through a registry entry (anth-volk, M3). +- **Publishing enforces C1 to C4.** For `national_default` and local-area + releases the publisher refuses a release without a certification, without a + shipped `run_outcome` of `completed`, or without a shipped Logbook row, and + refuses an unknown country. The UK dense line ships no + `release_certification.json`: its signed verdict is its gate report, + `uk_local_gates.json`. The plan does not yet say whether the dense line + composes a certification or whether that report counts as one. + +## The parity rule + +The strangler rule from microcosm#378 governs every move of a stage from an +incumbent implementation onto the shared path, including the M4 move of US +national onto the UK national chain and the M6 move of US source stages onto +`run_graph`: + +- **The incumbent stays authoritative** until the new stage proves parity on a + pinned fixture run. Then the stage leaves the incumbent. +- **Data, gate and export stages need byte-identical outputs.** +- **Solver stages need declared tolerance bands**, because there is a measured + nondeterminism floor (the amendment to #378, citing microcosm#307). A solve + that has proved deterministic on a platform must instead produce + bitwise-equal weights on that platform, which is the 10-09 report's M4 + acceptance for the dense solve. + +This document adds one requirement to #378's rule: the pull request that +deletes the incumbent stage records the pinned fixture, the declared tolerance +and the parity verdict. + +#378 closes when this document merges; this section supersedes it. + +## Enforcement timeline + +The 10-09 report planned these milestones; its owners are proposals. The last +column records what main holds on 2026-11-03. + +| Milestone | Planned | Contract change | Owner | On main, 2026-11-03 | +| --- | --- | --- | --- | --- | +| M0 | 10-16 | This document and the golden fixtures; nothing refuses | Max's session | This pull request, microcosm#1285 | +| M1 | 10-23 | A UK national cut on policyengine-uk ≥ 2.122.2 with staging v3, a Logbook row and a certification; the registry's UK default flips to it | juaristi22 | The registry flip, in microcosm#1432 (merged 11-02). This document does not check the promoted cut's staging, Logbook row or certification | +| M2 | 10-30 | `us/gates.json` and the US adapter (C1); the US writes staging v3 and a Logbook row and records `producer` (C3, C4); staging v1 goes; the registry gains the US state and district file (C5); one compatibility rule in both loaders (C2) | Max's session, PavelMakarchuk, anth-volk | None of it: no `us/gates.json` or US adapter, the fiscal-refresh tool still writes staging v1 and no Logbook row, the registry has no US state and district entry, and microcosm.data's loader keeps today's rule. This document does not check policyengine.py | +| M3 | 11-06 | Release manifest schema 2 and declared-country dispatch; the validator refuses unknown countries; the publisher refuses non-conforming `national_default` and local-area releases; policyengine.py's registry check | Max's session, juaristi22, anth-volk | None of it: `RELEASE_MANIFEST_SCHEMA_VERSION` is 1, and `test_release_contract_keying.py` passes on today's id keying. This document does not check policyengine.py | + +Max's rulings still decide anything that changes published numbers, such as +moving policyengine.py's UK default (UK-1). + +## Golden fixtures + +`packages/microcosm-data/tests/fixtures/release_contract/v1/` freezes C1 and C5 +in bytes, the way `packages/microcosm-build/tests/fixtures/staging/` freezes the +staging documents that calibration-diagnostics vendors and byte-checks. + +| File | Contract | Written by | +| --- | --- | --- | +| `c1/microcosm-uk-2024-25-national/build_manifest.json` | C1 | `uk_national_release_manifests`, the UK national assembler's own builder | +| `c1/microcosm-uk-2024-25-national/release_manifest.json` | C1 | the same | +| `c1/inventory.json` | C1 | the contract's constants: required files, the files the assembler ships, and the certification's kind, schema, required fields, part reports and signing-key variable | +| `c5/latest.json` | C5 | `latest_pointer_payload`, serialized as `publish_release` uploads it | +| `c5/latest-national.json` | C5 | `line_pointer_payload`, serialized the same way | +| `c5/registry.json` | C5 | `REGISTRY`, every `DatasetSpec` field | +| `SHA256SUMS` | | the generator | + +`tools/generate_release_contract_fixtures.py` writes them, and `--check` +regenerates them into a temporary directory and compares every byte. Every +digest, version, identifier and gate count in the C1 manifests and the pointers +is synthetic: each digest is the SHA-256 of `release-contract-fixture:`. +The release manifest's SPI Public Use Tape marker (microcosm#1095) names one +synthetic stage; its study number and conditions are the live constants of +`microcosm.build.uk_runtime.spi_dependency`, which a release that rests on the +tape restates. `inventory.json` and `registry.json` carry live values on +purpose, because they freeze the contract's constants and the registry. + +The fixtures carry no gate-battery report and no signed certification, on +purpose. Those embed the UK gate spec's live digests, which change with most UK +pull requests, so a byte-frozen copy would need regenerating almost daily. +`inventory.json` freezes the certification's shape instead, and +`test_assemble_green_release_dir`, which builds and validates a full signed +national release directory, keeps covering its signed validity. + +The tests: + +- `packages/microcosm-data/tests/engine_free/shared/test_release_contract_fixtures.py`: + checksums pinned; the C1 manifests pass the release contract's manifest checks; + the loader selects the national artifact from them through the registry's + national spec; the inventory equals the contract's live constants; both + pointers equal the publisher's payloads and pass the loader's pointer readers; + `registry.json` equals the live registry. +- `packages/microcosm-data/tests/engine_free/shared/test_release_contract_keying.py`: + executable statements of today's id-string keying, which C1 v1 replaces + deliberately. +- `packages/microcosm-build/tests/engine_free/shared/test_release_contract_fixture_generation.py`: + the generator reproduces the committed bytes, and a Hypothesis property holds + `uk_national_release_manifests` to the contract for every valid input. +- `test_assembled_release_has_the_release_contract_golden_shape` in + `packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py`: + a real assembly's manifests match the golden key for key and type for type, + and its directory holds exactly the inventory's files. + +**Regenerate** when the contract changes on purpose, in the pull request that +changes it and this document: + +```bash +uv run python tools/generate_release_contract_fixtures.py +``` + +Any change to the national assembler's manifests, the contract's required files +or certification fields, the pointer payloads or the registry fails `--check` +until the pull request regenerates the fixtures. That failure is the freeze +working. A consumer that vendors the fixtures (calibration-diagnostics, +policyengine.py's registry check) should pin the SHA-256 of `SHA256SUMS`, as +calibration-diagnostics does for the staging fixtures. + +## Retire list, as of 2026-10-15 + +The 10-09 report's "retire now" items, checked against GitHub and main on +2026-10-15. + +| Item | State | +| --- | --- | +| Graph-native drafts #893, #935, #945, #949, #950 | Closed 10-09 | +| #1011, #995, #996, #999, #1002 | Closed 10-15 under cos d1161; branches kept | +| Orrery: #1148 | Closed 10-12; #1144 and #1151 merged | +| #1083 (US process telemetry) | Open (PavelMakarchuk). The emitter now samples the build's process-tree CPU and memory, sends heartbeats and records runtime identity for the fiscal-refresh tool. Its failure classes exist in `run_outcome` but the US tool does not use them yet, and its `work` counters exist nowhere; both belong with microcosm#933. Noted on the PR for its author | +| #1082 (US staging always on) | Open (PavelMakarchuk). Belongs in microcosm#933 as staging version 3 for the fiscal-refresh tool. Noted on the PR for its author | +| #822, #824, #826 (Axiom and Belgium drafts) | Closed by this milestone; BE re-enters through the transport composer after BE-1 | +| #642 (US legislative-district layer) | Closed by this milestone; district products come from the one US population (M5) or #922 | +| #378 (Build K epic) | Closes when this document merges | +| #873 (typed model artifacts) | Kept open when checked: microcosm-dynamics then pinned `microcosm-frame` and `microcosm-graph` to 3ff92b0ae, a commit on #873's branch that is not on main. GitHub records #873 closed unmerged at 20:33Z the same day, as superseded; its closing comment says microcosm-dynamics#541 had moved that pin to microcosm main | +| `tools/build_uk_rowwise_dataset.py` | Kept. The report's reason, that the graph driver already takes `--candidate-clone-counts`, does not settle it: both drivers accept the flag only for a dry run. docs/uk-full-build-graph.md keeps the row-wise driver as a separate driver, and UK-3 (dense or local-k) is undecided | +| `format_uk_release_id`, `apply_uk_release_identity` | This milestone deletes them, with `UKReleaseIdentity`, which builds on `format_uk_release_id`: none had a caller outside its own tests and the `uk_runtime` re-export. The tier vocabulary and the line ids stay | +| Stale text, microcosm.institute `/calibration`, the evidence tier | Owned by a separate task (task_d2b70813) | +| microcosm-benchmarks, the calibration-diagnostics evaluation harness, policyengine-data's calibrate module | Each lives in another repository; proposed as a separate task (task_18c2af98) | +| cos d816 | Ruled 10-09 (per-country Core pins, microcosm#1143) | +| cos d928 | Parked; its 10-10 note records it resolved on main (the plan-2 exclusions renewed to 11-03). Nothing to rule | diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/__init__.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/__init__.py index c53ce33f9d..90411f338a 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/__init__.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/__init__.py @@ -427,9 +427,6 @@ UK_RELEASE_TIER_CPS_TRANSFER, UK_RELEASE_TIER_FRS, UK_RELEASE_TIERS, - UKReleaseIdentity, - apply_uk_release_identity, - format_uk_release_id, validate_uk_release_tier, ) from microcosm.build.uk_runtime.release_input_coverage import ( @@ -791,7 +788,6 @@ "UKRowwiseDatasetResult", "UKReleaseInputColumn", "UKReleaseInputCoverageManifest", - "UKReleaseIdentity", "UKEffectiveMassCoveragePolicy", "UKSPISupportResult", "UKSPIIncomeImputationResult", @@ -809,7 +805,6 @@ "VAT_LIABILITY_BANDS", "VINTAGES", "align_area_targets", - "apply_uk_release_identity", "apply_uk_source_lineage_modulus", "area_groups_from_codes", "area_support_tables", @@ -863,7 +858,6 @@ "fill_support_channel_from_source", "fit_tables", "footprint", - "format_uk_release_id", "frozen_vs_recomputed", "impute_uk_spi_income_support", "replace_uk_spi_support_tables", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_identity.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_identity.py index c2e3e04e60..786058d30c 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_identity.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_identity.py @@ -1,25 +1,24 @@ -"""Canonical source-tier and exact-cardinality identity for UK releases. +"""Canonical UK release ids and the closed source-tier vocabulary. The UK has one extra release-id axis that the US does not: the licence class -of the household spine. It is provenance, not a quality label. Keeping the -closed vocabulary and formatter together prevents a manifest writer from -minting an unratified tier or an id that disagrees with its manifest fields. +of the household spine. It is provenance, not a quality label. The national +and dense lines keep one constant id each across cuts. + +microcosm#1285 retired the minting helpers for the exact-cardinality +``populace-uk---k`` grammar, which had no caller +(docs/release-contract.md, retire list). Version 1 of the release contract, +planned for M3, derives a new release's id from C1's declared identity, +``microcosm---``. """ from __future__ import annotations -from collections.abc import Mapping -from dataclasses import dataclass - __all__ = [ "UK_DENSE_RELEASE_ID", "UK_NATIONAL_RELEASE_ID", "UK_RELEASE_TIERS", "UK_RELEASE_TIER_CPS_TRANSFER", "UK_RELEASE_TIER_FRS", - "UKReleaseIdentity", - "apply_uk_release_identity", - "format_uk_release_id", "validate_uk_release_tier", ] @@ -61,113 +60,3 @@ def validate_uk_release_tier(tier: object) -> str: "adjectives such as 'true', 'full', and 'public' are invalid." ) return tier - - -def _validate_positive_int(value: object, *, field: str) -> int: - if isinstance(value, bool) or not isinstance(value, int) or value <= 0: - raise ValueError( - f"UK release {field} must be a positive integer, got {value!r}." - ) - return value - - -def format_uk_release_id( - year: object, - tier: object, - record_count: object, -) -> str: - """Format ``populace-uk---k`` after strict validation.""" - - valid_year = _validate_positive_int(year, field="year") - valid_tier = validate_uk_release_tier(tier) - valid_record_count = _validate_positive_int(record_count, field="record_count") - return f"populace-uk-{valid_year}-{valid_tier}-k{valid_record_count}" - - -@dataclass(frozen=True) -class UKReleaseIdentity: - """The identity slice every future UK release manifest must record. - - :meth:`as_release_manifest_fields` deliberately returns the build id next - to its component fields. A manifest assembler consumes this validated - slice instead of accepting a free-form tier or a separately formatted id. - """ - - year: int - tier: str - record_count: int - - def __post_init__(self) -> None: - _validate_positive_int(self.year, field="year") - validate_uk_release_tier(self.tier) - _validate_positive_int(self.record_count, field="record_count") - - @property - def release_id(self) -> str: - """Return the canonical exact-cardinality release id.""" - - return format_uk_release_id(self.year, self.tier, self.record_count) - - def as_release_manifest_fields(self) -> dict[str, object]: - """Return identity fields, including the shared ``n_records`` seam.""" - - return { - "country": "uk", - "year": self.year, - "tier": self.tier, - "record_count": self.record_count, - "n_records": self.record_count, - "build": {"build_id": self.release_id}, - } - - -def apply_uk_release_identity( - manifest: Mapping[str, object], - identity: UKReleaseIdentity, -) -> dict[str, object]: - """Apply a validated UK identity without overwriting conflicting fields. - - Release manifests carry ``tier`` at the top level and ``build_id`` inside - the existing ``build`` object. The helper makes that schema seam - explicit, preserves all other build metadata, and refuses to relabel a - manifest whose identity was already assembled differently. - """ - - if not isinstance(manifest, Mapping): - raise TypeError("UK release manifest must be a mapping.") - if not isinstance(identity, UKReleaseIdentity): - raise TypeError("identity must be a UKReleaseIdentity.") - - expected_fields: dict[str, object] = { - "country": "uk", - "year": identity.year, - "tier": identity.tier, - "record_count": identity.record_count, - "n_records": identity.record_count, - } - for field, expected in expected_fields.items(): - if field in manifest and manifest[field] != expected: - raise ValueError( - f"UK release manifest {field!r} is {manifest[field]!r}, " - f"expected {expected!r}." - ) - - raw_build = manifest.get("build") - if raw_build is None: - build: dict[str, object] = {} - elif isinstance(raw_build, Mapping): - build = dict(raw_build) - else: - raise ValueError("UK release manifest 'build' must be an object.") - if "build_id" in build and build["build_id"] != identity.release_id: - raise ValueError( - "UK release manifest 'build.build_id' is " - f"{build['build_id']!r}, expected {identity.release_id!r}." - ) - build["build_id"] = identity.release_id - - return { - **dict(manifest), - **expected_fields, - "build": build, - } diff --git a/packages/microcosm-build/tests/engine_free/shared/test_release_contract_fixture_generation.py b/packages/microcosm-build/tests/engine_free/shared/test_release_contract_fixture_generation.py new file mode 100644 index 0000000000..3bd88c7e8b --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/shared/test_release_contract_fixture_generation.py @@ -0,0 +1,224 @@ +"""The release-contract fixture generator and the manifests it freezes. + +``tools/generate_release_contract_fixtures.py`` writes the C1 and C5 golden +fixtures (docs/release-contract.md) by calling the UK national assembler's own +manifest builder. The data shard's ``test_release_contract_fixtures.py`` checks +the committed bytes; this module checks that the generator reproduces them and +that the builder's output satisfies the release contract for every input, not +only the fixture's. +""" + +from __future__ import annotations + +import importlib.util +import json +import subprocess +import sys +from types import SimpleNamespace + +from hypothesis import given, settings +from hypothesis import strategies as st + +from microcosm.build.staging_v2 import validate_staging_delivery +from microcosm.build.uk_runtime.spi_dependency import ( + UK_SPI_PUBLIC_USE_TAPE_KEY, + uk_spi_public_use_tape_marker, +) +from microcosm.build.uk_runtime.spi_income import SPI_DONOR_UKDS_STUDY +from microcosm.data import contract, loader +from microcosm.data.registry import DEFAULT_VARIANT, REGISTRY +from test_support.paths import paths_for + +ROOT = paths_for("microcosm-build").repository + + +def _load_assembler(): + path = ROOT / "tools" / "assemble_uk_release_dir.py" + spec = importlib.util.spec_from_file_location("assemble_uk_release_dir", path) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +ASSEMBLER = _load_assembler() +#: The national line is the UK 2025 default load since microcosm#1432 (Max's +#: #823 ruling). +UK_NATIONAL_SPEC = REGISTRY[("uk", 2025, DEFAULT_VARIANT)] + + +def test_release_contract_fixtures_are_reproducible(): + subprocess.run( + [ + sys.executable, + str(ROOT / "tools" / "generate_release_contract_fixtures.py"), + "--check", + ], + cwd=ROOT, + check=True, + ) + + +_HEX = "0123456789abcdef" +_digest = st.text(_HEX, min_size=64, max_size=64) +_version = st.from_regex(r"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}", fullmatch=True) +_attempt_suffix = st.builds( + "{}{:02d}{:02d}T{:02d}{:02d}{:02d}Z-{}".format, + st.integers(2026, 2099).map(str), + st.integers(1, 12), + st.integers(1, 28), + st.integers(0, 23), + st.integers(0, 59), + st.integers(0, 59), + st.text(_HEX, min_size=8, max_size=8), +) +#: A shippable part's statuses: the certifier's census of its gates by status +#: (``_status_census``), which the contract requires to be all "passed". +_statuses = st.integers(1, 500).map(lambda count: {"passed": count}) +_staging = st.sampled_from( + [ + { + "contract_version": 2, + "enabled": True, + "mode": "local_and_remote", + "run_id": "uk-national-property", + "configured_repository": "policyengine/populace-uk-staging", + "upload_attempts": 3, + "upload_successes": 3, + "read_back": "passed", + "last_error_code": None, + "opt_out_reason": None, + }, + { + "contract_version": 2, + "enabled": False, + "mode": "disabled", + "run_id": None, + "configured_repository": None, + "upload_attempts": 0, + "upload_successes": 0, + "read_back": "not_requested", + "last_error_code": None, + "opt_out_reason": "--no-staging", + }, + ] +) +#: The SPI Public Use Tape marker of a run whose stages read the tape or not. +_spi_marker = st.sampled_from( + [ + uk_spi_public_use_tape_marker([]), + uk_spi_public_use_tape_marker( + [ + SimpleNamespace( + stage="property_stage", + artifacts=[{"ukds_study_number": SPI_DONOR_UKDS_STUDY}], + ) + ] + ), + ] +) + + +@st.composite +def _manifest_inputs(draw): + evidence_names = [name for _, _, name in ASSEMBLER._EVIDENCE] + attempt_id = ASSEMBLER._ATTEMPT_PREFIX + draw(_attempt_suffix) + return { + "measured": {name: draw(_digest) for name in ("candidate", *evidence_names)}, + "calibration_filename": f"{ASSEMBLER._DATASET_KEY}_calibration.npz", + "calibration_sha": draw(_digest), + "household_weight_year": { + "dataset_year": draw(st.integers(2020, 2040)), + "basis": draw(st.sampled_from(["dataset_year", "as_solved"])), + }, + "cut_tag": ASSEMBLER._cut_tag(attempt_id, None), + "attempt_id": attempt_id, + "runtime": {package: draw(_version) for package in ASSEMBLER._RUNTIME_PACKAGES}, + "code_pin": draw(st.text(_HEX, min_size=40, max_size=40)), + "target_surface": { + "sha256": draw(_digest), + "n_targets": draw(st.integers(1, 100_000)), + }, + "target_registry": { + "version": draw(st.text(_HEX, min_size=1, max_size=16)), + "n_specs": draw(st.integers(1, 100_000)), + }, + "spine_part": {"statuses": draw(_statuses)}, + "seam_part": {"statuses": draw(_statuses)}, + "release_cut_part": {"statuses": draw(_statuses)}, + "staging_delivery": validate_staging_delivery(draw(_staging)), + "spi_marker": draw(_spi_marker), + "created_at": "2026-01-01T00:00:00+00:00", + "publisher_labels": draw( + st.dictionaries(st.text(min_size=1), st.text(min_size=1), max_size=3) + ), + } + + +@settings(max_examples=60, deadline=None) +@given(_manifest_inputs()) +def test_national_manifests_satisfy_the_release_contract(inputs): + """For any valid inputs, the assembler's manifests pass the contract's + manifest checks, pin every artifact to the one cut tag, ship exactly the + signed evidence, restate the SPI Public Use Tape marker, and resolve + through the registry's national spec.""" + + build_manifest, release_manifest = ASSEMBLER.uk_national_release_manifests(**inputs) + release_id = contract._UK_NATIONAL_RELEASE_ID + + failures: list[str] = [] + contract._check_build_manifest(build_manifest, release_id, failures) + contract._check_release_manifest(release_manifest, release_id, failures) + contract._check_cross_manifest_consistency( + build_manifest, release_manifest, None, failures + ) + assert failures == [] + + artifacts = release_manifest["artifacts"] + assert {artifact["revision"] for artifact in artifacts.values()} == { + inputs["cut_tag"] + } + assert { + artifact["path"] + for artifact in artifacts.values() + if artifact["kind"] == "diagnostics" + } == {filename for _, filename, _ in ASSEMBLER._EVIDENCE} + assert set(contract.required_release_files(release_id)) <= { + "build_manifest.json", + "release_manifest.json", + *(filename for _, filename, _ in ASSEMBLER._EVIDENCE), + } + + certified = loader._certified_release_from_manifest( + UK_NATIONAL_SPEC, + release_id, + release_manifest, + expected_revision=inputs["cut_tag"], + ) + assert certified.artifact_sha256 == inputs["measured"]["candidate"] + assert certified.model.built_version == inputs["runtime"]["policyengine-uk"] + assert certified.core.built_version == inputs["runtime"]["policyengine-core"] + + # The release restates the SPI Public Use Tape marker: the flag on the + # microdata's own entry, the whole marker under licensed_data. + microdata = artifacts[release_manifest["default_datasets"]["national"]] + assert ( + microdata["spi_public_use_tape_dependent"] is inputs["spi_marker"]["dependent"] + ) + assert release_manifest["licensed_data"] == { + UK_SPI_PUBLIC_USE_TAPE_KEY: inputs["spi_marker"] + } + + # The manifest records each part's census exactly as the certification signed it. + assert build_manifest["gates"] == { + "spine": inputs["spine_part"]["statuses"], + "calibration_seam": inputs["seam_part"]["statuses"], + "release_cut": inputs["release_cut_part"]["statuses"], + } + + # Deterministic, key order included: the builder is a function of its inputs. + again = ASSEMBLER.uk_national_release_manifests(**inputs) + assert [json.dumps(document, indent=1) for document in again] == [ + json.dumps(document, indent=1) + for document in (build_manifest, release_manifest) + ] diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py index 68f4fc644a..87c92c42c4 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_assembler.py @@ -504,6 +504,69 @@ def test_assemble_preserves_successful_version_2_delivery( assert manifest["staging"] == delivery +_RELEASE_CONTRACT_C1 = ( + paths_for("microcosm-data").tests / "fixtures" / "release_contract" / "v1" / "c1" +) +#: Maps whose keys are data rather than schema: the provider ids of the labels. +_DATA_KEYED_PATHS = frozenset({("publisher_labels",)}) + + +def _json_shape(value: object, path: tuple[str, ...] = ()) -> object: + """A JSON document's keys and value types, without its values.""" + + if isinstance(value, dict): + if path in _DATA_KEYED_PATHS: + return {"*": sorted({_json_shape(item) for item in value.values()})} + return {key: _json_shape(item, (*path, key)) for key, item in value.items()} + if isinstance(value, list): + # A list's length is a value, not part of its shape: the golden's SPI + # Public Use Tape marker names one synthetic stage, while an assembly + # from the packaged spec names every stage that reads the tape. + shapes: list[object] = [] + for item in value: + shape = _json_shape(item, (*path, "[]")) + if shape not in shapes: + shapes.append(shape) + return shapes + return type(value).__name__ + + +def test_assembled_release_has_the_release_contract_golden_shape( + assembler_inputs, capsys +) -> None: + """The C1 golden fixture (docs/release-contract.md) is what this assembler + ships: a real assembly's manifests match it key for key and type for type, + and its directory holds exactly the inventory's files.""" + + record_path = assembler_inputs["diagnostics"].parent / "build_record.json" + record = json.loads(record_path.read_text()) + # The golden records a delivered run; a deliberate opt-out has the same keys + # but null strings, so match the delivery to compare types too. + golden_build = json.loads( + ( + _RELEASE_CONTRACT_C1 / UK_NATIONAL_RELEASE_ID / "build_manifest.json" + ).read_text() + ) + record["staging_delivery"] = golden_build["staging"] + record_path.write_text(json.dumps(record), encoding="utf-8") + + assert _load_driver_module().main(assembler_inputs["argv"]) == 0 + capsys.readouterr() + release_dir = assembler_inputs["out_dir"] / UK_NATIONAL_RELEASE_ID + + for name in ("build_manifest.json", "release_manifest.json"): + assembled = json.loads((release_dir / name).read_text()) + golden = json.loads( + (_RELEASE_CONTRACT_C1 / UK_NATIONAL_RELEASE_ID / name).read_text() + ) + assert _json_shape(assembled) == _json_shape(golden), name + inventory = json.loads((_RELEASE_CONTRACT_C1 / "inventory.json").read_text()) + assert ( + sorted(path.name for path in release_dir.iterdir()) + == inventory["release_directory_files"] + ) + + def test_assemble_refuses_candidate_sha_mismatch(assembler_inputs) -> None: assembler_inputs["candidate"].write_bytes( assembler_inputs["candidate"].read_bytes() + b"tampered" diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_identity.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_identity.py index 4055589876..f82707f247 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_identity.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_identity.py @@ -3,13 +3,11 @@ import pytest from microcosm.build import uk_runtime +from microcosm.build.uk_runtime import release_identity from microcosm.build.uk_runtime.release_identity import ( UK_RELEASE_TIER_CPS_TRANSFER, UK_RELEASE_TIER_FRS, UK_RELEASE_TIERS, - UKReleaseIdentity, - apply_uk_release_identity, - format_uk_release_id, validate_uk_release_tier, ) @@ -18,23 +16,12 @@ def test_uk_release_tiers_are_the_two_ratified_provenance_tokens() -> None: assert UK_RELEASE_TIER_FRS == "frs" assert UK_RELEASE_TIER_CPS_TRANSFER == "cps-transfer" assert UK_RELEASE_TIERS == frozenset({"frs", "cps-transfer"}) - assert uk_runtime.apply_uk_release_identity is apply_uk_release_identity + assert uk_runtime.validate_uk_release_tier is validate_uk_release_tier -@pytest.mark.parametrize( - ("tier", "expected"), - [ - ("frs", "populace-uk-2023-frs-k535080"), - ("cps-transfer", "populace-uk-2023-cps-transfer-k57240"), - ], -) -def test_uk_release_id_formatter_uses_tier_and_exact_record_count( - tier: str, - expected: str, -) -> None: - record_count = 535_080 if tier == "frs" else 57_240 - - assert format_uk_release_id(2023, tier, record_count) == expected +@pytest.mark.parametrize("tier", ["frs", "cps-transfer"]) +def test_uk_release_tier_accepts_the_ratified_tokens(tier: str) -> None: + assert validate_uk_release_tier(tier) == tier @pytest.mark.parametrize( @@ -58,95 +45,13 @@ def test_uk_release_tier_rejects_quality_adjectives_and_unknown_values( @pytest.mark.parametrize( - ("year", "record_count", "match"), - [ - (0, 1, "year"), - (-2023, 1, "year"), - (True, 1, "year"), - ("2023", 1, "year"), - (2023, 0, "record_count"), - (2023, -1, "record_count"), - (2023, True, "record_count"), - (2023, 1.5, "record_count"), - ], -) -def test_uk_release_id_formatter_rejects_non_positive_or_non_integer_axes( - year: object, - record_count: object, - match: str, -) -> None: - with pytest.raises(ValueError, match=match): - format_uk_release_id(year, "frs", record_count) - - -def test_uk_release_identity_is_the_manifest_assembly_surface() -> None: - identity = UKReleaseIdentity( - year=2023, - tier=UK_RELEASE_TIER_FRS, - record_count=535_080, - ) - - assert identity.release_id == "populace-uk-2023-frs-k535080" - assert identity.as_release_manifest_fields() == { - "country": "uk", - "year": 2023, - "tier": "frs", - "record_count": 535_080, - "n_records": 535_080, - "build": {"build_id": "populace-uk-2023-frs-k535080"}, - } - - -def test_uk_release_identity_validates_before_manifest_assembly() -> None: - with pytest.raises(ValueError, match="spine source licence class"): - UKReleaseIdentity(year=2023, tier="public", record_count=535_080) - - -def test_apply_uk_release_identity_preserves_existing_build_metadata() -> None: - identity = UKReleaseIdentity(year=2023, tier="frs", record_count=535_080) - manifest = { - "schema_version": 1, - "build": { - "built_with_core_package": { - "name": "policyengine-core", - "version": "3.19.0", - } - }, - } - - assembled = apply_uk_release_identity(manifest, identity) - - assert assembled["tier"] == "frs" - assert assembled["build"] == { - "build_id": "populace-uk-2023-frs-k535080", - "built_with_core_package": { - "name": "policyengine-core", - "version": "3.19.0", - }, - } - assert manifest == { - "schema_version": 1, - "build": { - "built_with_core_package": { - "name": "policyengine-core", - "version": "3.19.0", - } - }, - } - - -@pytest.mark.parametrize( - "manifest", - [ - {"tier": "cps-transfer"}, - {"n_records": 2}, - {"build": {"build_id": "populace-uk-2023-cps-transfer-k535080"}}, - ], + "retired", + ["UKReleaseIdentity", "apply_uk_release_identity", "format_uk_release_id"], ) -def test_apply_uk_release_identity_rejects_conflicting_identity( - manifest: dict[str, object], -) -> None: - identity = UKReleaseIdentity(year=2023, tier="frs", record_count=535_080) +def test_exact_k_id_minting_is_retired(retired: str) -> None: + """The exact-k minting helpers are gone: release contract version 1 + derives a new release's id from C1's declared identity + (docs/release-contract.md).""" - with pytest.raises(ValueError, match="tier|n_records|build.build_id"): - apply_uk_release_identity(manifest, identity) + assert not hasattr(release_identity, retired) + assert not hasattr(uk_runtime, retired) diff --git a/packages/microcosm-data/tests/engine_free/shared/test_release_contract_fixtures.py b/packages/microcosm-data/tests/engine_free/shared/test_release_contract_fixtures.py new file mode 100644 index 0000000000..7ff945b170 --- /dev/null +++ b/packages/microcosm-data/tests/engine_free/shared/test_release_contract_fixtures.py @@ -0,0 +1,253 @@ +"""The release contract's version 1 golden fixtures (docs/release-contract.md). + +The fixtures freeze C1 (the UK national line's manifests, and the contract's +inventory for that line) and C5 (the publisher's two pointer payloads and the +dataset registry). These tests hold every byte to ``SHA256SUMS`` and prove each +file is what the contract's own code accepts or writes. The build shard's +``test_release_contract_fixture_generation.py`` proves the generator reproduces +them. + +A failure here after a deliberate contract change means: regenerate with +``uv run python tools/generate_release_contract_fixtures.py`` and update +docs/release-contract.md in the same pull request. +""" + +from __future__ import annotations + +import dataclasses +import hashlib +import json +import shutil +from pathlib import Path + +import pytest + +from microcosm.data import contract, loader, release +from microcosm.data.contract import ReleaseContractError, validate_release_dir +from microcosm.data.registry import DEFAULT_VARIANT, REGISTRY +from test_support.paths import paths_for + +FIXTURES = paths_for("microcosm-data").tests / "fixtures" / "release_contract" / "v1" +UK_RELEASE_ID = contract._UK_NATIONAL_RELEASE_ID +UK_RELEASE_DIR = FIXTURES / "c1" / UK_RELEASE_ID +#: The national line is the UK 2025 default load since microcosm#1432 (Max's +#: #823 ruling). +UK_NATIONAL_SPEC = REGISTRY[("uk", 2025, DEFAULT_VARIANT)] +REGENERATE = ( + "regenerate with `uv run python tools/generate_release_contract_fixtures.py` " + "and update docs/release-contract.md" +) + + +def _json(path: Path) -> dict: + return json.loads(path.read_text()) + + +class _PointerHub: + """The one Hub call the pointer readers make, served from the fixtures.""" + + def __init__(self, path: Path) -> None: + self.path = path + self.requests: list[tuple[str, str, str]] = [] + + def hf_hub_download(self, *, repo_id: str, filename: str, repo_type: str): + self.requests.append((repo_id, filename, repo_type)) + assert filename == self.path.name + return str(self.path) + + +def test_every_fixture_byte_is_pinned_by_sha256sums(): + listed = {} + for line in (FIXTURES / "SHA256SUMS").read_text().splitlines(): + expected, relative_path = line.split(" ", 1) + listed[relative_path] = expected + on_disk = { + path.relative_to(FIXTURES).as_posix() + for path in FIXTURES.rglob("*") + if path.is_file() and path.name != "SHA256SUMS" + } + assert set(listed) == on_disk + for relative_path, expected in listed.items(): + observed = hashlib.sha256((FIXTURES / relative_path).read_bytes()).hexdigest() + assert observed == expected, f"{relative_path} changed; {REGENERATE}" + + +# --- C1 ----------------------------------------------------------------------- + + +def test_uk_national_manifests_pass_every_check_but_the_omitted_files(tmp_path): + """Every check that can run without the diagnostics, the certification and + the four evidence files passes: the only refusals are the two required files + the fixture omits on purpose. Those bind live gate digests + (docs/release-contract.md, Golden fixtures); the assembler's own tests + validate a full signed directory.""" + + release_dir = tmp_path / UK_RELEASE_ID + shutil.copytree(UK_RELEASE_DIR, release_dir) + with pytest.raises(ReleaseContractError) as refusal: + validate_release_dir(release_dir) + assert refusal.value.failures == [ + "required file 'calibration_diagnostics.json' is missing.", + "required file 'release_certification.json' is missing.", + ] + + +def test_uk_national_manifests_pass_the_manifest_checks_directly(): + failures: list[str] = [] + build_manifest = _json(UK_RELEASE_DIR / "build_manifest.json") + release_manifest = _json(UK_RELEASE_DIR / "release_manifest.json") + contract._check_build_manifest(build_manifest, UK_RELEASE_ID, failures) + contract._check_release_manifest(release_manifest, UK_RELEASE_ID, failures) + contract._check_cross_manifest_consistency( + build_manifest, release_manifest, None, failures + ) + assert failures == [] + + +def test_uk_national_gates_are_the_certifications_part_censuses(): + """The build manifest copies each certification part's ``statuses``, which + the contract requires to be a census of that part's gates, all passed.""" + + gates = _json(UK_RELEASE_DIR / "build_manifest.json")["gates"] + assert set(gates) == set(contract._UK_CERTIFICATION_PART_EVIDENCE_FILES) + for part, census in gates.items(): + assert list(census) == ["passed"], part + assert isinstance(census["passed"], int) and census["passed"] > 0, part + + +def test_loader_selects_the_national_artifact_through_the_registry(): + release_manifest = _json(UK_RELEASE_DIR / "release_manifest.json") + cut_tag = release_manifest["build"]["cut_tag"] + certified = loader._certified_release_from_manifest( + UK_NATIONAL_SPEC, UK_RELEASE_ID, release_manifest, expected_revision=cut_tag + ) + assert certified.artifact_repo_id == UK_NATIONAL_SPEC.hf_repo + assert certified.artifact_path == UK_NATIONAL_SPEC.filename + assert certified.artifact_revision == cut_tag + built = release_manifest["build"] + assert certified.model.name == UK_NATIONAL_SPEC.engine_package + assert certified.model.specifiers == ( + f"=={built['built_with_model_package']['version']}", + ) + assert certified.core.specifiers == ( + f"=={built['built_with_core_package']['version']}", + ) + + +def test_inventory_matches_the_live_contract(): + inventory = _json(FIXTURES / "c1" / "inventory.json") + certification = inventory["certification"] + assert inventory["release_id"] == UK_RELEASE_ID + assert inventory["line"] == contract.line_for_release_id(UK_RELEASE_ID) + assert inventory["dataset_role"] == contract.dataset_role_for_line( + inventory["line"] + ) + assert ( + inventory["release_manifest_schema_version"] + == contract.RELEASE_MANIFEST_SCHEMA_VERSION + ) + assert inventory["required_release_files"] == list( + contract.required_release_files(UK_RELEASE_ID) + ), REGENERATE + assert certification == { + "file": contract._UK_RELEASE_CERTIFICATION_FILE, + "schema_version": contract._UK_RELEASE_CERTIFICATION_SCHEMA_VERSION, + "kind": contract._UK_RELEASE_CERTIFICATION_KIND, + "required_fields": sorted(contract._UK_CERTIFICATION_REQUIRED_FIELDS), + "part_evidence_files": dict(contract._UK_CERTIFICATION_PART_EVIDENCE_FILES), + "score_receipt_file": contract._UK_CERTIFICATION_SCORE_RECEIPT_FILE, + "gate_battery_phases": list(contract._UK_GATE_BATTERY_PHASES), + "signing_key_env": contract._UK_GATE_BATTERY_SIGNING_KEY_ENV, + }, REGENERATE + + +def test_inventory_files_are_the_files_the_manifests_ship(): + inventory = _json(FIXTURES / "c1" / "inventory.json") + release_manifest = _json(UK_RELEASE_DIR / "release_manifest.json") + shipped = { + artifact["path"] + for artifact in release_manifest["artifacts"].values() + if artifact["kind"] == "diagnostics" + } + directory_files = set(inventory["release_directory_files"]) + assert directory_files == shipped | {"build_manifest.json", "release_manifest.json"} + assert set(inventory["required_release_files"]) <= directory_files + certification = inventory["certification"] + assert { + certification["file"], + certification["score_receipt_file"], + *certification["part_evidence_files"].values(), + } <= directory_files + + +# --- C5 ----------------------------------------------------------------------- + + +def test_repository_pointer_is_the_publishers_payload_and_reads_back(): + path = FIXTURES / "c5" / release.LATEST_POINTER_PATH + payload = _json(path) + assert ( + path.read_bytes() + == json.dumps( + release.latest_pointer_payload( + payload["release_id"], updated_at=payload["updated_at"] + ), + indent=1, + ).encode() + ) + hub = _PointerHub(path) + pointer = release.latest_release("policyengine/populace-us", api=hub) + assert hub.requests == [("policyengine/populace-us", "latest.json", "dataset")] + assert pointer.release_id == payload["release_id"] + assert pointer.revision == pointer.release_id + assert pointer.line is None + assert set(pointer.paths) == { + name.removesuffix(".json") + for name in contract.required_release_files(pointer.release_id) + } + + +def test_line_pointer_is_the_publishers_payload_and_reads_back(): + path = FIXTURES / "c5" / release.line_pointer_path("national") + assert path == FIXTURES / "c5" / UK_NATIONAL_SPEC.pointer_path + payload = _json(path) + assert ( + path.read_bytes() + == json.dumps( + release.line_pointer_payload( + payload["release_id"], + line=payload["line"], + revision=payload["revision"], + updated_at=payload["updated_at"], + ), + indent=1, + ).encode() + ) + pointer = release.latest_line_release( + UK_NATIONAL_SPEC.hf_repo, line="national", api=_PointerHub(path) + ) + assert pointer.line == "national" + assert pointer.release_id == UK_RELEASE_ID + + +def test_line_pointer_names_the_c1_release_at_its_cut_tag(): + pointer = _json(FIXTURES / "c5" / UK_NATIONAL_SPEC.pointer_path) + release_manifest = _json(UK_RELEASE_DIR / "release_manifest.json") + revisions = { + artifact["revision"] for artifact in release_manifest["artifacts"].values() + } + assert pointer["release_id"] == release_manifest["build"]["build_id"] + assert revisions == {pointer["revision"]} == {release_manifest["build"]["cut_tag"]} + assert set(pointer["paths"].values()) == { + f"releases/{UK_RELEASE_ID}/{name}" + for name in contract.required_release_files(UK_RELEASE_ID) + } + + +def test_registry_snapshot_matches_the_live_registry(): + snapshot = _json(FIXTURES / "c5" / "registry.json") + assert snapshot["default_variant"] == DEFAULT_VARIANT + assert snapshot["entries"] == [ + {"key": list(key), **dataclasses.asdict(REGISTRY[key])} + for key in sorted(REGISTRY) + ], f"the dataset registry changed; {REGENERATE} (C5)" diff --git a/packages/microcosm-data/tests/engine_free/shared/test_release_contract_keying.py b/packages/microcosm-data/tests/engine_free/shared/test_release_contract_keying.py new file mode 100644 index 0000000000..3849ee0e26 --- /dev/null +++ b/packages/microcosm-data/tests/engine_free/shared/test_release_contract_keying.py @@ -0,0 +1,112 @@ +"""Executable statements of how the release contract keys on id strings today. + +docs/release-contract.md (C1, "The id-string keying") describes these gaps, and +C1 version 1, planned for milestone M3, replaces them with dispatch on the +manifest's declared country. Each test pins one of today's behaviours so that +the change is deliberate: when a change flips one, update the document and the +test together. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from microcosm.data import contract +from microcosm.data.contract import ( + REQUIRED_RELEASE_FILES, + ReleaseContractError, + line_for_release_id, + required_release_files, + validate_release_dir, +) +from microcosm.data.registry import DEFAULT_VARIANT, REGISTRY +from microcosm.data.release import line_pointer_payload +from test_support.microcosm_data.release import release_dir # noqa: F401 + +#: Ids in the C5 grammar for countries and lines the contract does not list. +UNLISTED_IDS = ( + "microcosm-nz-2025-national", + "microcosm-be-2025-national", + "microcosm-am-2025-national", + "microcosm-us-2024-national", +) + + +@pytest.mark.parametrize("release_id", UNLISTED_IDS) +def test_an_unlisted_id_gets_only_the_base_files_and_no_model_package(release_id): + assert required_release_files(release_id) == REQUIRED_RELEASE_FILES + assert contract._expected_model_package(release_id) is None + + +@pytest.mark.parametrize( + "release_id", (*UNLISTED_IDS, "populace-us-2024-0000000-20260101") +) +def test_only_uk_ids_belong_to_a_publication_line(release_id): + assert line_for_release_id(release_id) is None + + +def test_the_uk_national_and_local_lines_are_the_only_lines(): + assert line_for_release_id(contract._UK_NATIONAL_RELEASE_ID) == "national" + assert line_for_release_id("microcosm-uk-2024-25-local-k55000") == "local-k55000" + assert line_for_release_id(contract._UK_DENSE_RELEASE_ID) is None + + +def test_no_other_country_can_build_a_line_pointer(): + with pytest.raises(ValueError, match="belongs to line None"): + line_pointer_payload( + "microcosm-nz-2025-national", + line="national", + revision="microcosm-nz-2025-national", + ) + + +def test_the_registry_year_is_not_the_uk_national_ids_vintage(): + # The national line is the UK 2025 default load since microcosm#1432 + # (Max's #823 ruling). + spec = REGISTRY[("uk", 2025, DEFAULT_VARIANT)] + assert spec.pointer_path == "latest-national.json" + assert contract._UK_NATIONAL_RELEASE_ID == "microcosm-uk-2024-25-national" + assert not contract._UK_NATIONAL_RELEASE_ID.startswith(f"microcosm-uk-{spec.year}-") + + +def _retarget(directory: Path, release_id: str, *, drop: tuple[str, ...] = ()): + """Rename a release to ``release_id``, optionally dropping files.""" + + target = directory.with_name(release_id) + directory.rename(target) + build = json.loads((target / "build_manifest.json").read_text()) + build["build_id"] = release_id + (target / "build_manifest.json").write_text(json.dumps(build)) + manifest = json.loads((target / "release_manifest.json").read_text()) + manifest["build"]["build_id"] = release_id + for key in list(manifest["artifacts"]): + if manifest["artifacts"][key]["path"] in drop: + del manifest["artifacts"][key] + else: + manifest["artifacts"][key]["revision"] = release_id + (target / "release_manifest.json").write_text(json.dumps(manifest)) + for name in drop: + (target / name).unlink() + return target + + +def test_the_same_bytes_pass_under_an_unlisted_id_and_fail_under_a_us_id( + release_dir, # noqa: F811 +): + """A new country's release fails open: renaming a US release to an NZ id + lifts the source-coverage and stored-input requirements with nothing else + changed.""" + + drop = ("us_source_coverage.json", "stored_inputs.json") + unlisted = _retarget(release_dir, "microcosm-nz-2025-national", drop=drop) + validate_release_dir(unlisted) + + us = _retarget(unlisted, "populace-us-2024-0000000-20260101") + with pytest.raises(ReleaseContractError) as refusal: + validate_release_dir(us) + assert "required file 'us_source_coverage.json' is missing." in ( + refusal.value.failures + ) diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/SHA256SUMS b/packages/microcosm-data/tests/fixtures/release_contract/v1/SHA256SUMS new file mode 100644 index 0000000000..e01744d58f --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/SHA256SUMS @@ -0,0 +1,6 @@ +6b10db377aa40971a75c505ed83824744593bfd56d38c42dff2bb7587a3347fd c1/inventory.json +352749756824d6a6a4e586cbda8a7511c72ebea1d6bdc467b23aff776a6a7e8e c1/microcosm-uk-2024-25-national/build_manifest.json +30cf2bb211d8608c0b8009527a3adf47b82ca6fd2e2a0e6f51de1969a9a60b59 c1/microcosm-uk-2024-25-national/release_manifest.json +7ff1ecabf804a5aaaa8df547bb4285310e5c60b69e469f80c54a17fa5787b606 c5/latest-national.json +987ca2f38c67012c1ff0d36661c208f9e023f23bb4c7f91402f3caae826b8a33 c5/latest.json +f1e2f3306c7c20fcc8f6aacb65545305c1f9894c5773d5516f2425efc873005d c5/registry.json diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/inventory.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/inventory.json new file mode 100644 index 0000000000..c1952e21fa --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/inventory.json @@ -0,0 +1,58 @@ +{ + "certification": { + "file": "release_certification.json", + "gate_battery_phases": [ + "preflight", + "assembled", + "transferred", + "terminal" + ], + "kind": "uk_release_certification", + "part_evidence_files": { + "calibration_seam": "terminal_gates.json", + "release_cut": "release_cut_gates.json", + "spine": "spine_gates.json" + }, + "required_fields": [ + "attestation", + "candidate", + "country", + "diagnostics_sha256", + "doctrine", + "exclusions_evaluated_on", + "kind", + "parent_spine", + "parts", + "release_id", + "schema_version", + "score_receipt", + "shippable", + "spec" + ], + "schema_version": 1, + "score_receipt_file": "score_vs_enhanced_frs.json", + "signing_key_env": "MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY" + }, + "dataset_role": "national_default", + "line": "national", + "release_directory_files": [ + "build_manifest.json", + "calibration_diagnostics.json", + "release_certification.json", + "release_cut_gates.json", + "release_manifest.json", + "score_vs_enhanced_frs.json", + "spine_gates.json", + "terminal_gates.json" + ], + "release_id": "microcosm-uk-2024-25-national", + "release_manifest_schema_version": 1, + "required_release_files": [ + "build_manifest.json", + "release_manifest.json", + "calibration_diagnostics.json", + "release_certification.json" + ], + "schema_name": "microcosm.release-contract.c1-inventory", + "schema_version": 1 +} diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/build_manifest.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/build_manifest.json new file mode 100644 index 0000000000..5c614c5ac3 --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/build_manifest.json @@ -0,0 +1,63 @@ +{ + "build_id": "microcosm-uk-2024-25-national", + "code": { + "repository": "PolicyEngine/microcosm", + "git_commit": "a8fe1b954d0d42c1a6b10d663024d11201717a50", + "git_dirty": false + }, + "build_sha": "a8fe1b9", + "runtime": { + "python": "3.13.0", + "policyengine-core": "3.0.0", + "policyengine-uk": "2.0.0", + "microcosm-data": "0.1.0", + "microcosm-build": "0.1.0", + "microcosm-calibrate": "0.1.0" + }, + "dataset": { + "filename": "microcosm_uk_2024_25.h5", + "sha256": "d357112f939979d7e34696637ef874ca0531902c86bd34f503e1885d02eafb3e" + }, + "calibration": { + "filename": "microcosm_uk_2024_25_calibration.npz", + "sha256": "d1356e9ae3fb9baea7e7860931eeb274874f5ac61fc0b87277fc8245eac0801f", + "household_weight_year": { + "dataset_year": 2024, + "basis": "dataset_year" + }, + "target_surface": { + "sha256": "48b07537f3af196a534c7591ef2638babeb38c3d8dcfef910512b457ee5f2de8", + "n_targets": 2 + }, + "target_registry": { + "version": "release-contract-fixture-v1", + "n_specs": 2 + } + }, + "gates": { + "spine": { + "passed": 3 + }, + "calibration_seam": { + "passed": 2 + }, + "release_cut": { + "passed": 4 + } + }, + "attempt_id": "uk-frs-calibration-attempt-20260101T000000Z-00000000", + "cut_tag": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "created_at": "2026-01-01T00:00:00+00:00", + "staging": { + "contract_version": 2, + "enabled": true, + "mode": "local_and_remote", + "run_id": "uk-national-release-contract-fixture", + "configured_repository": "policyengine/populace-uk-staging", + "upload_attempts": 6, + "upload_successes": 6, + "read_back": "passed", + "last_error_code": null, + "opt_out_reason": null + } +} \ No newline at end of file diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/release_manifest.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/release_manifest.json new file mode 100644 index 0000000000..83f59eed8a --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c1/microcosm-uk-2024-25-national/release_manifest.json @@ -0,0 +1,112 @@ +{ + "schema_version": 1, + "data_package": { + "name": "microcosm-data", + "version": "0.1.0" + }, + "default_datasets": { + "national": "microcosm_uk_2024_25" + }, + "build": { + "build_id": "microcosm-uk-2024-25-national", + "built_at": "2026-01-01T00:00:00+00:00", + "built_with_core_package": { + "name": "policyengine-core", + "version": "3.0.0" + }, + "built_with_model_package": { + "name": "policyengine-uk", + "version": "2.0.0" + }, + "attempt_id": "uk-frs-calibration-attempt-20260101T000000Z-00000000", + "cut_tag": "microcosm-uk-2024-25-national-20260101T000000Z-00000000" + }, + "compatible_core_packages": [ + { + "name": "policyengine-core", + "specifier": "==3.0.0" + } + ], + "compatible_model_packages": [ + { + "name": "policyengine-uk", + "specifier": "==2.0.0" + } + ], + "artifacts": { + "microcosm_uk_2024_25": { + "kind": "microdata", + "path": "microcosm_uk_2024_25.h5", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "d357112f939979d7e34696637ef874ca0531902c86bd34f503e1885d02eafb3e", + "spi_public_use_tape_dependent": true + }, + "microcosm_uk_2024_25_calibration": { + "kind": "calibration", + "path": "microcosm_uk_2024_25_calibration.npz", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "d1356e9ae3fb9baea7e7860931eeb274874f5ac61fc0b87277fc8245eac0801f" + }, + "calibration_diagnostics": { + "kind": "diagnostics", + "path": "calibration_diagnostics.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "835fb1bf9e7ae31b5446813f76a8bc2c1fbb5e73d3dad2aae71558102c6988b5" + }, + "release_certification": { + "kind": "diagnostics", + "path": "release_certification.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "a587dbc0e451e476bc45d2e49898935bcd2f58b52d8805b4710c79ad25615dd9" + }, + "terminal_gates": { + "kind": "diagnostics", + "path": "terminal_gates.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "4da83cdcad9ae46e5c2d455476f5da1103490ca8398c543db4401cb1b4dacb40" + }, + "release_cut_gates": { + "kind": "diagnostics", + "path": "release_cut_gates.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "5d5bc71f11821673fe286473e15dcf74770cbccde099d2a893f9300c13ed23a8" + }, + "spine_gates": { + "kind": "diagnostics", + "path": "spine_gates.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "63eb81ac3d1dba7a137813c57358455c4b08c9e8383639d9ae26406121bece9a" + }, + "score_vs_enhanced_frs": { + "kind": "diagnostics", + "path": "score_vs_enhanced_frs.json", + "repo_id": "policyengine/populace-uk-private", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000", + "sha256": "75064c846279290ec8980518224b0ef5d65b2646813883dc643ef3954ef59cf4" + } + }, + "description": "UK national calibration pipeline release assembled from attempt uk-frs-calibration-attempt-20260101T000000Z-00000000 at immutable cut microcosm-uk-2024-25-national-20260101T000000Z-00000000.", + "publisher_labels": { + "ons": "Office for National Statistics" + }, + "licensed_data": { + "spi_public_use_tape": { + "study": "SN 9422", + "dependent": true, + "stages": [ + "release_contract_fixture_stage" + ], + "conditions": [ + "No attempt may be made to attribute a record to a named individual.", + "Material that includes analyses of the tape goes to HMRC at least ten working days before it is distributed or published." + ] + } + } +} \ No newline at end of file diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest-national.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest-national.json new file mode 100644 index 0000000000..c158dee8ba --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest-national.json @@ -0,0 +1,13 @@ +{ + "schema_version": 1, + "release_id": "microcosm-uk-2024-25-national", + "updated_at": "2026-01-01T00:00:00+00:00", + "paths": { + "build_manifest": "releases/microcosm-uk-2024-25-national/build_manifest.json", + "release_manifest": "releases/microcosm-uk-2024-25-national/release_manifest.json", + "calibration_diagnostics": "releases/microcosm-uk-2024-25-national/calibration_diagnostics.json", + "release_certification": "releases/microcosm-uk-2024-25-national/release_certification.json" + }, + "line": "national", + "revision": "microcosm-uk-2024-25-national-20260101T000000Z-00000000" +} \ No newline at end of file diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest.json new file mode 100644 index 0000000000..d81dffc7ad --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/latest.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "release_id": "populace-us-2024-0000000-20260101", + "updated_at": "2026-01-01T00:00:00+00:00", + "paths": { + "build_manifest": "releases/populace-us-2024-0000000-20260101/build_manifest.json", + "release_manifest": "releases/populace-us-2024-0000000-20260101/release_manifest.json", + "calibration_diagnostics": "releases/populace-us-2024-0000000-20260101/calibration_diagnostics.json", + "us_source_coverage": "releases/populace-us-2024-0000000-20260101/us_source_coverage.json" + } +} \ No newline at end of file diff --git a/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/registry.json b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/registry.json new file mode 100644 index 0000000000..6c50933c08 --- /dev/null +++ b/packages/microcosm-data/tests/fixtures/release_contract/v1/c5/registry.json @@ -0,0 +1,71 @@ +{ + "default_variant": "compact", + "entries": [ + { + "country": "uk", + "engine_class": "UKSingleYearDataset", + "engine_module": "policyengine_uk.data", + "engine_package": "policyengine-uk", + "filename": "populace_uk_2023.h5", + "hf_repo": "policyengine/populace-uk-private", + "key": [ + "uk", + 2023, + "compact" + ], + "pointer_path": "latest.json", + "variant": "compact", + "year": 2023 + }, + { + "country": "uk", + "engine_class": "UKSingleYearDataset", + "engine_module": "policyengine_uk.data", + "engine_package": "policyengine-uk", + "filename": "microcosm_uk_2024_25.h5", + "hf_repo": "policyengine/populace-uk-private", + "key": [ + "uk", + 2025, + "compact" + ], + "pointer_path": "latest-national.json", + "variant": "compact", + "year": 2025 + }, + { + "country": "uk", + "engine_class": "UKSingleYearDataset", + "engine_module": "policyengine_uk.data", + "engine_package": "policyengine-uk", + "filename": "microcosm_uk_2024_25_dense.h5", + "hf_repo": "policyengine/populace-uk-private", + "key": [ + "uk", + 2025, + "dense" + ], + "pointer_path": "latest.json", + "variant": "dense", + "year": 2025 + }, + { + "country": "us", + "engine_class": "USSingleYearDataset", + "engine_module": "policyengine_us.data", + "engine_package": "policyengine-us", + "filename": "populace_us_2024.h5", + "hf_repo": "policyengine/populace-us", + "key": [ + "us", + 2024, + "compact" + ], + "pointer_path": "latest.json", + "variant": "compact", + "year": 2024 + } + ], + "schema_name": "microcosm.release-contract.registry", + "schema_version": 1 +} diff --git a/tools/assemble_uk_release_dir.py b/tools/assemble_uk_release_dir.py index decc753c08..c6b706752a 100644 --- a/tools/assemble_uk_release_dir.py +++ b/tools/assemble_uk_release_dir.py @@ -54,6 +54,17 @@ "microcosm-build", "microcosm-calibrate", ) +#: The signed evidence a national release ships beside its manifests, in +#: release-manifest order: the artifact key, the release-directory filename, and +#: the name ``_assemble`` measures its digest under. +_EVIDENCE = ( + ("calibration_diagnostics", "calibration_diagnostics.json", "diagnostics"), + ("release_certification", "release_certification.json", "certification"), + ("terminal_gates", "terminal_gates.json", "calibration seam report"), + ("release_cut_gates", "release_cut_gates.json", "release-cut report"), + ("spine_gates", "spine_gates.json", "spine report"), + ("score_vs_enhanced_frs", "score_vs_enhanced_frs.json", "score receipt"), +) def _uk_publisher_labels() -> dict[str, str]: @@ -366,18 +377,12 @@ def _assemble(args: argparse.Namespace) -> dict[str, object]: calibration_path.unlink(missing_ok=True) -def _stage_and_finalize( +def uk_national_release_manifests( *, - args: argparse.Namespace, - staging_parent: Path, - destination: Path, - npz_destination: Path, - calibration_path: Path, + measured: Mapping[str, str], calibration_filename: str, - household_weight: np.ndarray, - initial_household_weight: np.ndarray, + calibration_sha: str, household_weight_year: Mapping[str, object], - measured: Mapping[str, str], cut_tag: str, attempt_id: str, runtime: Mapping[str, str], @@ -387,20 +392,20 @@ def _stage_and_finalize( spine_part: Mapping[str, object], seam_part: Mapping[str, object], release_cut_part: Mapping[str, object], - spine_report_path: Path, staging_delivery: Mapping[str, object], spi_marker: Mapping[str, object], -) -> dict[str, object]: - np.savez( - calibration_path, - household_weight=household_weight, - initial_household_weight=initial_household_weight, - ) - calibration_sha = _sha256(calibration_path) + created_at: str, + publisher_labels: Mapping[str, str], +) -> tuple[dict[str, object], dict[str, object]]: + """Return the national line's build and release manifests. + + A pure function of its inputs: the assembler writes exactly these two + documents, and ``tools/generate_release_contract_fixtures.py`` writes the + release-contract golden fixtures (docs/release-contract.md, C1) with this + same function, so a change to either manifest's shape is a change to those + fixtures. + """ - created_at = datetime.now(UTC).isoformat() - release_dir = staging_parent / UK_NATIONAL_RELEASE_ID - release_dir.mkdir(parents=True) build_manifest = { "build_id": UK_NATIONAL_RELEASE_ID, "code": { @@ -409,7 +414,7 @@ def _stage_and_finalize( "git_dirty": False, }, "build_sha": code_pin[:7], - "runtime": runtime, + "runtime": dict(runtime), "dataset": { "filename": _DATASET_FILENAME, "sha256": measured["candidate"], @@ -443,29 +448,6 @@ def _stage_and_finalize( "created_at": created_at, "staging": dict(staging_delivery), } - build_manifest_path = release_dir / "build_manifest.json" - _write_json(build_manifest_path, build_manifest) - - evidence = { - "calibration_diagnostics": ( - args.diagnostics_json, - "calibration_diagnostics.json", - ), - "release_certification": ( - args.certification_json, - "release_certification.json", - ), - "terminal_gates": (args.seam_gate_report, "terminal_gates.json"), - "release_cut_gates": ( - args.release_cut_gate_json, - "release_cut_gates.json", - ), - "spine_gates": (spine_report_path, "spine_gates.json"), - "score_vs_enhanced_frs": ( - args.score_receipt, - "score_vs_enhanced_frs.json", - ), - } def artifact(kind: str, path: str, sha256: str) -> dict[str, str]: return { @@ -520,55 +502,91 @@ def artifact(kind: str, path: str, sha256: str) -> dict[str, str]: "calibration", calibration_filename, calibration_sha ), **{ - key: artifact("diagnostics", filename, measured[name]) - for key, (_source, filename), name in ( - ( - "calibration_diagnostics", - evidence["calibration_diagnostics"], - "diagnostics", - ), - ( - "release_certification", - evidence["release_certification"], - "certification", - ), - ( - "terminal_gates", - evidence["terminal_gates"], - "calibration seam report", - ), - ( - "release_cut_gates", - evidence["release_cut_gates"], - "release-cut report", - ), - ( - "spine_gates", - evidence["spine_gates"], - "spine report", - ), - ( - "score_vs_enhanced_frs", - evidence["score_vs_enhanced_frs"], - "score receipt", - ), - ) + key: artifact("diagnostics", filename, measured[measured_as]) + for key, filename, measured_as in _EVIDENCE }, }, "description": ( "UK national calibration pipeline release assembled from attempt " f"{attempt_id} at immutable cut {cut_tag}." ), - "publisher_labels": _uk_publisher_labels(), + "publisher_labels": dict(publisher_labels), "licensed_data": {UK_SPI_PUBLIC_USE_TAPE_KEY: dict(spi_marker)}, } + return build_manifest, release_manifest + + +def _stage_and_finalize( + *, + args: argparse.Namespace, + staging_parent: Path, + destination: Path, + npz_destination: Path, + calibration_path: Path, + calibration_filename: str, + household_weight: np.ndarray, + initial_household_weight: np.ndarray, + household_weight_year: Mapping[str, object], + measured: Mapping[str, str], + cut_tag: str, + attempt_id: str, + runtime: Mapping[str, str], + code_pin: str, + target_surface: Mapping[str, object], + target_registry: Mapping[str, object], + spine_part: Mapping[str, object], + seam_part: Mapping[str, object], + release_cut_part: Mapping[str, object], + spine_report_path: Path, + staging_delivery: Mapping[str, object], + spi_marker: Mapping[str, object], +) -> dict[str, object]: + np.savez( + calibration_path, + household_weight=household_weight, + initial_household_weight=initial_household_weight, + ) + calibration_sha = _sha256(calibration_path) + + created_at = datetime.now(UTC).isoformat() + release_dir = staging_parent / UK_NATIONAL_RELEASE_ID + release_dir.mkdir(parents=True) + build_manifest, release_manifest = uk_national_release_manifests( + measured=measured, + calibration_filename=calibration_filename, + calibration_sha=calibration_sha, + household_weight_year=household_weight_year, + cut_tag=cut_tag, + attempt_id=attempt_id, + runtime=runtime, + code_pin=code_pin, + target_surface=target_surface, + target_registry=target_registry, + spine_part=spine_part, + seam_part=seam_part, + release_cut_part=release_cut_part, + staging_delivery=staging_delivery, + spi_marker=spi_marker, + created_at=created_at, + publisher_labels=_uk_publisher_labels(), + ) + build_manifest_path = release_dir / "build_manifest.json" + _write_json(build_manifest_path, build_manifest) release_manifest_path = release_dir / "release_manifest.json" _write_json(release_manifest_path, release_manifest) + sources = { + "calibration_diagnostics": args.diagnostics_json, + "release_certification": args.certification_json, + "terminal_gates": args.seam_gate_report, + "release_cut_gates": args.release_cut_gate_json, + "spine_gates": spine_report_path, + "score_vs_enhanced_frs": args.score_receipt, + } evidence_summary: dict[str, dict[str, str]] = {} - for key, (source, filename) in evidence.items(): + for key, filename, _measured_as in _EVIDENCE: copy_path = release_dir / filename - shutil.copyfile(source, copy_path) + shutil.copyfile(sources[key], copy_path) evidence_summary[key] = { "path": str(destination / filename), "sha256": _sha256(copy_path), diff --git a/tools/generate_release_contract_fixtures.py b/tools/generate_release_contract_fixtures.py new file mode 100644 index 0000000000..0a113fbf89 --- /dev/null +++ b/tools/generate_release_contract_fixtures.py @@ -0,0 +1,277 @@ +"""Generate or verify the release-contract golden fixtures (version 1). + +docs/release-contract.md defines the release contract, C1 to C5. These fixtures +freeze two of its parts in bytes, the way ``generate_staging_contract_fixtures.py`` +freezes the staging documents that calibration-diagnostics byte-checks: + +* C1, the release manifest: the UK national line's ``build_manifest.json`` and + ``release_manifest.json`` as the national assembler writes them (the generator + calls ``uk_national_release_manifests`` and ``_write_json`` from + ``tools/assemble_uk_release_dir.py``), plus ``inventory.json``, the files and + certification fields the release contract requires of that line, read from + ``microcosm.data.contract``. +* C5, publication: the two pointer payloads the publisher writes (the + repository-global ``latest.json`` and the national line's + ``latest-national.json``), serialized as ``publish_release`` serializes them, + and ``registry.json``, a snapshot of ``microcosm.data.registry.REGISTRY``. + +Every digest, version, identifier and gate count in the C1 manifests and the +pointers is synthetic. A digest is the SHA-256 of ``release-contract-fixture:``, +so no reader can mistake one for a real artifact. The SPI Public Use Tape marker names +one synthetic stage; its study number and conditions are the live constants of +``microcosm.build.uk_runtime.spi_dependency``, which a dependent release restates. +``inventory.json`` and ``registry.json`` carry live values on purpose: they freeze +the contract's constants and the registry. +The fixtures deliberately carry no gate-battery report or signed certification: +those bind the UK gate spec's live digests, which change with most UK pull +requests, so ``inventory.json`` freezes the certification's shape instead, and the +assembler's tests keep covering its signed validity. + +Regenerate after a deliberate contract change, in the same pull request that +updates docs/release-contract.md:: + + uv run python tools/generate_release_contract_fixtures.py + +``--check`` regenerates into a temporary directory and fails unless every byte +matches the committed fixtures. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import hashlib +import importlib.util +import json +import shutil +import tempfile +from pathlib import Path +from types import ModuleType, SimpleNamespace + +from microcosm.build.staging_v2 import validate_staging_delivery +from microcosm.build.uk_runtime.spi_dependency import uk_spi_public_use_tape_marker +from microcosm.build.uk_runtime.spi_income import SPI_DONOR_UKDS_STUDY +from microcosm.data import contract +from microcosm.data.registry import DEFAULT_VARIANT, REGISTRY +from microcosm.data.release import ( + LATEST_POINTER_PATH, + latest_pointer_payload, + line_pointer_path, + line_pointer_payload, +) + +ROOT = Path(__file__).resolve().parents[1] +DEFAULT_OUTPUT = ( + ROOT + / "packages" + / "microcosm-data" + / "tests" + / "fixtures" + / "release_contract" + / "v1" +) +ASSEMBLER = ROOT / "tools" / "assemble_uk_release_dir.py" + +#: One instant for every timestamp the fixtures carry. +FIXTURE_TIME = "2026-01-01T00:00:00+00:00" +#: A calibration attempt id in the grammar the assembler derives cut tags from. +UK_ATTEMPT_ID = "uk-frs-calibration-attempt-20260101T000000Z-00000000" +#: A US national release id in today's ``populace-us-`` grammar, for the +#: repository-global pointer. +US_RELEASE_ID = "populace-us-2024-0000000-20260101" +UK_LINE = "national" +#: Synthetic runtime versions. Each is a resolved PEP 440 version, which is all the +#: contract asks of a runtime pin. +UK_RUNTIME = { + "python": "3.13.0", + "policyengine-core": "3.0.0", + "policyengine-uk": "2.0.0", + "microcosm-data": "0.1.0", + "microcosm-build": "0.1.0", + "microcosm-calibrate": "0.1.0", +} +#: A successful version 2 staging delivery, matching the staging fixtures' +#: ``delivery_success`` case apart from its run id. +UK_STAGING_DELIVERY = { + "contract_version": 2, + "enabled": True, + "mode": "local_and_remote", + "run_id": "uk-national-release-contract-fixture", + "configured_repository": "policyengine/populace-uk-staging", + "upload_attempts": 6, + "upload_successes": 6, + "read_back": "passed", + "last_error_code": None, + "opt_out_reason": None, +} +#: One synthetic spine stage that reads the SPI Public Use Tape, so the C1 +#: manifests carry the marker a tape-dependent national release restates +#: (microcosm#1095): the flag on the microdata entry and ``licensed_data``. +UK_SPI_STAGE = SimpleNamespace( + stage="release_contract_fixture_stage", + artifacts=[{"ukds_study_number": SPI_DONOR_UKDS_STUDY}], +) + + +def fixture_digest(name: str) -> str: + """The synthetic SHA-256 the fixtures use for ``name``.""" + + return hashlib.sha256(f"release-contract-fixture:{name}".encode()).hexdigest() + + +def _load_assembler() -> ModuleType: + spec = importlib.util.spec_from_file_location("assemble_uk_release_dir", ASSEMBLER) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def _json_bytes(payload: object) -> bytes: + return (json.dumps(payload, indent=2, sort_keys=True) + "\n").encode() + + +def _c1_uk_national(root: Path, assembler: ModuleType) -> str: + """Write the UK national manifests; return the line's cut tag.""" + + cut_tag = assembler._cut_tag(UK_ATTEMPT_ID, None) + measured = { + name: fixture_digest(name) + for name in ("candidate", *(name for _, _, name in assembler._EVIDENCE)) + } + build_manifest, release_manifest = assembler.uk_national_release_manifests( + measured=measured, + calibration_filename=f"{assembler._DATASET_KEY}_calibration.npz", + calibration_sha=fixture_digest("calibration"), + household_weight_year={"dataset_year": 2024, "basis": "dataset_year"}, + cut_tag=cut_tag, + attempt_id=UK_ATTEMPT_ID, + runtime=UK_RUNTIME, + code_pin=hashlib.sha1(b"release-contract-fixture").hexdigest(), + target_surface={"sha256": fixture_digest("target surface"), "n_targets": 2}, + target_registry={"version": "release-contract-fixture-v1", "n_specs": 2}, + # Each part's statuses are the certifier's census of its gates by status + # (``_status_census``); a shippable part is all "passed". The counts are + # synthetic, so the fixture does not move when a UK gate scope grows. + spine_part={"statuses": {"passed": 3}}, + seam_part={"statuses": {"passed": 2}}, + release_cut_part={"statuses": {"passed": 4}}, + staging_delivery=validate_staging_delivery(UK_STAGING_DELIVERY), + spi_marker=uk_spi_public_use_tape_marker([UK_SPI_STAGE]), + created_at=FIXTURE_TIME, + publisher_labels={"ons": "Office for National Statistics"}, + ) + release_dir = root / "c1" / contract._UK_NATIONAL_RELEASE_ID + assembler._write_json(release_dir / "build_manifest.json", build_manifest) + assembler._write_json(release_dir / "release_manifest.json", release_manifest) + + inventory = { + "schema_name": "microcosm.release-contract.c1-inventory", + "schema_version": 1, + "release_id": contract._UK_NATIONAL_RELEASE_ID, + "line": contract.line_for_release_id(contract._UK_NATIONAL_RELEASE_ID), + "dataset_role": contract.dataset_role_for_line(UK_LINE), + "release_manifest_schema_version": contract.RELEASE_MANIFEST_SCHEMA_VERSION, + "required_release_files": list( + contract.required_release_files(contract._UK_NATIONAL_RELEASE_ID) + ), + "release_directory_files": sorted( + [ + "build_manifest.json", + "release_manifest.json", + *(filename for _, filename, _ in assembler._EVIDENCE), + ] + ), + "certification": { + "file": contract._UK_RELEASE_CERTIFICATION_FILE, + "schema_version": contract._UK_RELEASE_CERTIFICATION_SCHEMA_VERSION, + "kind": contract._UK_RELEASE_CERTIFICATION_KIND, + "required_fields": sorted(contract._UK_CERTIFICATION_REQUIRED_FIELDS), + "part_evidence_files": dict(contract._UK_CERTIFICATION_PART_EVIDENCE_FILES), + "score_receipt_file": contract._UK_CERTIFICATION_SCORE_RECEIPT_FILE, + "gate_battery_phases": list(contract._UK_GATE_BATTERY_PHASES), + "signing_key_env": contract._UK_GATE_BATTERY_SIGNING_KEY_ENV, + }, + } + (root / "c1" / "inventory.json").write_bytes(_json_bytes(inventory)) + return cut_tag + + +def _c5_publication(root: Path, *, uk_cut_tag: str) -> None: + c5 = root / "c5" + c5.mkdir(parents=True, exist_ok=True) + # publish_release uploads ``json.dumps(payload, indent=1)`` with no newline. + (c5 / LATEST_POINTER_PATH).write_bytes( + json.dumps( + latest_pointer_payload(US_RELEASE_ID, updated_at=FIXTURE_TIME), indent=1 + ).encode() + ) + (c5 / line_pointer_path(UK_LINE)).write_bytes( + json.dumps( + line_pointer_payload( + contract._UK_NATIONAL_RELEASE_ID, + line=UK_LINE, + revision=uk_cut_tag, + updated_at=FIXTURE_TIME, + ), + indent=1, + ).encode() + ) + registry = { + "schema_name": "microcosm.release-contract.registry", + "schema_version": 1, + "default_variant": DEFAULT_VARIANT, + "entries": [ + {"key": list(key), **dataclasses.asdict(REGISTRY[key])} + for key in sorted(REGISTRY) + ], + } + (c5 / "registry.json").write_bytes(_json_bytes(registry)) + + +def _generate(root: Path) -> None: + root.mkdir(parents=True, exist_ok=True) + cut_tag = _c1_uk_national(root, _load_assembler()) + _c5_publication(root, uk_cut_tag=cut_tag) + files = sorted(path for path in root.rglob("*") if path.is_file()) + checksums = [ + f"{hashlib.sha256(path.read_bytes()).hexdigest()} " + f"{path.relative_to(root).as_posix()}" + for path in files + ] + (root / "SHA256SUMS").write_text("\n".join(checksums) + "\n") + + +def _tree(root: Path) -> dict[str, bytes]: + return { + path.relative_to(root).as_posix(): path.read_bytes() + for path in root.rglob("*") + if path.is_file() + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--check", action="store_true") + parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT) + args = parser.parse_args(argv) + + with tempfile.TemporaryDirectory(prefix="microcosm-release-contract-") as temporary: + generated = Path(temporary) / "v1" + _generate(generated) + if args.check: + if _tree(generated) != _tree(args.output): + raise SystemExit( + "release-contract fixtures are not reproducible: regenerate " + "with `uv run python tools/generate_release_contract_fixtures.py` " + "and update docs/release-contract.md in the same change" + ) + return 0 + if args.output.exists(): + shutil.rmtree(args.output) + shutil.copytree(generated, args.output) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())