Repository navigation
ensure_datasets: consume Microcosm-published UK releases (manifest-verified), so analyses can swap bases by URI #495
Description
Doctrine says policyengine.py is the permanent interface and Microcosm is where UK data construction is migrating (PolicyEngine/microcosm#665). The consumption path is currently missing: pe.uk.ensure_datasets materialises per-year files from hf://policyengine/policyengine-uk-data/..., and there is no way to point it at a Microcosm-published UK release (certified single-year artifact + release manifest with pinned SHA-256s).
Concrete need, with a live consumer waiting: PolicyEngine/uk-cgt-paper#6 and PolicyEngine/uk-cgt-reform#1 — both reform analyses should rerun on the Microcosm UK build once it certifies (currently gated on microcosm#630), ideally changing nothing but a dataset URI.
Scope sketch:
- accept a Microcosm release identifier (HF repo + release id) in
ensure_datasets; - verify artifact SHA-256 against the release manifest before use (Microcosm releases pin these; the interface should check, matching Microcosm's own runtime verification discipline);
- resolve the release's
compatible_model_packages/compatible_core_packagesconstraints and fail loudly on mismatch with the installed bundle; - forward-year materialisation (uprating 2023 → simulation years) with the same semantics the uk-data path applies today, so results are comparable across the swap.
Related: #491 (publisher claim). Happy to pair on the design — the UK CGT arc (microcosm#552) is the first artifact that wants this path.
Activity
MaxGhenis commentedon Aug 19, 2026
Scope correction after reading the bundle machinery: the consumption path I claimed was missing largely exists — the bundle manifest's UK entry already carries a microcosm release as the default dataset (revision-addressed URI, release-manifest URI, certification block pairing it to the model version), and ensure_datasets resolves it. What remains of this issue: (1) the paired update ceremony for a NEW release (scripts/prepare_package_bundle_update.py handles package pins only; the dataset entry + certification block need the same scripted path), (2) runtime sha256 verification of the resolved artifact against the release manifest, matching microcosm's own discipline, and (3) forward-year materialisation semantics documented for the microcosm artifact. Next concrete consumer: pairing policyengine-uk 2.91.0 with the first CGT-family release (supersedes the closed #496 approach).
MaxGhenis commentedon Aug 20, 2026
Further scope collapse after reading docs/engineering/skills/data-certification.md (which I should have read first): the paired-update ceremony already exists —
scripts/bundle.py certify-data --country uk --data-producer populace --manifest-uri hf://…fetches and validates the release manifest, writes the bundle entry, exact-pins the model package, regenerates derived metadata, and writes the fragment, with the certification gate refusing pairs outside built-with or the publisher's compatible_model_packages claim.So the remaining substance here is exactly one gap, confirmed by reading the resolve path:
ensure_datasets→materialize_dataset_sourceperforms no byte-level sha256 verification (thebundle installCLI path hashes; the ensure path does not). PR for that incoming.One operational note for the next UK certification: the certification gate needs the new microcosm release to either be built with the model version being certified or declare it in
compatible_model_packages— so a release built with 2.89.x pairs with 2.91.0 only if the release manifest carries that claim.juaristi22 commentedon Nov 10, 2026
Closing as done. After the 20 August scope note above, what remained was SHA-256 verification on the
ensure_datasetspath. #503 (merged 29 August) added it: UKensure_datasetsmaterializes datasets throughprovenance/dataset_materialization.py, which refuses a managed dataset with no certified SHA-256 and checks each download against it before use (#498 closed as superseded by #503).The consumption this issue asked for is in use. A Microcosm UK national and local-area pair is the certified UK default since #553 (closed 21 October through #685), and later pairs follow the same path, most recently #744 (merged 9 November). For forward years, #730 refuses dataset years the model does not project to, and #735 (open) records the supported years in the bundle manifest.
Open UK data work lives in PolicyEngine/microcosm#1095. PolicyEngine/microcosm#665, the migration epic that pointed here, closes today.