Skip to content

Route UK data issues to the repo that builds the run's dataset - #110

Draft
MaxGhenis wants to merge 3 commits into
mainfrom
uk-data-issue-routing
Draft

MaxGhenis wants to merge 3 commits into
mainfrom
uk-data-issue-routing

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

report-logger filed every UK INVESTIGATE data issue in PolicyEngine/policyengine-uk-data. policyengine.py 6.16.0 loads Microcosm's microcosm_uk_2024_25 whenever a UK caller names no dataset, and policyengine-api runs only that default. So those issues went to a repo that didn't build the run's data. report-logger now files each data issue in the repo that built the run's dataset:

Run Data issue goes to
US PolicyEngine/microcosm (unchanged)
UK on the certified default (every API run) PolicyEngine/microcosm
UK on a named Enhanced FRS or FRS file (enhanced_frs_*, frs_*) PolicyEngine/policyengine-uk-data
Any other country PolicyEngine/policyengine-{country}-data (unchanged)

Structural issues still go to PolicyEngine/policyengine-{country}.

What changes

  • report-logger.md picks the repo from the run's country and, for UK runs, a new dataset_at_run frontmatter field, in both its routing prose and its issue-filing bash. Microcosm now receives data issues from both countries, so the issue body names the dataset and its data version.
  • microsim-runner.md adds UK guidance for the API path (region uk, baseline policy 1, no dataset parameter) and for the local path, where pe.uk.ensure_datasets(datasets=[dataset], years=[year]) loads a named dataset. It accepts an optional dataset for local runs and returns dataset and data_version with each result. /analyze-policy gains --dataset, which needs --mode local, and records dataset_at_run in the archive.
  • calibration-diagnostics.md points a UK hypothesis at Microcosm's UK target files, or at policyengine-uk-data when the run used one of its files. It also tells the agent that the calibration API lists only US releases.
  • The policyengine skill said the pre-2026 datasets "are gone", and the policyengine-data skill said the per-area files "no longer exist" and "were removed". The policyengine and policyengine-uk skills now say that policyengine.py serves the Enhanced FRS and FRS files only when a caller names one, from the gated policyengine/policyengine-uk-data-private repo. The policyengine and policyengine-data skills now say that the bundle serves no per-area file.
  • test_country_routing.py checks the rule with example, property and differential tests. The differential tests run report-logger's bash and resolve_issue_repo, a Python restatement of the rule, on the same inputs.
  • The workflow's log_to description and target-repo comment, reform-pipeline.md, docs/structured-output-and-chaining.md and tests/pipeline/README.md repeat the rule, and analyses/README.md documents dataset_at_run. The stale-reference guard's message for per-area paths now says the bundle serves no such file.

Evidence

Claim Source
policyengine.py 6.16.0's bundle manifest names microcosm_uk_2024_25 as the UK default manifest.json:106
The manifest lists the Enhanced FRS and FRS files under dataset_overlays, in policyengine/policyengine-uk-data-private, and _apply_dataset_overlays raises an error for any overlay that would shadow a certified dataset manifest.json:261-320, provenance/manifest.py:756-788
resolve_managed_dataset_reference returns the certified default when the caller names no dataset, and the named file otherwise provenance/manifest.py:1357
pe.uk.ensure_datasets loads named datasets, and pe.uk exports economic_impact_analysis but no calculate_budgetary_impact uk/datasets.py:378-424, uk/__init__.py:46-61
policyengine-api builds every economy request with dataset="default" routes/economy_routes.py:128, :290
Its economy query models reject undeclared fields. GET /uk/economy/1/over/1?region=uk&time_period=2026&dataset=enhanced_frs_2024_25 returned HTTP 400, "dataset: Extra inputs are not permitted", on 2026-11-08, and the US route answered the same way query_parameters.py:114-117, :152
Each economy result carries a policyengine_bundle with the model version, data version and dataset services/economy_impact_execution.py:402
/uk/metadata lists one dataset under economy_options.datasets: microcosm_uk_2024_25 live API, 2026-11-08
Microcosm keeps its UK targets in build/uk/ and its UK build stages in build/uk_runtime/ microcosm 3db6515
The calibration API lists 30 releases, all US GET https://calibration-diagnostics.vercel.app/calibration/dashboard/api/microcosm/releases, 2026-11-08
Hugging Face's policyengine/policyengine-us-data still holds 51 state and 436 district H5 files, and the bundle manifest references no file from that repo Hugging Face tree API and manifest.json, 2026-11-08

Invariants

test_country_routing.py checks these for any dataset string Hypothesis generates:

  • routing returns None or one PolicyEngine/<repo> name;
  • only INVESTIGATE and structural verdicts open an issue;
  • structural routing and US INVESTIGATE routing ignore the dataset;
  • UK INVESTIGATE routing lands in microcosm or policyengine-uk-data and reads only the dataset's name, which it takes as policyengine.py's dataset_logical_name does (drop any @revision, keep the last path segment), so a dataset name and its hf:// URI route alike;
  • report-logger's bash and resolve_issue_repo agree for us and uk.

A strict xfail records the one case where the two disagree: the bash sends ca to policyengine-ca-data, the prose sends it to policyengine-canada-data, and neither repo exists.

Not in this PR

  • microsim-runner's US API path still sends a dataset query parameter, which policyengine-api answers with HTTP 400. Its table of advertised US datasets also lists names that /us/metadata no longer serves. model-corroborator.md:105 sends the same parameter, and tests/pipeline/test_dataset_naming.py treats the old US names as advertised and the live default as a bad name.
  • Neither PolicyEngine/microcosm nor PolicyEngine/policyengine-uk-data has the calibration or auto-filed label that report-logger's gh issue create --label passes, and gh stops before it creates an issue whose label the repo lacks (params.go:106-109).
  • The ca route, which sends data issues to repos that don't exist.

Tests

uv run pytest --ignore=tests/test_skill_examples.py passes 256 tests and xfails the ca case. scripts/build_claude_wrapper.py copies the eight edited agents, commands and skills into the wrapper byte for byte.

Before merge

🤖 Generated with Claude Code

MaxGhenis added a commit that referenced this pull request Nov 8, 2026
#110's 597ab30 made this paragraph take both values from the metadata
default when a result has no policyengine_bundle. The rebase kept this
branch's rewrite of the paragraph, which now names that case and the
budget-window results it covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 3 commits November 8, 2026 15:29
policyengine.py 6.16.0 defaults UK runs to Microcosm's microcosm_uk_2024_25,
and policyengine-api runs only that default, yet report-logger sent every UK
INVESTIGATE issue to policyengine-uk-data. It now routes US and UK data issues
to PolicyEngine/microcosm, and a UK run that named an Enhanced FRS or FRS file
to policyengine-uk-data, which builds those files. The archive frontmatter
gains dataset_at_run, taken from the microsim result's dataset, to carry that
choice.

- report-logger: country-and-dataset routing in prose and in the issue bash;
  the issue body names the dataset and its data version.
- microsim-runner: UK API and local-path guidance, an optional dataset input,
  and dataset and data_version in its output; /analyze-policy gains --dataset
  for local runs.
- calibration-diagnostics: where Microcosm keeps its UK targets, and that the
  live calibration API lists US releases only.
- policyengine and policyengine-uk skills: policyengine.py still serves the
  Enhanced FRS and FRS files by name, as dataset overlays, and serves no
  per-area file.
- test_country_routing: example, property and differential tests; the
  differential tests run report-logger's bash against the Python reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…egment

report-logger took the last path segment of dataset_at_run, so a URI whose
revision contains a slash, such as enhanced_frs_2024_25.h5@refs/pr/3, routed
to microcosm. It now drops any @revision first, as policyengine.py's
dataset_logical_name does, then keeps the last path segment. The reference
in test_country_routing.py does the same, the revision strategy now allows
slashes, and a refs/pr/3 example joins the example and differential tests.

microsim-runner also falls back to /{country}/metadata when a result carries
no policyengine_bundle, and calibration-diagnostics names the UK default
where it said "their place".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…they're gone

The data skill said the old per-area H5s "no longer exist" and that
policyengine-us-data's per-area files "were removed". Hugging Face's
policyengine/policyengine-us-data still holds 51 state and 436 district H5
files; policyengine.py 6.16.0's bundle manifest references none of that
repo's files. Both lines now say what the bundle serves, matching the
policyengine skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the uk-data-issue-routing branch from 597ab30 to b4f7854 Compare November 8, 2026 20:30
MaxGhenis added a commit that referenced this pull request Nov 8, 2026
#110's 597ab30 made this paragraph take both values from the metadata
default when a result has no policyengine_bundle. The rebase kept this
branch's rewrite of the paragraph, which now names that case and the
budget-window results it covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant