Skip to content

Keep report-logger's data issues from failing on missing labels, and file none for Canada - #111

Draft
MaxGhenis wants to merge 2 commits into
uk-data-issue-routingfrom
report-logger-labels-ca
Draft

MaxGhenis wants to merge 2 commits into
uk-data-issue-routingfrom
report-logger-labels-ca

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

report-logger files each US and UK data issue in PolicyEngine/microcosm or PolicyEngine/policyengine-uk-data, and its issue-filing bash passed --label "calibration,auto-filed" to gh issue create. On 2026-11-08 neither repo had either label. gh looks up each label name before it creates an issue and stops when the repo lacks one, so every auto-filed data issue failed. The bash also sent ca data issues to policyengine-ca-data, the prose sent them to policyengine-canada-data, and neither repo exists.

report-logger now passes only the labels the target repo has, and it files no data issue for Canada:

Run Data issue goes to Labels it gets today
US, or UK on a Microcosm dataset PolicyEngine/microcosm none, since the repo has neither
UK on a named Enhanced FRS or FRS file PolicyEngine/policyengine-uk-data none, since the repo has neither
Canada no issue; the archive keeps the report

Labels

The block asks gh for the repo's labels and passes --label for each of calibration and auto-filed that it finds. It compares names in lower case, because gh matches them regardless of case. If gh can't list the labels, the block files the issue without them. Creating the two labels in microcosm and policyengine-uk-data would also unblock filing, but it would change two other teams' repos. If either repo adds one, report-logger applies it from then on. Dropping --label altogether would unblock filing too, but it would also lose calibration where a repo has it. policyengine-us and policyengine-uk both do, and an analyst can send an issue to either with --log-to issue:<repo>.

Canada

PolicyEngine has no Canada microdata. policyengine.py's 6.16.0 bundle lists uk and us as its only countries, policyengine-canada's one dataset holds three people, and microsim-runner supports only household calculations for Canada. No repo builds the data a Canada issue would concern, so report-logger files none and the archive keeps the report. An analyst can still file one with --log-to issue:<repo>. report-logger now names PolicyEngine/policyengine-canada for structural Canada issues, because filling policyengine-{country} with ca gives policyengine-ca, which doesn't exist either.

What changes

  • In report-logger.md, the issue-filing bash leaves DATA_REPO empty for ca, as for any country but us and uk, and then calls gh not at all. For us and uk it runs gh label list --repo "$DATA_REPO" --limit 1000 --json name --jq '.[].name | ascii_downcase' and passes --label for each wanted name in the output. The routing prose, the shortlist and verdict-default tables and the frontmatter description say the same, and tell the agent to list a Canada data issue under skipped in its output.
  • The differential tests in test_country_routing.py run the whole block with a stand-in gh that records each call, and check the issue's repo and labels against the Python reference. They run under bash and, when the machine has it, zsh, since Claude Code on a Mac can run the agent's shell commands in zsh. One property test makes the stand-in fail gh label list and checks that the block still files the issue. issue_labels restates the label rule, and the strict xfail for ca goes.
  • The workflow's log_to description, reform-pipeline.md, docs/structured-output-and-chaining.md and tests/pipeline/README.md repeat the new routes.

Evidence

Claim Source
gh resolves --label names in LabelsToIDs, which compares them with strings.EqualFold and returns '<name>' not found for a missing one, and AddMetadataToIssueParams wraps that error as could not add label params.go:106-109, queries_repo.go:809-823
gh issue create returns on that error before it calls api.IssueCreate create.go:456-478
gh reads --label as a pflag string slice and splits each value at commas create.go:186, pflag v1.0.10 string_slice.go:22-29
gh label list fetches 30 labels unless --limit says otherwise, and policyengine-us has 113 label/list.go:74; gh label list --repo PolicyEngine/policyengine-us --limit 1000, 2026-11-08
microcosm and policyengine-uk-data have neither label; policyengine-us and policyengine-uk have calibration and not auto-filed GET /repos/PolicyEngine/<repo>/labels, 2026-11-08
For every letter of calibration and auto-filed, strings.EqualFold matches nothing beyond its ASCII upper and lower case, so a lower-case comparison of ASCII names agrees with gh a scan of every Unicode code point with Go 1.27.1, 2026-11-08 (only k and s match more, U+212A and U+017F)
PolicyEngine/policyengine-ca-data, policyengine-canada-data and policyengine-ca return 404, and policyengine-canada exists GitHub API, 2026-11-08
policyengine.py's bundle lists uk and us as its only countries manifest.json:9-16
policyengine-canada's one dataset, CountryTemplateDataset, holds three people in two households country_template_dataset.py
microsim-runner tells the analyst that Canada has no microdata and supports only household calculations microsim-runner.md:310

Invariants

test_country_routing.py checks these for any dataset string and any set of repo labels that Hypothesis generates:

  • routing returns None or one PolicyEngine/<repo> name;
  • only INVESTIGATE and structural verdicts open an issue, and INVESTIGATE opens one only for us and uk;
  • structural routing and US INVESTIGATE routing ignore the dataset;
  • UK INVESTIGATE routing lands in microcosm or policyengine-uk-data, and reads only the last path segment left after it drops any @revision;
  • the issue carries each wanted label the repo has in any letter case and no other label, so gh never meets a label the repo lacks;
  • adding labels to the repo never removes one from the issue, and swapping a label's ASCII case changes nothing;
  • report-logger's block files the issue in the reference's repo with the reference's labels, reads that same repo's labels first, and calls gh not at all for ca;
  • when gh label list fails, the block still files the issue, without labels.

Each of these edits to the block makes the differential tests fail:

  • restoring --label "calibration,auto-filed";
  • restoring the old ca route;
  • dropping ascii_downcase, or grep's -x;
  • reading labels from a fixed repo;
  • passing "${LABEL_ARGS[*]}";
  • dropping --limit 1000, the frs_* pattern or the @revision strip;
  • exiting when gh label list fails.

Adding grep's -i, which changes nothing once jq lowers the names, passes.

Tests

uv run pytest --ignore=tests/test_skill_examples.py passes 319 tests, where it passed 256 and xfailed the ca case before. scripts/build_claude_wrapper.py copies report-logger.md into the wrapper byte for byte.

Order

#110 rewrites the same bash block, so this branch starts from #110's head (b4f7854) and the PR targets #110's branch for now.

Before merge

  • Rebase onto main after Route UK data issues to the repo that builds the run's dataset #110 merges, check that the PR targets main, and rerun the suite.
  • Regenerate dashboard/src/data/manifest.json, which carries report-logger's description.
  • Get an independent review that agrees at head.
  • Squash-merge on green checks and confirm that the Sync Claude Wrapper run carries the change to policyengine-claude.

🤖 Generated with Claude Code

@MaxGhenis
MaxGhenis force-pushed the uk-data-issue-routing branch from 597ab30 to b4f7854 Compare November 8, 2026 20:30
MaxGhenis and others added 2 commits November 8, 2026 15:33
…file none for Canada

gh looks up every --label name among the target repo's labels before it
creates an issue, and stops without creating one when a name is missing
(cli/cli pkg/cmd/pr/shared/params.go, api/queries_repo.go). report-logger
passed --label "calibration,auto-filed", and on 2026-11-08 neither
PolicyEngine/microcosm nor PolicyEngine/policyengine-uk-data had either
label, so every auto-filed data issue failed. The block now lists the
repo's labels and passes only the ones it has. It compares names in lower
case, because gh ignores case when it matches them.

The block sent ca data issues to policyengine-ca-data and the prose to
policyengine-canada-data, and neither repo exists. PolicyEngine has no
Canada microdata: policyengine.py's bundle holds data for us and uk only,
and policyengine-canada's one dataset is a three-person template. So
report-logger files no data issue for ca, and the archive keeps the
report. The prose now names policyengine-canada for structural ca issues,
since policyengine-{country} gave policyengine-ca, which doesn't exist
either.

test_country_routing's differential tests now run the whole issue-filing
block, under bash and zsh, with a stand-in gh that records its calls, and
compare the issue's repo and labels with the Python reference. The strict
xfail for ca goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne's Canada data route

reform-pipeline still sent other countries' data issues to
policyengine-{country}-data, which gives policyengine-canada-data for ca, a
repo that doesn't exist. It now says report-logger files none for ca.

report-logger's block already files the issue without labels when
`gh label list` fails; report-logger.md now says so, and a property test
fails the stand-in's label lookup for any country, dataset and label set
and checks that the block still files the reference's issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the report-logger-labels-ca branch from 3175ce7 to c3a5ffc Compare November 8, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant