Skip to content

Infer target repos from whole-token mentions in the whole body - #109

Draft
MaxGhenis wants to merge 3 commits into
manifest-pyyamlfrom
repo-coverage
Draft

MaxGhenis wants to merge 3 commits into
manifest-pyyamlfrom
repo-coverage

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes how scripts/build_manifest.py decides which PolicyEngine repos each skill, agent and command targets: the follow-up that #107's section 5 left out. infer_target_repos fills each artifact's target_repos in dashboard/src/data/manifest.json, and repo_coverage and some registry owners follow from it. It matched aliases as case-insensitive substrings, matched a repo name inside a longer token, and read only the first 6,000 characters of a body. This PR removes 107 of its 267 artifact-to-repo links and adds 61.

This PR stacks on #107 (manifest-pyyaml). When #107 merges, I rebase onto main, regenerate the manifest and point this PR at main. It stays a draft until then and until an independent review agrees.

1. The rule

RepoMatcher reads an artifact's name, description, whole body and category from left to right. At each position it takes the longest term that stands there as a whole token, records that term's repo, and carries on after it. The terms are each repo name in the inventory, its import name (policyengine_us) and the aliases in REPO_ALIASES.

  • Whole tokens. A letter, digit or underscore on either side makes a different token, so "SSI" in "missing" and "PIP" in "pipeline" don't count.
  • Slugs. Repo names, and aliases with a hyphen or a dot in them ("app-v2", "api.policyengine.org"), are slugs. A hyphen on either side, or a dot before, makes a longer slug or a hostname, so "policyengine-model-development" doesn't name policyengine-model and "v2.api.policyengine.org" doesn't name policyengine-api. A dot after a slug ends it, so "policyengine-us.git" and a sentence-ending "policyengine-us." count.
  • Words. Every other alias is a word or a phrase, and a hyphen leaves it whole: "SNAP-eligible" and "non-SSI" count.
  • Longest first. "policyengine-us-data" names policyengine-us-data and never policyengine-us. "api.policyengine.org" names policyengine-api, not the web app.
  • Case. A term in capitals counts only in capitals; every other term counts in any case (section 2).
  • Order. The result lists the repos named outright in inventory order, then the repos that only an alias implies, in REPO_ALIASES order, each once. The old function used the same order.

The old function ran one regex per repo and one substring test per alias. The new one runs str.find per term and checks the characters on either side, so it never builds a regex from repo names. A full build takes about 2 seconds here.

2. Decisions

Case. An acronym in lowercase is usually code or another word: uv pip install, foo.cpp, OBBBA's 2030 "snap-back", a ctc filter tag. So a term written in capitals (PIP, SNAP, CRM) counts only in capitals. I measured the alternative on today's 80 artifacts: counting the capitals aliases in any case adds 7 links. Four of them come from pip and tag policyengine-uk, one comes from the ri-ctc-calculator-backend service name and tags policyengine-us, and two are arguable (the parameter file gov/irs/credits/ctc.yaml in reference-validator's example table, and the ctc/eitc filter tags in /prior-analysis). Model variables and parameter paths such as tax_unit.eitc and gov.usda.snap appear only in artifacts that name policyengine-us anyway, so capitals-only drops none of those links.

Every other term, repo names included, counts in any case. GitHub resolves repo names in any case (gh api repos/policyengine/aca-calc returns PolicyEngine/ACA-Calc), prose capitalizes them, and trigger lists write phrases in lowercase ("universal credit"). The old function matched repo names only as written; ignoring case adds 3 links, all real: "a new PolicyEngine-US state benefit" in encode-policy-v2, and aca-calc in /create-dashboard and /new-tool.

The cap. Gone. A cap at any position breaks "adding text never removes a repo", because text added near the top pushes a later mention past it. 47 of the 80 artifacts have bodies longer than 6,000 characters, and reading them whole adds 36 links. 33 are plain mentions of the repo, such as policyengine-household-api in the cloud-costs skill's table of past incidents. The other 3 come from the limits in section 6.

The alias table.

  • policyengine.org and app-v2 now point at policyengine-app-v2, not the legacy policyengine-app. The policyengine-app skill describes app-v2 as "the site served at policyengine.org".
  • api.policyengine.org now points at policyengine-api. Before, the substring policyengine.org inside it tagged the legacy web app.
  • "frontend" no longer counts. The generic word tagged three dashboard artifacts as the legacy app.

Registry owners. owner_for_artifact now treats the country-models category as country-model work. Two agents in that category, model-worker and review-worker, owed their only country-model link to a false match, so without this change they would move to the generic owner.

3. What changes

Only target_repos, registry_owner and the per-repo counts in repo_coverage change. Links between artifacts and repos, by repo kind:

Repo kind Before After Added Removed
country-model 129 77 5 57
platform 78 90 38 26
other 26 9 2 19
interactive-instance 13 17 8 4
library 7 7 0 0
research-platform 7 8 2 1
internal 3 4 1 0
data-pipeline 2 7 5 0
long-lived-tool 2 2 0 0
All 267 221 61 107

69 of the 80 artifacts gain or lose a link. A 70th, pr-pusher, keeps its repos in a new order: its policyengine-us link now comes from the repo's name, not an alias, so it sorts with the named repos. Why each link went:

Removed because Links Examples
an alias sat inside a longer word 55 "SSI" in missing, guessing, permissions, Accessibility, classifies; "PIP" in pipeline, piped; "GIS" in registry, registered, legislative, legislature; "SNAP" in snapshot
a repo name sat inside a longer token 33 policyengine-model in policyengine-model-development (18); policyengine-app in policyengine-app-v2/... (7); atlas in policyengine-atlas-pbif-grant.md; poverty-dashboard in child-poverty-dashboard
the web-app aliases now name policyengine-app-v2 14 "embeds in policyengine.org", api.policyengine.org/calculate
"frontend" no longer counts 3 "Wires frontend components to backend"
an acronym in lowercase 2 "bare pip", "non-uv pip installs"
Added because Links Examples
the mention sat past the old 6,000-character cut 36 policyengine-household-api and policyengine-api in the cloud-costs skill's table of past incidents; state-legislative-tracker in microsim-runner
the web-app aliases now name policyengine-app-v2 16 "Next.js multi-zones behind policyengine.org"
api.policyengine.org now names policyengine-api 5 "call api.policyengine.org" in microsim-runner
an import name 2 'policyengine_uk' : 'policyengine_us' in dashboard-integrator
a repo name in another case 2 aca-calc

By repo:

Repo Kind Before After Added Removed
policyengine-us country-model 73 49 1 25
policyengine-app platform 33 10 1 24
policyengine-model other 18 0 0 18
policyengine-canada country-model 25 8 1 18
policyengine-uk country-model 31 20 3 14
atlas interactive-instance 1 0 0 1
ctc-calculator interactive-instance 2 1 0 1
marriage other 3 2 0 1
policyengine-api-v2 platform 4 3 0 1
poverty-dashboard interactive-instance 1 0 0 1
uk-autumn-budget-dashboard interactive-instance 1 0 0 1
CRM other 3 4 1 0
calibration-diagnostics research-platform 7 8 2 1
newsletters internal 1 2 1 0
policyengine-household-api platform 0 1 1 0
snap-bbce-repeal other 0 1 1 0
uk-spring-statement-2026 interactive-instance 0 1 1 0
ACA-Calc interactive-instance 0 2 2 0
policyengine-claude platform 1 3 2 0
policyengine-uk-data data-pipeline 0 2 2 0
south-carolina-2026-tax-changes interactive-instance 2 4 2 0
policyengine-core platform 6 9 3 0
policyengine-us-data data-pipeline 2 5 3 0
policyengine.py platform 4 7 3 0
state-legislative-tracker interactive-instance 1 4 3 0
policyengine-api platform 12 17 6 1
policyengine-skills platform 2 7 5 0
policyengine-app-v2 platform 13 30 17 0
  • Repos that any artifact targets: 39 → 40. Five join (ACA-Calc, policyengine-household-api, policyengine-uk-data, snap-bbce-repeal, uk-spring-statement-2026). Four leave (atlas, policyengine-model, poverty-dashboard, uk-autumn-budget-dashboard), and in each case the only link came from a longer token.
  • Tooling-relevant repos that no artifact targets: 35 → 33.
  • Registry owners: /prior-analysis and /prior-scores move from "Country model engineering" to "PolicyEngine". Their only country-model links came from "pipeline", "registry", "not-possible" and the ctc/eitc filter tags.

The Catalog's repo filter and chips, the drawer's "Target repos" list and the Workflows page's repo chips read target_repos. The drawer, the Catalog and the Cleanup page show registry_owner. No page reads repo_coverage yet.

4. Invariants

These hold for every input, and the tests check each one:

  • A term with a letter, digit or underscore on either side never counts. Nor does a slug with a hyphen on either side or a dot before it (Hypothesis, any generated term).
  • A term between common delimiters (spaces, brackets, backticks, quotes, slashes, sentence punctuation, .git, 's) always counts (Hypothesis).
  • A hyphen on either side leaves a word alias whole (Hypothesis).
  • A term in capitals counts only as written; every other term counts in lowercase, capitals, title case or swapped case (Hypothesis).
  • Inserting lines never removes a repo, and inserting lines that name no repo changes nothing. The generated lines include one longer than the old 6,000-character cut (Hypothesis).
  • The order of lines never changes the result (Hypothesis).
  • The matcher agrees with reference_targets, a plain character-by-character scan of the same rule. The tests check this for the real table, and for random small tables whose terms overlap, nest, repeat and start one another (Hypothesis, two differential tests).
  • Every repo name in the inventory, and every import name, targets exactly its own repo. Every alias targets a known repo (exhaustive over the inventory).
  • A table with an empty alias, or an alias that repeats a repo name, an import name or another alias in any case, raises ValueError.
  • The result lists each repo once: named repos in inventory order, then alias-only repos in REPO_ALIASES order.

5. Tests

  • tests/test_target_repos.py: 64 tests, 8 of them Hypothesis properties, in under 30 seconds. Most examples come from this repo's own skills, agents and commands, with at least one for each kind of false positive in section 3.
  • I ran every property with 4,000 examples, not the default 100 or 300. All passed.
  • Mutation check. Each of these 18 edits to the matcher makes at least two tests fail, at least one of them a fixed example:
    • dropping the left boundary, or the right one;
    • letting a hyphen follow a slug, or a dot precede one;
    • treating every alias as a word, or as a slug;
    • letting capitals terms ignore case, making every term exact, or making repo names exact;
    • dropping the merge, letting the shortest term win, or listing alias-only repos first;
    • dropping import names, restoring the 6,000-character cap, or restoring the old alias table;
    • accepting duplicate aliases, matching only a term's first occurrence, or restoring substring aliases.
  • uv run pytest --ignore=tests/test_skill_examples.py: 286 passed under Python 3.15.2 and under 3.11.15 (macOS).
  • uv run python scripts/build_manifest.py --check passes under Python 3.11 and 3.15.

6. Limits this PR leaves

  • Repo names that are common words count wherever the word stands alone: strategy, marriage, newsletters, atlas, pulse, mortality. Reference-validator's "News articles/newsletters" now tags the newsletters repo. The old function had the same problem.
  • Skills and agents that share a repo's name. "See the policyengine-app skill" tags the legacy policyengine-app repo, though that skill covers app-v2. A mention of the calibration-diagnostics agent tags the calibration-diagnostics repo.
  • Research posts. A policyengine.org/us/research/... URL tags policyengine-app-v2, which serves the posts, even when the artifact only reads them.
  • v2.api.policyengine.org, the API v2 alpha, has no alias.

Other PRs that touch the manifest

Rebase onto main and regenerate with uv run python scripts/build_manifest.py rather than hand-merging the manifest.

axiom: n/a: tooling (dashboard manifest builder), no policy change.

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits November 7, 2026 08:44
- build_manifest.py exits 1, writing nothing, when it can't import PyYAML,
  instead of falling back to a subset parser that turned numbers into
  strings and dropped nested fields (which broke the Analyses page). It
  also stops, naming the file, on frontmatter that isn't a YAML mapping.
  The subset parser and the regex category fallback are gone.
- The dashboard's manifest hooks run the builder through uv. manifest:safe
  keeps the committed manifest when the build can't run.
- --check builds the manifest without writing it and exits 1 when the copy
  on disk differs anywhere but generated_at. A new CI job runs it on Linux
  under Python 3.11 and 3.14.
- IDF comes from decimal's correctly rounded ln rather than math.log, which
  rounds as each platform's C library does; shared_terms break ties by term.
- Overlap pairs stay in artifact order, so a one-skill edit rewrites only
  the scores it moves (#106's edit: 242 changed lines instead of 1,084).
  The drawer and the Duplicates view sort pairs by score themselves.
- Tests: a builder without PyYAML, frontmatter types and errors, --check,
  overlap order, guards against the built-in sum() and math.log, and a
  Hypothesis property that overlaps ignore artifact and word order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anifest

Python 3.15 has shipped (CPython is at 3.15.2), so the manifest job's
newer entry moves from 3.14 to 3.15. The job's comment called its pair
the oldest and newest Python the repo supports, which 3.14 no longer is.
The test job already runs 3.15.2. On macOS, the committed manifest
matches a fresh build under every Python from 3.11 to 3.15.

--check printed a traceback when the file on disk held JSON that wasn't
an object. It now says so and exits 1, and a test covers that case and
invalid JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
infer_target_repos matched aliases as case-insensitive substrings, so
"SSI" in "missing", "PIP" in "pipeline" and "GIS" in "registry" tagged
the country models. It matched a repo name inside a longer token, so
policyengine-app-v2 tagged policyengine-app and the skill name
policyengine-model-development tagged policyengine-model. And it read
only the first 6,000 characters of a body.

Now a repo name, its import name or an alias counts only where it stands
as a whole token, and at each position the longest term wins. A term in
capitals (PIP, CRM) counts only in capitals; every other term counts in
any case, as GitHub resolves repo names. The matcher reads the whole
body. The web-app aliases point at policyengine-app-v2,
api.policyengine.org points at policyengine-api, and "frontend" no
longer counts. The registry owner rule treats the country-models
category as country-model work, so two agents that lost false links keep
their owner.

target_repos links: 267 -> 221 (+61 / -107) across 69 artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant