[MaxGhenis]

Closes the gaps #105 left in scripts/build_manifest.py and the dashboard's build hooks: its follow-up items 1 to 3, plus the diff-size problem #106 hit. Draft until CI passes on Linux and an independent review agrees.

1. A missing PyYAML stops the build

• Without PyYAML, main() prints how to run the script through uv and exits 1 before it writes anything. parse_frontmatter raises the same message for direct callers.
• I deleted the subset parser (parse_yaml_subset). Every frontmatter block in the repo's skills, agents, commands and analyses parses as a YAML mapping, so nothing needs it, and its output broke the Analyses page. The regex scan for metadata.category in load_skills existed only for that path, and no skill used it, so it went too.
• Frontmatter that isn't valid YAML, or isn't a mapping, now stops the build with a FrontmatterError that names the file. Before, it dropped silently to the subset parser.
• In dashboard/package.json, manifest runs uv run python scripts/build_manifest.py. manifest:safe runs the same command and keeps the committed manifest when the build can't run.

Before this PR, /usr/bin/python3 scripts/build_manifest.py (macOS's Python 3.9.6, no PyYAML) exited 0. It wrote "child_poverty_pct_change": "-34.8" and "country": null into every analysis. Now:

build_manifest.py needs PyYAML, and /Applications/Xcode.app/Contents/Developer/usr/bin/python3 can't import it.
Run the script through uv, which installs the repo's dependencies:
    uv run python scripts/build_manifest.py
The script wrote nothing.

I ran the package.json scripts through bun, which runs the same script strings:

Case	Result
manifest, uv on PATH	builds through uv
manifest, no uv on PATH	uv: command not found, exit 127, manifest untouched
manifest:safe, no uv on PATH	keeps the committed manifest, exit 0
manifest:safe in a copy of dashboard/ alone	keeps the committed manifest, exit 0
builder under /usr/bin/python3	exit 1 with the message above, manifest untouched

2. CI checks that the manifest is fresh

• --check builds the manifest without writing it and compares it with the file on disk, with generated_at blanked in both. On a mismatch it prints the sections that differ, the first 200 diff lines and the command that fixes it, then exits 1. When the file on disk isn't valid JSON, or holds JSON that isn't an object, it says so instead of raising a traceback.
• A new manifest job in .github/workflows/test.yml runs --check on ubuntu-latest under Python 3.11 and 3.15, the oldest and newest Python the repo supports. The existing test job already runs the suite under 3.15.
• I couldn't run Linux here (colima's Docker context is down), so I took the platform out of the question instead of waiting to see what Linux does. compute_overlaps now takes IDF from the decimal module's correctly rounded ln, not math.log, which rounds however the platform's C library does. On macOS, math.log misses the correctly rounded value for 44 of the 79,799 IDF ratios (n + 1) / (df + 1) with 2 ≤ n < 400. So a manifest built on a Mac could fail the Linux check whenever the artifact count hit one of those ratios. None of them occurs at today's 80 artifacts, so no score changes.
• CI confirms it. On Ubuntu 24.04 (x86_64, glibc), both manifest jobs report that the manifest I built on macOS matches a fresh build: under CPython 3.11.18 and 3.15.2 at the head, and under 3.11.18 and 3.14.10 at the first push. In a fresh clone, editing one skill without rebuilding makes the same command exit 1 and list artifacts, overlaps as the sections that differ.
• On macOS (arm64), --check matches under Python 3.11.15, 3.12.14, 3.13.13, 3.14.7 and 3.15.2.
• shared_terms now break exact ties by term, so their order never depends on where a word first appears in a text. On main's current corpus that reorders 7 of the 326 lists; in 3 of them a different term makes the 8-term cut.

3. A test guards fsum

test_overlaps_never_use_builtin_sum swaps the built-in sum(), as build_manifest sees it, for one that returns 1% more for floats, then checks that compute_overlaps returns the same pairs. A manifest comparison would catch a return to sum() only on Python 3.11, where sum([0.1] * 10) gives 0.9999999999999999 (3.12 on gives 1.0). This test fails on every version. test_overlaps_never_use_the_c_librarys_log does the same for math.log.

Mutation check: each of these edits to the builder makes its test fail:

• sum() for the norm
• sum() for the score
• math.log for IDF
• dropping the term tie-break
• sorting pairs by score
• swallowing bad YAML
• a --check that ignores content
• removing both PyYAML guards

Removing only main()'s PyYAML check changes nothing a user sees, since parse_frontmatter still raises the same message, so that test still passes. test_check_fails_cleanly_on_a_malformed_manifest fails against this PR's first push, whose --check raised AttributeError on a manifest holding [].

4. Overlap diffs shrink

• compute_overlaps no longer sorts pairs by score. They follow artifact order: by a, then by b.
• dashboard/src/data.ts exports overlapsByScore, a stable sort by score, and Drawer.tsx (first 8) and Duplicates.tsx (first 50) read it. A stable sort keeps tied scores in artifact order, the order the old manifest gave them, so both views show exactly what they showed before. I checked:
  • the whole list: on main's current corpus, a stable sort of this PR's 326 pairs by score reproduces main's builder's order exactly, across 26 adjacent ties;
  • the Duplicates raw view: 29 same-kind pairs at 30% or more, checked in the browser at the first push and recomputed at the head;
  • the policyengine skill's drawer: the same 8 pairs in the same order, three of them tied at 23%, checked the same two ways.
• I rebuilt Correct how the data skill says the US default got its 57,240 households #106's edit both ways: with pairs ordered by score it changes 1,084 manifest lines, and in artifact order 242.
• Stop the data skill steering agents to microimpute and uk-data #108 edited the data skill without rebuilding the manifest, so this PR's rebuild also carries that edit: the edited artifact, one new overlap pair, 113 moved scores and a repo_coverage change. Built from the same corpus, main's builder and this branch's give the same 326 pairs and the same scores. The manifest diff changes about 3,900 lines, nearly all from reordering overlaps once.

5. Left out: the 6,000-character cap in infer_target_repos

47 of the 80 artifacts have bodies longer than 6,000 characters. Dropping the cap adds 66 artifact-to-repo links across 26 artifacts. It would also make a separate bug worse: aliases match as substrings. "SSI" inside "missing" tags 17 artifacts as policyengine-us, "PIP" inside "pipeline" tags 11 as policyengine-uk, and "GIS" inside "registry" or "legislative" tags others as policyengine-canada. Word-boundary aliases on their own remove 64 links. Both changes move repo_coverage enough to need their own review, so they belong in a follow-up.

Invariants

• A rebuild with no content change rewrites only generated_at, whatever the hash seed (tests), the Python version from 3.11 to 3.15 (macOS runs of each; CI checks 3.11 and 3.15 on Linux), or the platform (CI on Linux against the manifest I built on macOS).
• The builder either writes a manifest built from fully parsed YAML, or writes nothing and exits non-zero.
• --check never writes.
• overlaps lists each pair once, a before b, in artifact order.
• A pair's score and shared_terms depend only on the words in the corpus, not on the order of artifacts or of words within a text (Hypothesis property test).
• Overlap scores don't depend on how the built-in sum() or the C library's log rounds.

Tests

• uv run pytest --ignore=tests/test_skill_examples.py: 222 passed (Python 3.14.7, macOS). CI's test job runs the same suite under CPython 3.15.2 on Linux.
• tests/test_build_manifest.py: 19 passed under Python 3.11 too.
• bun run build (tsc -b && vite build) passes.
• In the built dashboard, the Analyses page renders its numbers, and the console shows no errors.

Other PRs that touch the manifest

Rebase onto main and regenerate with uv run python scripts/build_manifest.py rather than hand-merging; this PR did that on top of #108. The open drafts #100 and #109 to #112 all change files that feed the manifest, so once this lands, the new manifest check will fail each of them until it does the same.

axiom: n/a: tooling (dashboard manifest builder, hooks and CI), no policy change.

🤖 Generated with Claude Code