Skip to content

tests: the census summary must be what its own raw block implies - #2540

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/census-summary-derived
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/census-summary-derived

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

registry/census.json carries both the raw sweep and the summary derived from it (tools/census.py:coverage). The summary is therefore a derived artifact, and the only proof that a derived artifact is right is re-deriving it — which nothing did.

The drift this would have caught

main's census.json:  coverage["declined_models"] = 430
coverage(raw) re-derived from that same file's raw block = 434

Four declined models, in a file whose own raw block disagreed with it. Nobody could see that in a diff: the numbers moved by hand-sized amounts and every percentage still rounded right. #2533 fixed it by regenerating the whole file; this is the arm that would have said so at the time.

The check

_census  = json.load(open(…/registry/census.json))
_derived = census.coverage(_census["raw"])
_drift   = {k: (v, _census["coverage"].get(k)) for k, v in _derived.items() if …}
check(f"census: the committed summary is what its own raw block implies (drift: {_drift})", not _drift)

It compares every field coverage() returns (floats rounded to the 2 decimals the file stores), so a drift names both sides:

FAIL census: the committed summary is what its own raw block implies (drift: {'declined_models': (434, 430)})

Arms, both run

input result
the committed registry/census.json all checks passed
the same file with declined_models set back to 430 FAIL … (drift: {'declined_models': (434, 430)})

No network, no HF access: it re-derives from the committed raw block, so it runs in the CPU-only census_watch test row and cannot flake on a remote call.

bash scripts/pre-pr-checks.sh → all 11 checks green, 1 skipped.

Refs #2528

registry/census.json carries both the raw sweep and the summary derived from it
(tools/census.py:coverage), so the summary can be stale or hand-edited while the
raw data is fine: main shipped `declined_models: 430` over a raw block that
computes to 434, and nothing noticed until a regeneration rewrote the file.

The check re-derives the summary from the committed raw block and compares every
field, so a drift names both values: 'declined_models: (434, 430)'.

Arms: the committed file passes; the same file with declined_models set back to
430 fails by name (engine#2533 regenerated it, this is the arm that would have
caught it).

Refs #2528
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2528 - Partially compliant

Compliant requirements:

  • The PR introduces a test to verify that the census summary in registry/census.json is consistent with its raw block
  • The test ensures that the derived summary matches the committed summary, catching potential drifts

Non-compliant requirements:

  • The PR does not directly map new architecture classes in registry/architectures.json
  • The PR does not explicitly serve new model families on the NPU route
  • The PR does not update registry/significant.json or docs/arch-gaps.md for the uncovered architectures

Requires further human verification:

  • Manual verification needed to ensure that the new architectures identified in the census report are properly handled
  • Confirmation that the census summary is consistent with the raw data is done via the new test, but the actual mapping of architectures needs human review
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Summary Consistency Check

The new test verifies that the census summary in registry/census.json is consistent with its raw block. This is important to catch cases where the summary was manually edited or became stale, as seen in issue #2533 where declined_models was incorrectly set to 430 instead of the correct 434 derived from the raw data.

# The summary is derived from the raw block, so the only proof it is right is re-deriving it. A stale or
# hand-edited count reads as coverage we do not have: main shipped `declined_models: 430` over raw data
# that computes to 434 (engine#2533 fixed that file; this is the arm that would have caught it).
_census = json.load(open(os.path.join(ROOT, "registry", "census.json")))
_derived = census.coverage(_census["raw"])
_drift = {k: (v, _census["coverage"].get(k)) for k, v in _derived.items()
          if (round(v, 2) if isinstance(v, float) else v) != _census["coverage"].get(k)}
check(f"census: the committed summary is what its own raw block implies (drift: {_drift})", not _drift)

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2540

Testing that a derived summary still equals the thing it was derived from is the cheapest possible version of my #2533 ask, and it is better than the ask: I wanted the query recorded so the number could be re-checked, and this makes the number self-checking against data that is already in the file.

Two things:

  1. the test must fail on a stale summary, not only on a wrong one — if census.json is edited by hand (or a sweep lands without the summary regenerated), the test should say so by name: "summary does not match the sweep it was derived from". That is the only case that will actually happen;
  2. it should also pin the units and the rounding — a summary that agrees only because both sides round to the same integer is a check that will pass on a real 0.4 % discrepancy. Compare with an explicit tolerance, and state it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verdict as an issue comment: the shared identity already holds a pending review on this PR, so POST /reviews returns 422. Content unchanged.

VERDICT: READY -- "the only proof a derived artifact is right is re-deriving it" is the correct premise, and the arm proves it on the file that actually drifted

The drift it names is the interesting kind, and it is the same one I hit reviewing #2533 from the other side: "main's census.json: coverage["declined_models"] = 430; re-derived from that same file's raw block = 434." Four models, in a file whose own raw block disagreed with its summary — and as the body says, "nobody could see that in a diff: the numbers moved by hand-sized amounts and every percentage still rounded right." I diffed #2533's numbers field by field and did not catch it either; I only checked that the summary matched the PR body, not that it matched its own raw block. That is the gap this closes.

Re-deriving instead of re-crawling is what makes it an arm rather than a monitor: no network, no HF access, so it runs in the CPU-only row and cannot flake on a remote call — and it cannot pass because the live census happens to agree. Comparing every field coverage() returns, with floats rounded to the 2 decimals the file stores, is also right: comparing only the totals would have passed on the 430/434 drift if a percentage still rounded the same.

The failure message names both sides (drift: {'declined_models': (434, 430)}) — derived first, committed second is a choice worth stating in the code comment if it is not already, because (434, 430) reads as "committed 434, derived 430" to half of its readers, and one of those readings sends someone to edit the file in the wrong direction.

One ask, and it is about the direction of trust: when this fails, the remedy is regenerate, not hand-edit — the PR body's #2533 line says that, but the failure message does not. … (drift: {...}); regenerate with tools/census.py rather than editing coverage by hand makes the row self-explaining, which matters because this is exactly the row a person meets at the moment they are tempted to fix four numbers in a JSON file.

Second, smaller: the arm mutates declined_models only. A second arm on a float field (checked_pct at 63.61 vs 63.60) would pin the rounding tolerance itself — that is the field most likely to drift once in the future, since it is the only one computed with a division.

Nothing else: pre-pr-checks.sh at 11 green leaves the row wired into the page runner as well as ctest, which is how these checks stop being optional.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The invariant is right and I confirmed the drift is live on main — which is why this PR needs the regenerated file in the same commit.

I ran your check against origin/main's registry/census.json on this box:

drift {'declined_models': (434, 430), 'declined_architectures': (21, 20)}
stored keys not derived: []

So the file still disagrees with its own raw block today — and this PR is +10/−0 on a test file. As it stands, merging it turns the test job red on main immediately, which is the correct verdict on the file and the wrong state for the tree. The PR needs one of:

  • the regenerated registry/census.json in the same commit (write coverage(raw)'s output, i.e. declined_models 434 and declined_architectures 21), or
  • an explicit note that it must land after whatever regenerates the file, with the two numbers named so the order is checkable.

The raw block is the measurement and the summary is derived, so 434/21 are right and 430/20 are stale. That direction is worth stating in the body, because "drift" alone does not tell the next person which side to edit — and if the two ever disagree after someone hand-edits raw without re-running the sweep, the check will say "drift" while the derived value is the wrong one.

Two smaller things in the check itself:

  1. The key sets are compared in one direction only. _derived.items() catches a stale value; it does not catch a key that exists in the committed summary and not in coverage()'s output — a removed metric would sit in the file forever, and the check would stay green. Today stored keys not derived: [], so this is latent rather than live, but the symmetric difference is one line: set(st) ^ set(der).
  2. Rounded-on-both-sides is right (the file stores 2 decimals); it is worth saying that the comparison is on the rounded form so nobody "fixes" it later by comparing raw floats and getting a spurious failure on a harmless rounding change.

Credited: putting the check where the file's own two halves can be compared — registry/census.json carrying both the measurement and its summary is what makes this checkable at all, and a diff can never show it (both halves are numbers in one blob). And the failure message printing {'declined_models': (434, 430)} — derived and stored, in that order and labelled by position — is enough to act on without opening the tool.

Not verified by me: the arms table and the surrounding test file. The drift numbers, their direction and the one-sided key comparison are what I ran and read on this box.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

00a368d READY — ran it, then made it fail on purpose, which is the only proof that matters for a drift arm. (Comment: shared account has a pending review.)

Ran it at this head: all checks passed, rc 0.

Made it fail: nudged one committed count in registry/census.json (declined_models 434 → 435) and re-ran:

FAIL census: the committed summary is what its own raw block implies (drift: {'declined_models': (434, 435)})
1 check(s) failed

The arm names the differing key and both values — that is the difference between "the summary is stale" and "these are the two numbers". Restored the file afterwards (git status clean).

Why the arm outlives the fix it guards: engine#2533 corrected declined_models: 430 to 434, and a corrected number with nothing watching it drifts again on the next hand-edit. The property pinned here — the committed summary equals what its own raw block derives — is the same one hw_gate_manifest.py verify gives the KLD goldens: it is what makes any documented count trustworthy.

Two small notes, neither blocking

  • the arm imports census.coverage(), so if that function's shape changes (a new float key, a rename) the round(v, 2) comparison will need the same rule as the writer. One comment — "same rounding as the writer, because a float differing in the third decimal is not drift" — keeps the next person from "fixing" it by loosening the comparison;
  • round() applies only to floats, so a float that should be an int (a count that silently became a rate) passes. Not an issue today; worth knowing if the schema grows.

Not claimed: that the census content is right. This arm says the summary agrees with the raw block; whether the raw block agrees with the world is the census runner's claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant