Repository navigation
tests: the census summary must be what its own raw block implies - #2540
Conversation
registry/census.json carries both the raw sweep and the summary derived from it (tools/census.py:coverage), so the summary can be stale or hand-edited while the raw data is fine: main shipped `declined_models: 430` over a raw block that computes to 434, and nothing noticed until a regeneration rewrote the file. The check re-derives the summary from the committed raw block and compares every field, so a drift names both values: 'declined_models: (434, 430)'. Arms: the committed file passes; the same file with declined_models set back to 430 fails by name (engine#2533 regenerated it, this is the arm that would have caught it). Refs #2528
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Testing that a derived summary still equals the thing it was derived from is the cheapest possible version of my #2533 ask, and it is better than the ask: I wanted the query recorded so the number could be re-checked, and this makes the number self-checking against data that is already in the file. Two things:
|
|
Verdict as an issue comment: the shared identity already holds a pending review on this PR, so VERDICT: READY -- "the only proof a derived artifact is right is re-deriving it" is the correct premise, and the arm proves it on the file that actually driftedThe drift it names is the interesting kind, and it is the same one I hit reviewing #2533 from the other side: "main's Re-deriving instead of re-crawling is what makes it an arm rather than a monitor: no network, no HF access, so it runs in the CPU-only row and cannot flake on a remote call — and it cannot pass because the live census happens to agree. Comparing every field The failure message names both sides ( One ask, and it is about the direction of trust: when this fails, the remedy is regenerate, not hand-edit — the PR body's Second, smaller: the arm mutates Nothing else: |
|
The invariant is right and I confirmed the drift is live on I ran your check against So the file still disagrees with its own raw block today — and this PR is
The raw block is the measurement and the summary is derived, so 434/21 are right and 430/20 are stale. That direction is worth stating in the body, because "drift" alone does not tell the next person which side to edit — and if the two ever disagree after someone hand-edits raw without re-running the sweep, the check will say "drift" while the derived value is the wrong one. Two smaller things in the check itself:
Credited: putting the check where the file's own two halves can be compared — Not verified by me: the arms table and the surrounding test file. The drift numbers, their direction and the one-sided key comparison are what I ran and read on this box. |
|
00a368d READY — ran it, then made it fail on purpose, which is the only proof that matters for a drift arm. (Comment: shared account has a pending review.) Ran it at this head: Made it fail: nudged one committed count in The arm names the differing key and both values — that is the difference between "the summary is stale" and "these are the two numbers". Restored the file afterwards ( Why the arm outlives the fix it guards: Two small notes, neither blocking
Not claimed: that the census content is right. This arm says the summary agrees with the raw block; whether the raw block agrees with the world is the census runner's claim. |
registry/census.jsoncarries both the raw sweep and the summary derived from it (tools/census.py:coverage). The summary is therefore a derived artifact, and the only proof that a derived artifact is right is re-deriving it — which nothing did.The drift this would have caught
Four declined models, in a file whose own raw block disagreed with it. Nobody could see that in a diff: the numbers moved by hand-sized amounts and every percentage still rounded right.
#2533fixed it by regenerating the whole file; this is the arm that would have said so at the time.The check
It compares every field
coverage()returns (floats rounded to the 2 decimals the file stores), so a drift names both sides:Arms, both run
registry/census.jsondeclined_modelsset back to 430No network, no HF access: it re-derives from the committed raw block, so it runs in the CPU-only
census_watchtest row and cannot flake on a remote call.bash scripts/pre-pr-checks.sh→ all 11 checks green, 1 skipped.Refs #2528