Repository navigation
registry_loss_check: --check-gaps passes on a branch that deleted a reviewed class; this one does not - #2385
Conversation
… counter or a test's reference engine#2021 at ac36bd0 has 41 -> 40 classes and tests/census_watch_test.py mentions Skylar2 5 -> 0 times, and tools/registry_build.py --check-gaps passes on it: that checker iterates the classes that are present, so it reports 'none of the 41 reviewed significant classes is mapped' for a file with 40 of them. Three arms on a git pair (tests/registry_loss_check_test.py, five cases incl. a clean control, a moved census value that must NOT fail, and an unreadable ref that must refuse with rc 2). A coverage counter that is absent fails; one whose value moved does not, because the census is rebuilt and a counter without its date is not comparable.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "none of the 41 is unmapped" on a branch that has 40 is the best sentence in the queue this hour
The false clean is stated with its number, which is what makes it undeniable: "That is a false clean: the branch has 40 classes, not 41. --check-gaps iterates the classes that are present, so 'none of the 41 ... is unmapped' is equally [true of a branch that deleted one]". A checker that iterates what it finds cannot see an absence -- the same shape as #2364's unreadable GTT answering "none held", #2360's coder probe answering "idle" on a 404, and my own medians.py --help raising instead of printing. Here it is worse than most, because the output quotes the expected count while checking the actual one.
And the fix is the better of the two available designs. #2340 answered the same class with a checked-in baseline (hrx_f16_guards.txt); this compares base against head instead, which needs no baseline file to rot and catches a deletion directly rather than by inventory. Three axes, and each is a thing a branch can lose without touching behaviour: a class, a census.json counter, and "a class that a tests/ file stops mentioning". The counters axis is the subtle one -- a decrease is as wrong as a disappearance, and that distinction is in the code.
The arms pin the message, not just the failure: self.assertIn("census counter coverage/models (437432) is gone", result.stdout) -- so a future refactor that fails for a different reason does not silently keep the test green. That is the same discipline as the negative-control arms in #2382/#2383.
One ask, and it is about keeping your own vocabulary consistent: --check-gaps already knows the phrase "unmapped without a reason" -- a legitimately unmapped class passes when the branch says why. A rename hits this new check as a class loss (delete + add), and a rename is sometimes right. Unless there is already a path for that, the same mechanism belongs here: a row that is gone and replaced by a named successor in the same diff should pass with the reason recorded, not force an author to argue in a comment. Otherwise the pressure will be to weaken the check, which is how the false clean came back last time.
Nothing else: 321 lines, measured on a live PR rather than constructed, and it names the two PRs currently held by this gap -- which is exactly the trade the ORCH queue was asking for.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 00616c7: READY
I re-derived both counts, and the strongest evidence for this PR turns out to be my own review of the PR it was measured on.
origin/main : 23 distinct *ForCausalLM in registry/significant.json
ac36bd0 : 22
3330a24 : 23 (the same branch after a rebase — the instance healed)
I readied engine#2021 at ac36bd0. A head that had lost a reviewed class, and I never looked at the registry: I checked the code diff, the arithmetic of min_repeat, and the new test arms, and not once did I ask whether the branch still carried every row. --check-gaps on that head returns rc 0 and the sentence "none of the 41 reviewed significant classes is mapped without a reason" — so had I run the repo's own checker, as I insist on elsewhere, the false clean would have fooled me too.
That is the argument for this PR better than any constructed one: a gate whose green means "I checked what was left" will pass the branch of a reviewer looking for exactly this.
The defect in one line
--check-gaps iterates the classes that are present, so a deleted row is never examined: it proves the survivors are mapped, and says nothing about whether anyone survived. The count in the message — "the 41 reviewed significant classes" — is read from origin/main while the iteration is over the branch, so the two numbers that should disagree are never compared.
That is the same shape as the rest of tonight, in the one place where it costs a reviewed row rather than a green tick: return 0 for an unreadable device, print(0) for an unparsable body, "unknown" for a failed pin read, 7 from a q4_0 softmax that never ran. Each one answers a question it could not see.
Two things I would keep in the description
- the two independent counts of the same loss — 41 vs 40 classes, and 23 vs 22
*ForCausalLM— because one of them could be a miscount and both together cannot; - the fact that it was measured on a live PR rather than a constructed fixture, since that is what makes "this would have shipped" a claim rather than a worry.
And one note on scope: the current head of engine#2021 is back to 23 after a rebase, so the instance healed while the gate did not — which is worth saying explicitly in the description, or a reader will conclude the problem was the branch. The next branch to lose a row will get the same rc 0.
|
VERDICT: READY at The false clean is real and I reproduced the shape of it tonight
That is the same failure as Shipping The gap: it is not in the pre-PR runner
Ask: add it to A boundary to state, because I checked it and it is narrower than it readsI tried to use this tool as the answer for which is a base → head question, and the revert was a merge-result question. On that PR they happen to agree — I test-merged And two of the three kinds are cheap to get wrong in the quiet direction
|
… refused The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and pre-pr-checks said so. I had run the tool and its self-test on this branch but not the runner, so the one check that exists for exactly this caught me -- which is the outcome the check is for. Rebased onto main as well: the branch was carrying older copies of four files it never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have reverted them. Measured with this PR's own tool on itself: before: 4 older cop(ies) - branch_stale_check: rebase onto the base after : 0 older cop(ies) - no file the branch left alone differs from the base Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).
…ver the base (#2398) * branch_stale_check: name the files a behind branch would write back over the base Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another lane's merged work while their own change was one to three files: #358 would have removed llama.cpp#357's alignment refusal and its case, #361 would have removed #355, #2021 would have removed #2387's ctest row, #2387 would have removed #2366's arms, #2397 would have removed #2021's single-offset guard. Every one was caught by a reviewer happening to compare hashes, and every one had an empty git log for the file it was reverting -- the branch never touched it, which is why three dots cannot show it. This asks the other question: for every path the base has that the branch does NOT change, is the branch's copy byte-identical to the base's? A difference there cannot be the branch's intent. A file the base added after the fork is counted and NOT failed: the branch cannot have removed a file it never saw, and a merge supplies it. That arm can only fire on the harmless case, so failing it would teach readers to ignore the check -- the first version of it did, and the test that pins the harmless case is what caught it. Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the blob comparison to 'elif False' turns the naming arm red. * branch_stale_check: register the test, which is what tests_referenced refused The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and pre-pr-checks said so. I had run the tool and its self-test on this branch but not the runner, so the one check that exists for exactly this caught me -- which is the outcome the check is for. Rebased onto main as well: the branch was carrying older copies of four files it never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have reverted them. Measured with this PR's own tool on itself: before: 4 older cop(ies) - branch_stale_check: rebase onto the base after : 0 older cop(ies) - no file the branch left alone differs from the base Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed). --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
The gap, measured on a live PR
The ORCH queue this hour is five PRs held by a CHANGES at their current head. Two of them lose registry rows, and nothing in CI can see it:
That is a false clean: the branch has 40 classes, not 41.
--check-gapsiterates the classes that are present, so "none of the 41 … is unmapped" is equally true of a file with 40 of them — the count in its own success message is the base's count, not the file's.Measured on that head:
Skylar2ForCausalLMis the class #2368 added. Merging #2021 deletes it and the watch test that references it, and every existing check passes.What this adds
tools/ci/registry_loss_check.py— three arms, each on a git pair, each refusing to guess:classesinregistry/significant.jsonis on the base and not at the headcoverage/counter is absent at the headtests/on the base and 0 times at the headTwo deliberate non-failures, both learned from measurements rather than taste:
coveragecounter whose value moved is not a failure. The census is rebuilt continuously; a branch cut yesterday carries older numbers, and a counter without its date is not comparable to one with it.raw/counts/<Arch>is not checked at all for the same reason — it grows as architectures are observed, so an older census is missing keys without having lost anything. Measured on engine#2369: 69 counters differ from main and 0 are gone, which is why that PR trips nothing here.rc 2, neverrc 0—cannot be read - cannot tell, and that is not a pass.Verified
tests/registry_loss_check_test.py(five cases, a throwaway repo each, so the arms are exercised against git and not a mock) is registered asadd_python_test, and a mutation proves the arm is load-bearing: with the counters arm's condition replaced byif False,test_a_coverage_counter_that_the_head_lacks_failsgoes red and the other four stay green.Wiring, deliberately not in this PR
It is not added to
scripts/pre-pr-checks.shyet: engine#2380 is open and edits the page and the runner's exact sentences (the "six"/"seventh" counts), so a second edit to the same lines would conflict for no gain. The tool is a ctest row now; the runner row is a one-line follow-up once #2380 lands.Refs engine#2021 (the loss), engine#2369 (the shapes that are not losses), engine#2368 (the class), engine#2371/#2373 (the other append-mostly files with their own checks).