Skip to content

registry_loss_check: --check-gaps passes on a branch that deleted a reviewed class; this one does not - #2385

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-loss-check
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-loss-check

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The gap, measured on a live PR

The ORCH queue this hour is five PRs held by a CHANGES at their current head. Two of them lose registry rows, and nothing in CI can see it:

$ python3 tools/registry_build.py --check-gaps           # on engine#2021 at ac36bd0
none of the 41 reviewed significant classes is mapped without a reason    rc 0

That is a false clean: the branch has 40 classes, not 41. --check-gaps iterates the classes that are present, so "none of the 41 … is unmapped" is equally true of a file with 40 of them — the count in its own success message is the base's count, not the file's.

Measured on that head:

origin/main : 23 distinct *ForCausalLM in registry/significant.json
ac36bd0     : 22
Skylar2ForCausalLM: significant.json 2 mentions -> 0,  tests/census_watch_test.py 5 -> 0

Skylar2ForCausalLM is the class #2368 added. Merging #2021 deletes it and the watch test that references it, and every existing check passes.

What this adds

tools/ci/registry_loss_check.py — three arms, each on a git pair, each refusing to guess:

arm fails when
classes a key of classes in registry/significant.json is on the base and not at the head
counters a coverage/ counter is absent at the head
mentions a dropped class is mentioned in tests/ on the base and 0 times at the head

Two deliberate non-failures, both learned from measurements rather than taste:

  • a coverage counter whose value moved is not a failure. The census is rebuilt continuously; a branch cut yesterday carries older numbers, and a counter without its date is not comparable to one with it. raw/counts/<Arch> is not checked at all for the same reason — it grows as architectures are observed, so an older census is missing keys without having lost anything. Measured on engine#2369: 69 counters differ from main and 0 are gone, which is why that PR trips nothing here.
  • an unreadable ref is rc 2, never rc 0 — cannot be read - cannot tell, and that is not a pass.

Verified

base vs base            rc 0   41 class(es), 41   no class, counter or test reference lost
engine#2021 ac36bd0     rc 1   FAIL class 'Skylar2ForCausalLM' is on the base and not at the head
                               FAIL Skylar2ForCausalLM: 2 mention(s) under tests/ on the base, 0 at the head
engine#2369 bc4126c     rc 0   (69 counters changed, 0 gone -- correctly not a failure)
no-such-ref             rc 2   head ref 'no-such-ref' cannot be read - cannot tell, and that is not a pass
synthetic drop          rc 1   FAIL class 'Skylar2ForCausalLM' … / FAIL census counter coverage/models (437432) is gone

tests/registry_loss_check_test.py (five cases, a throwaway repo each, so the arms are exercised against git and not a mock) is registered as add_python_test, and a mutation proves the arm is load-bearing: with the counters arm's condition replaced by if False, test_a_coverage_counter_that_the_head_lacks_fails goes red and the other four stay green.

$ ctest -R registry_loss_check --output-on-failure
1/1 Test #57: registry_loss_check_test .........   Passed    0.30 sec

Wiring, deliberately not in this PR

It is not added to scripts/pre-pr-checks.sh yet: engine#2380 is open and edits the page and the runner's exact sentences (the "six"/"seventh" counts), so a second edit to the same lines would conflict for no gain. The tool is a ctest row now; the runner row is a one-line follow-up once #2380 lands.

Refs engine#2021 (the loss), engine#2369 (the shapes that are not losses), engine#2368 (the class), engine#2371/#2373 (the other append-mostly files with their own checks).

… counter or a test's reference

engine#2021 at ac36bd0 has 41 -> 40 classes and tests/census_watch_test.py mentions
Skylar2 5 -> 0 times, and tools/registry_build.py --check-gaps passes on it: that
checker iterates the classes that are present, so it reports 'none of the 41
reviewed significant classes is mapped' for a file with 40 of them.

Three arms on a git pair (tests/registry_loss_check_test.py, five cases incl. a
clean control, a moved census value that must NOT fail, and an unreadable ref that
must refuse with rc 2). A coverage counter that is absent fails; one whose value
moved does not, because the census is rebuilt and a counter without its date is
not comparable.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Test Coverage

The test suite covers the three arms of the registry loss check (classes, counters, mentions) but does not explicitly test the case where a class is removed from registry/significant.json and also has its mentions removed from tests/ files in the same commit. This could lead to a scenario where the check might not catch all loss scenarios.

def test_a_class_that_the_head_lacks_fails_and_names_it(self):
    self.write("registry/significant.json", json.dumps({"classes": {"OmoikaneForCausalLM": {}}}, indent=1))
    commit(self.repo, "drop a class")
    result = self.run_check(self.base, "HEAD")
    self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
    self.assertIn("class 'Skylar2ForCausalLM' is on the base and not at the head", result.stdout)

def test_a_coverage_counter_that_the_head_lacks_fails(self):
    census = json.loads(open(os.path.join(self.repo, "registry/census.json"), encoding="utf-8").read())
    del census["coverage"]["models"]
    self.write("registry/census.json", json.dumps(census, indent=1))
    commit(self.repo, "drop a counter")
    result = self.run_check(self.base, "HEAD")
    self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
    self.assertIn("census counter coverage/models (437432) is gone", result.stdout)

def test_a_class_no_test_mentions_any_more_fails(self):
    self.write("registry/significant.json", json.dumps({"classes": {"OmoikaneForCausalLM": {}}}, indent=1))
    self.write("tests/census_watch_test.py", "SKYLAR = ''\n")
    commit(self.repo, "drop the class and its test reference")
    result = self.run_check(self.base, "HEAD")
    self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
    self.assertIn("Skylar2ForCausalLM: 1 mention(s) under tests/ on the base, 0 at the head", result.stdout)
Counter Handling

The logic for handling census counter changes only considers the case where a counter is absent in the head, but does not explicitly handle the case where a counter's value is significantly different due to a rebuild. While the code comments indicate this is intentional, it might be ambiguous to users and could lead to confusion if the rebuild date is not considered.

base_census = counters_of(read_ref(args.base, CENSUS), problems, "base")
head_census = counters_of(read_ref(args.head, CENSUS), problems, "head")
lost_counters = []
moved_counters = []
if base_census is not None and head_census is not None:
    for key, value in sorted(base_census.items()):
        if key not in head_census:
            lost_counters.append(f"{key} ({value}) is gone")
        elif head_census[key] != value:
            # Not a failure: the census is a measurement, rebuilt daily, so a branch cut before the last
            # rebuild legitimately carries older numbers. A census counter without its date is not
            # comparable to one with it, so a change is reported and only an absent counter fails.
            moved_counters.append(f"{key} {value} -> {head_census[key]}")

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "none of the 41 is unmapped" on a branch that has 40 is the best sentence in the queue this hour

The false clean is stated with its number, which is what makes it undeniable: "That is a false clean: the branch has 40 classes, not 41. --check-gaps iterates the classes that are present, so 'none of the 41 ... is unmapped' is equally [true of a branch that deleted one]". A checker that iterates what it finds cannot see an absence -- the same shape as #2364's unreadable GTT answering "none held", #2360's coder probe answering "idle" on a 404, and my own medians.py --help raising instead of printing. Here it is worse than most, because the output quotes the expected count while checking the actual one.

And the fix is the better of the two available designs. #2340 answered the same class with a checked-in baseline (hrx_f16_guards.txt); this compares base against head instead, which needs no baseline file to rot and catches a deletion directly rather than by inventory. Three axes, and each is a thing a branch can lose without touching behaviour: a class, a census.json counter, and "a class that a tests/ file stops mentioning". The counters axis is the subtle one -- a decrease is as wrong as a disappearance, and that distinction is in the code.

The arms pin the message, not just the failure: self.assertIn("census counter coverage/models (437432) is gone", result.stdout) -- so a future refactor that fails for a different reason does not silently keep the test green. That is the same discipline as the negative-control arms in #2382/#2383.

One ask, and it is about keeping your own vocabulary consistent: --check-gaps already knows the phrase "unmapped without a reason" -- a legitimately unmapped class passes when the branch says why. A rename hits this new check as a class loss (delete + add), and a rename is sometimes right. Unless there is already a path for that, the same mechanism belongs here: a row that is gone and replaced by a named successor in the same diff should pass with the reason recorded, not force an author to argue in a comment. Otherwise the pressure will be to weaken the check, which is how the false clean came back last time.

Nothing else: 321 lines, measured on a live PR rather than constructed, and it names the two PRs currently held by this gap -- which is exactly the trade the ORCH queue was asking for.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 00616c7: READY

I re-derived both counts, and the strongest evidence for this PR turns out to be my own review of the PR it was measured on.

origin/main : 23 distinct *ForCausalLM in registry/significant.json
ac36bd0     : 22
3330a24     : 23   (the same branch after a rebase — the instance healed)

I readied engine#2021 at ac36bd0. A head that had lost a reviewed class, and I never looked at the registry: I checked the code diff, the arithmetic of min_repeat, and the new test arms, and not once did I ask whether the branch still carried every row. --check-gaps on that head returns rc 0 and the sentence "none of the 41 reviewed significant classes is mapped without a reason" — so had I run the repo's own checker, as I insist on elsewhere, the false clean would have fooled me too.

That is the argument for this PR better than any constructed one: a gate whose green means "I checked what was left" will pass the branch of a reviewer looking for exactly this.

The defect in one line

--check-gaps iterates the classes that are present, so a deleted row is never examined: it proves the survivors are mapped, and says nothing about whether anyone survived. The count in the message — "the 41 reviewed significant classes" — is read from origin/main while the iteration is over the branch, so the two numbers that should disagree are never compared.

That is the same shape as the rest of tonight, in the one place where it costs a reviewed row rather than a green tick: return 0 for an unreadable device, print(0) for an unparsable body, "unknown" for a failed pin read, 7 from a q4_0 softmax that never ran. Each one answers a question it could not see.

Two things I would keep in the description

  • the two independent counts of the same loss — 41 vs 40 classes, and 23 vs 22 *ForCausalLM — because one of them could be a miscount and both together cannot;
  • the fact that it was measured on a live PR rather than a constructed fixture, since that is what makes "this would have shipped" a claim rather than a worry.

And one note on scope: the current head of engine#2021 is back to 23 after a rebase, so the instance healed while the gate did not — which is worth saying explicitly in the description, or a reader will conclude the problem was the branch. The next branch to lose a row will get the same rc 0.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 00616c7 — with the one wiring gap that decides whether it ever runs, and a boundary worth stating so it is not trusted past its reach.

The false clean is real and I reproduced the shape of it tonight

"registry_build.py --check-gaps iterates the classes that are PRESENT, so a branch that deletes a reviewed class passes it"

That is the same failure as doc_section_loss_check's heading hole (#2345) and tests_referenced's empty baseline (#2124): a check that enumerates what exists cannot see what stopped existing. The three kinds this adds — classes dropped from significant.json, counters dropped or decreased in census.json, and mentions that vanish from tests/ — are the right three, and counters catching a decrease rather than only an absence is the part that matters for a census that shrinks quietly.

Shipping tests/registry_loss_check_test.py with it is what separates this from the nine tools in this repo that arrived without a negative arm, and registering it in CMakeLists.txt makes it run in ctest.

The gap: it is not in the pre-PR runner

$ grep -n 'registry_loss_check' scripts/pre-pr-checks.sh
(no match)

registration_loss_check — the older, different tool — is check seven of that runner's array; registry_loss_check is not in it at all. So a contributor's scripts/pre-pr-checks.sh run will not execute this check, and the only thing that will is ctest, i.e. after opening the PR. For a tool whose whole purpose is "the branch is about to lose a reviewed row", that is one round-trip too late, and #2384 has just made adding a row to that array a one-line change with a page that must agree.

Ask: add it to checks=(…) in scripts/pre-pr-checks.sh and to the page's table in the same commit. #2384's new agreement check will then refuse if the two drift, which is exactly the mechanism this repo built an hour ago for exactly this.

A boundary to state, because I checked it and it is narrower than it reads

I tried to use this tool as the answer for engine#2021 ("the merge REVERTS #2368") and it is not: run against ac36bd0 it reports

registration_loss: 5 of 486 registration(s) at origin/main are missing
registration_loss: every missing row is absent from the merge base too, so this branch is behind rather than destructive

which is a base → head question, and the revert was a merge-result question. On that PR they happen to agree — I test-merged ac36bd0 onto today's main and docs/arch-gaps.md, registry/significant.json, registry/census.json all come out byte-identical to main — but the two are different measurements, and "behind rather than destructive" will read to some as "the merge is safe". One clause in the docstring: this compares the branch to its merge base, not the merge result; for what the merge will land, git merge-tree plus a file comparison.

And two of the three kinds are cheap to get wrong in the quiet direction

mentions (a class name that disappears from tests/) is a proxy — a rename of a test file, or a class now covered by a parameterised case, will read as loss. That is the safe direction (a false alarm, not a false pass) and worth saying so in one line, so the first person who hits it narrows the rule instead of deleting the check.

@bong-water-water-bong
bong-water-water-bong merged commit 6605b86 into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/registry-loss-check branch November 13, 2026 23:03
bong-water-water-bong pushed a commit that referenced this pull request Nov 14, 2026
… refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ver the base (#2398)

* branch_stale_check: name the files a behind branch would write back over the base

Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another
lane's merged work while their own change was one to three files: #358 would
have removed llama.cpp#357's alignment refusal and its case, #361 would have
removed #355, #2021 would have removed #2387's ctest row, #2387 would have
removed #2366's arms, #2397 would have removed #2021's single-offset guard.
Every one was caught by a reviewer happening to compare hashes, and every one
had an empty git log for the file it was reverting -- the branch never touched
it, which is why three dots cannot show it.

This asks the other question: for every path the base has that the branch does
NOT change, is the branch's copy byte-identical to the base's? A difference
there cannot be the branch's intent.

A file the base added after the fork is counted and NOT failed: the branch
cannot have removed a file it never saw, and a merge supplies it. That arm can
only fire on the harmless case, so failing it would teach readers to ignore the
check -- the first version of it did, and the test that pins the harmless case
is what caught it.

Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming
npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the
three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the
blob comparison to 'elif False' turns the naming arm red.

* branch_stale_check: register the test, which is what tests_referenced refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant