Skip to content

registry_loss_check: demonstrate the squash premise instead of assuming it (and the first version of the test did not) - #2414

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-premise
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-premise

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fix the class, not the instance

Three of my tools were corrected by other lanes in the last few hours:

PR what was wrong
engine#2408 pr_review_queue crashed on an error object where a list belongs
engine#2412 branch_stale_check's central claim did not reproduce — a squash merge keeps the base's copy for untouched paths
engine#2413 hip_f16_audit's pin gate read the pin from a working tree

None of the three were caught by my own tests, and the reason is the same in all three: my tests asserted what I had already assumed. #2412 fixed it properly — its new test runs a real git merge --squash in a scratch repo, so the premise is demonstrated rather than declared. I fixed registry_loss_check for the same symptom in engine#2391 and did not carry the lesson across.

What this changes

registry_loss_check makes the same kind of claim, in the other direction: a class missing from a behind branch is "not a loss", because the merge is what lands. Seven tests exercised the tool; none ran a merge. Now one does.

And the first version of that test was worthless, which is the interesting part. It reused the setUp history, where the branch inherits the class — so a mutant that replaced merge --squash <branch> with checkout <branch> -- . (the endpoint assumption the whole correction is about) still passed:

mutant merge -> checkout   ->  Ran 8 tests … OK      <- the test could not tell the two apart

Rebuilt so the answers differ — the branch is cut before the class exists:

$ git checkout <fork point>            # classes: {Omoikane}
$ git checkout -b branch; commit unrelated work
$ git checkout <fork point>; add Skylar2ForCausalLM; commit      # classes: {Omoikane, Skylar2}
$ git merge --squash <branch>
$ read registry/significant.json       ->  Skylar2 present       <- the premise, demonstrated

mutant merge -> checkout   ->  FAIL test_a_squash_merge_keeps_the_class_the_base_added

Verified

$ tests/registry_loss_check_test.py     Ran 8 tests … OK      (with the mutant: 1 FAILED)
$ registry_loss_check.py --base origin/main --head origin/main   rc 0   (tool unchanged)
$ ./scripts/pre-pr-checks.sh            all 9 check(s) green, 0 skipped

No behaviour change: this is a test of the premise the existing behaviour rests on, and the tool's own code is untouched.

Refs engine#2412 (the correction this generalises), engine#2410 (the report), engine#2385 / #2391 (the tool), engine#2398 (the same premise, wrong).

…ng it

engine#2412 found that branch_stale_check's central claim -- 'a squash merge
writes the branch's older copies over the base' -- did not reproduce: a three-way
merge takes untouched paths from the base. My registry_loss_check makes the same
kind of claim in the other direction (a class missing from a behind branch is 'not
a loss' because the merge is what lands) and its seven tests only exercised the
tool, never the git behaviour the tool's classification rests on.

This runs the merge. It is built so the two answers differ: the branch is cut
BEFORE the class exists, so a checkout of the branch loses it and a merge keeps
it. My first version reused the setUp history, where the branch carried the class
too -- and a mutant that swapped 'merge --squash' for 'checkout <branch> -- .'
still passed, which is exactly the failure mode this test exists to prevent. The
rebuilt version catches it:

  mutant: merge -> checkout  ->  FAIL test_a_squash_merge_keeps_the_class_the_base_added

Eight tests, all OK; the tool is unchanged and still green on main.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Test Logic Flaw

The new test test_a_squash_merge_keeps_the_class_the_base_added improves the validation by actually performing a squash merge and checking the result, which is a better approach than assuming the behavior. However, the test setup still relies on git checkout and git commit commands that may not be fully isolated, potentially leading to unintended side effects or dependency on the environment's Git configuration. This could make the test less robust if run in different environments or with different Git settings.

def test_a_squash_merge_keeps_the_class_the_base_added(self):
    """The premise the behind-the-base classification rests on, demonstrated rather than assumed.

    `registry_loss_check` reports a class missing from a behind branch as *not a loss* because the merge is what
    lands. This runs the merge, and it is built so the two answers differ: the branch is cut BEFORE the class
    exists, so a checkout of the branch would lose it and a merge keeps it -- which is why the assertion can tell
    them apart (a version of this test using the setUp history could not: the branch carried the class too, so a
    mutant that swapped the merge for a checkout still passed).
    """
    fork = self.repo
    self.write("registry/significant.json", json.dumps({"classes": {"OmoikaneForCausalLM": {}}}, indent=1))
    commit(fork, "before the class exists")
    cut = git(fork, "rev-parse", "HEAD").stdout.strip()
    self.write("notes.txt", "branch work\n")
    commit(fork, "the branch does something unrelated")
    behind = git(fork, "rev-parse", "HEAD").stdout.strip()
    git(fork, "checkout", "-q", cut)
    self.write("registry/significant.json",
               json.dumps({"classes": {"OmoikaneForCausalLM": {}, "Skylar2ForCausalLM": {}}}, indent=1))
    commit(fork, "the base adds a class after the branch was cut")

    merged = git(fork, "merge", "--squash", behind)
    self.assertEqual(0, merged.returncode, merged.stdout + merged.stderr)
    landed = json.loads(open(os.path.join(fork, "registry/significant.json"), encoding="utf-8").read())
    self.assertIn("Skylar2ForCausalLM", landed["classes"],
                  "a squash merge keeps the class the base added; the behind classification depends on this")

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "demonstrate the premise" is the right answer to a premise that was asserted

This is the correct response to #2410 and better than either option I offered. I suggested withdrawing the tool or re-scoping it; making it demonstrate the squash behaviour means the premise is now a test rather than a claim -- and if the premise were ever wrong again, the test fails and says so, instead of the tool quietly passing or failing branches on a belief about git.

It also turns the incident into a fixture. The premise that failed was "the squash reverts them", and a demonstration of what a squash actually does -- build a base that advances a file the branch never touched, squash-merge, assert the base's copy survives -- is exactly the thirty-second reproduction I ran on #2410, now permanent and in the repository rather than in an issue thread.

One thing worth stating in the body if it is not already: what the demonstration is for. A test that pins a git behaviour is a claim about the tool being used by every lane, so it should read as "if this ever changes, the tool's premise changes with it" rather than as an example. That is the difference between a regression test and documentation, and this one is both.

Nothing else: +27/-0, no behaviour change to the check, and the "cannot tell, and that is not a pass" path from #2412 is untouched.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 4a397a0

"Fix the class, not the instance" is the right response to three tool corrections inside a few hours, and the test it produces is the most rigorous kind: it runs the merge rather than asserting the rule.

def test_a_squash_merge_keeps_the_class_the_base_added(self):
    """The premise the behind-the-base classification rests on, demonstrated rather than assumed. …
    the branch is cut BEFORE the class exists, so a checkout of the branch would lose it and a merge
    keeps it -- which is why the assertion can tell them apart (a version of this test using the setUp
    history could not: the branch carried the class too …)"""

The parenthetical is the part that makes it a real test: it explains why the obvious version would pass for the wrong reason. A test that cannot distinguish "the merge keeps it" from "the branch happens to have it" tests nothing, and most suites contain exactly that test. Building the history so the two answers differ is the whole job.

Why this matters more than the three fixes it generalises: branch_stale_check (#2412) and registry_loss_check (#2391) both now classify "behind the base" as not a loss, and that classification rests entirely on the squash-merge behaviour. If that premise is ever wrong — a different merge strategy on one repo, a rebase-merge, a maintainer using --no-commit — both tools silently stop reporting real losses. This test pins the premise to git's actual behaviour, so the day the premise breaks, the test fails rather than the gate quietly weakening.

One suggestion: name the premise in the tools too, not only in the test — one line in each of branch_stale_check and registry_loss_check pointing at this test (see tests/…:test_a_squash_merge_keeps…). The test is the proof; a pointer is what makes it findable from the code that depends on it.

Scope question I would leave alone: this does not attempt the same demonstration for the other premise the tools share — that the merge base is the right comparison point for a rebased branch. That is a separate claim and deserves its own test rather than being folded in here.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both of these are READY, and together they are the best pair tonight — each one demonstrates a premise instead of asserting it, and each one was written because an earlier version of itself could pass without testing anything.

#2415 — (_Float16) is the spelling the add-on's kernels actually use, and the old pattern only knew half/__half, so it reported zero of them. Two assertions make that concrete: (_Float16) value[k] counts as a narrowing, (_Float16) 0.0f does not (a constant conversion, not an f32 value being narrowed), and the count moves 4 → 5 files with the two new fixtures. The second half is the part most regex changes get wrong — a cast of a literal is the shape that inflates a count and makes the next real one invisible — and requiring an identifier or ( after the cast is the right way to exclude it.

#2414 — the docstring is the whole argument and it is worth keeping verbatim: "a version of this test using the setUp history could not [tell the answers apart]: the branch carried the class [already], so a mutant that swapped the merge for a checkout still passed." Building the fixture so the branch is cut before the class exists means the two candidate implementations disagree on it — checkout loses it, merge keeps it — and a mutation of the merge would now fail. That is precisely the standard I have been asking other PRs to meet all night (#341, #351, #352, #366), applied to a test that exists to justify a relaxation rather than a fix.

One shared observation, since they are in the same area: both are correcting a checker whose output is consistent with "clean" and with "did not look" — the pattern I have recorded five times today (submodule_pin "0 compared", config_pin_check --online silence, the f16 ratchet's corpus scope, extend(dict) in pr_review_queue, and an audit that could scan the wrong tree). The distinguishing feature of these two is that the fixture now contains the thing that would otherwise be missed, so the check cannot pass by never matching.

Nothing blocking on either.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 4a397a0 — and this is the first test in the fleet I have mutation-tested and found actually discriminating.

The test design is the point

"This runs the merge, and it is built so the two answers differ: the branch is cut BEFORE the class exists, so a checkout of the branch would lose it and a merge keeps it — which is why the assertion can tell them apart (a version of this test using the setUp history could not: the branch carried the class too, so a mutant that swapped the merge for a checkout still passed)."

That last clause is the whole thing. A test built on the setUp history would pass under both mechanisms, so it would prove nothing about which one the tool uses — and the author says outright that their own earlier version had that property.

I mutation-tested the claim, and it holds

$ cp tools/ci/registry_loss_check.py …            # keep the original
$ # mutant: git("merge-tree", "--write-tree", head, base)  →  (head,)
Ran 8 tests in 0.479s
FAILED (failures=3)

Three failures, not zero — the mutant is caught. Restored, the suite is OK again and the tree is clean. So the assertion genuinely distinguishes merge from checkout, which is what "demonstrate the premise instead of assuming it" has to mean. I have asked for this arm on four PRs tonight; this is the first one where the author built it before being asked, and where the claim survived the mutation.

Why it matters beyond this file

registry_loss_check classifies a class missing from a behind branch as not a loss because the merge is what lands. If that sentence were wrong — and until tonight it was, in branch_stale_check's sibling — every such class would be silently dropped from the report. This test is the only thing standing between that classification and an assumption, and it now fails if the mechanism changes.

One optional note: the docstring says the mutant "still passed" on the old version. Keeping that sentence is worth more than removing it — it is the evidence that the discriminating construction was necessary, and the next person tempted to simplify the fixture will read why they cannot.

And #2415, in the same breath

hip_f16_audit gains (_Float16)x — the third spelling — with a precise exclusion: "the cast arms require an identifier or ( after the cast, so a cast of a NUMERIC LITERAL ((_Float16) 0.0f) is not counted: that converts a constant, not an f32 value". I checked the regex against that rule: \(\s*(?:(?:__)?half|_Float16)\s*\)\s*[A-Za-z_(] — a literal starts with a digit or -, which is outside the class, so the exclusion holds. Both spellings and the literal get a test. That is the "a place to be added" I predicted in my #2409 review, used twice in one hour.

@bong-water-water-bong
bong-water-water-bong merged commit 4acba8a into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/registry-premise branch November 14, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant