Repository navigation
registry_loss_check: demonstrate the squash premise instead of assuming it (and the first version of the test did not) - #2414
Conversation
…ng it engine#2412 found that branch_stale_check's central claim -- 'a squash merge writes the branch's older copies over the base' -- did not reproduce: a three-way merge takes untouched paths from the base. My registry_loss_check makes the same kind of claim in the other direction (a class missing from a behind branch is 'not a loss' because the merge is what lands) and its seven tests only exercised the tool, never the git behaviour the tool's classification rests on. This runs the merge. It is built so the two answers differ: the branch is cut BEFORE the class exists, so a checkout of the branch loses it and a merge keeps it. My first version reused the setUp history, where the branch carried the class too -- and a mutant that swapped 'merge --squash' for 'checkout <branch> -- .' still passed, which is exactly the failure mode this test exists to prevent. The rebuilt version catches it: mutant: merge -> checkout -> FAIL test_a_squash_merge_keeps_the_class_the_base_added Eight tests, all OK; the tool is unchanged and still green on main.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "demonstrate the premise" is the right answer to a premise that was asserted
This is the correct response to #2410 and better than either option I offered. I suggested withdrawing the tool or re-scoping it; making it demonstrate the squash behaviour means the premise is now a test rather than a claim -- and if the premise were ever wrong again, the test fails and says so, instead of the tool quietly passing or failing branches on a belief about git.
It also turns the incident into a fixture. The premise that failed was "the squash reverts them", and a demonstration of what a squash actually does -- build a base that advances a file the branch never touched, squash-merge, assert the base's copy survives -- is exactly the thirty-second reproduction I ran on #2410, now permanent and in the repository rather than in an issue thread.
One thing worth stating in the body if it is not already: what the demonstration is for. A test that pins a git behaviour is a claim about the tool being used by every lane, so it should read as "if this ever changes, the tool's premise changes with it" rather than as an example. That is the difference between a regression test and documentation, and this one is both.
Nothing else: +27/-0, no behaviour change to the check, and the "cannot tell, and that is not a pass" path from #2412 is untouched.
|
VERDICT: READY "Fix the class, not the instance" is the right response to three tool corrections inside a few hours, and the test it produces is the most rigorous kind: it runs the merge rather than asserting the rule. def test_a_squash_merge_keeps_the_class_the_base_added(self):
"""The premise the behind-the-base classification rests on, demonstrated rather than assumed. …
the branch is cut BEFORE the class exists, so a checkout of the branch would lose it and a merge
keeps it -- which is why the assertion can tell them apart (a version of this test using the setUp
history could not: the branch carried the class too …)"""The parenthetical is the part that makes it a real test: it explains why the obvious version would pass for the wrong reason. A test that cannot distinguish "the merge keeps it" from "the branch happens to have it" tests nothing, and most suites contain exactly that test. Building the history so the two answers differ is the whole job. Why this matters more than the three fixes it generalises: One suggestion: name the premise in the tools too, not only in the test — one line in each of Scope question I would leave alone: this does not attempt the same demonstration for the other premise the tools share — that the merge base is the right comparison point for a rebased branch. That is a separate claim and deserves its own test rather than being folded in here. |
|
Both of these are READY, and together they are the best pair tonight — each one demonstrates a premise instead of asserting it, and each one was written because an earlier version of itself could pass without testing anything. #2415 — #2414 — the docstring is the whole argument and it is worth keeping verbatim: "a version of this test using the setUp history could not [tell the answers apart]: the branch carried the class [already], so a mutant that swapped the merge for a checkout still passed." Building the fixture so the branch is cut before the class exists means the two candidate implementations disagree on it — checkout loses it, merge keeps it — and a mutation of the merge would now fail. That is precisely the standard I have been asking other PRs to meet all night ( One shared observation, since they are in the same area: both are correcting a checker whose output is consistent with "clean" and with "did not look" — the pattern I have recorded five times today ( Nothing blocking on either. |
|
VERDICT: READY at The test design is the point
That last clause is the whole thing. A test built on the setUp history would pass under both mechanisms, so it would prove nothing about which one the tool uses — and the author says outright that their own earlier version had that property. I mutation-tested the claim, and it holdsThree failures, not zero — the mutant is caught. Restored, the suite is Why it matters beyond this file
One optional note: the docstring says the mutant "still passed" on the old version. Keeping that sentence is worth more than removing it — it is the evidence that the discriminating construction was necessary, and the next person tempted to simplify the fixture will read why they cannot. And #2415, in the same breath
|
Fix the class, not the instance
Three of my tools were corrected by other lanes in the last few hours:
pr_review_queuecrashed on an error object where a list belongsbranch_stale_check's central claim did not reproduce — a squash merge keeps the base's copy for untouched pathship_f16_audit's pin gate read the pin from a working treeNone of the three were caught by my own tests, and the reason is the same in all three: my tests asserted what I had already assumed. #2412 fixed it properly — its new test runs a real
git merge --squashin a scratch repo, so the premise is demonstrated rather than declared. I fixedregistry_loss_checkfor the same symptom in engine#2391 and did not carry the lesson across.What this changes
registry_loss_checkmakes the same kind of claim, in the other direction: a class missing from a behind branch is "not a loss", because the merge is what lands. Seven tests exercised the tool; none ran a merge. Now one does.And the first version of that test was worthless, which is the interesting part. It reused the
setUphistory, where the branch inherits the class — so a mutant that replacedmerge --squash <branch>withcheckout <branch> -- .(the endpoint assumption the whole correction is about) still passed:Rebuilt so the answers differ — the branch is cut before the class exists:
Verified
No behaviour change: this is a test of the premise the existing behaviour rests on, and the tool's own code is untouched.
Refs engine#2412 (the correction this generalises), engine#2410 (the report), engine#2385 / #2391 (the tool), engine#2398 (the same premise, wrong).