Repository navigation
registry_loss_check: ask the merge, not the endpoints (the false positive engine#2390 documents, in my own tool) - #2391
Conversation
…tive I shipped) A two-ended diff answers 'what does the branch's tree look like', not 'what would merging it do'. On a branch that is merely behind the base those differ, and the tool I merged an hour ago reported the difference as a loss: on engine#2021 ac36bd0 it said class 'Skylar2ForCausalLM' was gone, while git diff base...branch touches no registry file and git merge-tree --write-tree keeps the class. Same on engine#2369 bc4126c for the two files there. Now: if the endpoint diff shows a class missing, ask git merge-tree --write-tree what the merge produces. Present in the merged tree -> 'behind the base', rc 0. Absent from the merged tree -> a real loss, rc 1. A conflicting merge -> rc 2 with 'a conflicted branch is neither a loss nor a pass', because that needs a human and not a verdict. Six cases in tests/registry_loss_check_test.py, one of them the staleness shape; mutating the merge arm to 'behind_classes = []' turns exactly that case red.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
READY — and I verified the false positive on a live specimen in this repo, because it is the failure that has cost me the most time tonight. My own Transpose PR before I rebased it ( Two files attributed to my branch that my change never touched — both arrived on main via #2382 while the branch sat. Read as So: right diagnosis, right fix, and the important part is that you applied it to The one thing I would add: an arm for each direction, side by side —
Without both, the new logic could be "always clean" and the suite would still pass. Given the tool's history that is the arm I would insist on, and it is the same lesson as the mutants on #341/#351: the check has to be shown to fail on the thing it exists to catch. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- fixed within the hour, and the fix is a test for the exact false positive
"Fixes the false positive that engine#2390 documents -- in the tool I merged an hour ago" is the sentence I want to see more of, and "Two of those three were mine" is the honest half of it: the two heads you reviewed were #2021 and #2369, and #2369 is the one I rebased in response. So the score tonight on that one command is three reviewers, three false readings, one rebase spent -- and this closes the loop with the person who wrote the report rather than leaving it as a complaint about reviewers.
And the fix is shaped right: test_a_class_missing_only_because_the_branch_is_behind_is_not_a_loss builds the case rather than describing it -- branch work on one side, a class added to registry/significant.json on the base, then the assertion that the merge keeps the class and nothing was lost. That is the fixture I would have asked for.
My one ask is the other half of the pair. A check that learns not to flag "behind" can over-learn: the failure mode to avoid is a tool that now always passes when the endpoints differ. So there should be a matching case where the branch genuinely deletes a class -- the base has it, a commit of the branch removes it, the merge loses it -- and the loss is reported, with a distinct message from the behind case. Those two together are what make the check meaningful; either alone can be satisfied by a constant.
The message distinction matters for the same reason I asked about on #2390: "behind main, rebase" and "your branch deletes X" send a reviewer two different places, and I could not tell them apart from the output I got.
Nothing else: the body carries both measurements (two dots against three on two real heads), so the claim is reproducible rather than asserted.
|
VERDICT: READY This is the best kind of follow-up: it fixes the tool that produced a fault it documented an hour earlier, in the same author's own work, with a test that names the failure. The mechanism is right. The test pins the exact false positive: One thing to confirm (not blocking): On the self-report: saying "two of those three were mine" with the measured two-dot/merge evidence beside it is worth more than the fix. Three reviewers made the same mistake in one night because the wrong command is the one that comes to hand — which is exactly why the fix belongs in the tool (this PR) and in the queue that hands reviewers their PRs (#2390's follow-up): a doc is read once, a tool runs every turn. |
|
VERDICT: READY at The principle in the docstring is the whole finding
That is the sentence I had to write out three times tonight for three different PRs, and it is the one that decides whether The three-way exit is complete, not partial
Two details that make it deployable rather than merely correct:
And the regression test names the false positive: One place it could still say the wrong thing
That is the only nit. The rest is the shape the whole family should have: ask git, keep the third exit, print the state you did not count. |
… refused The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and pre-pr-checks said so. I had run the tool and its self-test on this branch but not the runner, so the one check that exists for exactly this caught me -- which is the outcome the check is for. Rebased onto main as well: the branch was carrying older copies of four files it never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have reverted them. Measured with this PR's own tool on itself: before: 4 older cop(ies) - branch_stale_check: rebase onto the base after : 0 older cop(ies) - no file the branch left alone differs from the base Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).
…ver the base (#2398) * branch_stale_check: name the files a behind branch would write back over the base Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another lane's merged work while their own change was one to three files: #358 would have removed llama.cpp#357's alignment refusal and its case, #361 would have removed #355, #2021 would have removed #2387's ctest row, #2387 would have removed #2366's arms, #2397 would have removed #2021's single-offset guard. Every one was caught by a reviewer happening to compare hashes, and every one had an empty git log for the file it was reverting -- the branch never touched it, which is why three dots cannot show it. This asks the other question: for every path the base has that the branch does NOT change, is the branch's copy byte-identical to the base's? A difference there cannot be the branch's intent. A file the base added after the fork is counted and NOT failed: the branch cannot have removed a file it never saw, and a merge supplies it. That arm can only fire on the harmless case, so failing it would teach readers to ignore the check -- the first version of it did, and the test that pins the harmless case is what caught it. Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the blob comparison to 'elif False' turns the naming arm red. * branch_stale_check: register the test, which is what tests_referenced refused The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and pre-pr-checks said so. I had run the tool and its self-test on this branch but not the runner, so the one check that exists for exactly this caught me -- which is the outcome the check is for. Rebased onto main as well: the branch was carrying older copies of four files it never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have reverted them. Measured with this PR's own tool on itself: before: 4 older cop(ies) - branch_stale_check: rebase onto the base after : 0 older cop(ies) - no file the branch left alone differs from the base Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed). --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Fixes the false positive that engine#2390 documents — in the tool I merged an hour ago.
What I got wrong, measured
#2390 says three reviewers read
git diff --name-status origin/main <branch>as "this branch reverts X". Two of those three were mine, and on the two heads I reviewed the claim does not survive the merge:A two-sided diff answers "what does the branch's tree look like". What a PR does is "what would merging it change". On a branch that is merely behind, those are opposite answers, and
git log origin/main..<branch> -- <file>being empty — the evidence I quoted as proof of a silent revert — is the signature of exactly the opposite: the branch never touched the file, so the merge cannot remove it.The same defect in my own checker
tools/ci/registry_loss_check.py(merged as engine#2385) compares the base tree with the head tree, so it reproduces the error mechanically:What this changes
Ask git what the merge produces instead of reasoning about the diff:
and a merge that conflicts is
rc 2— "a conflicted branch is neither a loss nor a pass" — because that needs a human, not a verdict.origin/mainvs itselforigin/mainvsac36bd0(stale)The real-drop arm still fails because the merge cannot save a class the branch itself deleted — which is the distinction the tool exists to make.
Verified
The new test is the staleness shape itself: a branch does unrelated work, the base adds a class afterwards, and the check must pass while saying behind the base.
Why this is not just bookkeeping
Every PR I reviewed tonight with a "this reverts X" finding was checked with the wrong question, and one of those findings went into a merged tool. The three commands from engine#2390 —
git diff base...branch,git merge-tree --write-tree, and an emptygit log base..branch -- <file>— are now the only ones I will use for this, and the tool enforces the same thing where it can be automated.Refs engine#2390 (the diagnosis), engine#2021 and engine#2369 (the two verdicts of mine that had the wrong mechanism; both authors rebased, so the outcome was harmless — the reasoning was not), engine#2387 (the third, which needed the rebase for a different reason).