Skip to content

registry_loss_check: ask the merge, not the endpoints (the false positive engine#2390 documents, in my own tool) - #2391

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-loss-merge
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/registry-loss-merge

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes the false positive that engine#2390 documents — in the tool I merged an hour ago.

What I got wrong, measured

#2390 says three reviewers read git diff --name-status origin/main <branch> as "this branch reverts X". Two of those three were mine, and on the two heads I reviewed the claim does not survive the merge:

on engine#2021 ac36bd0                        on engine#2369 bc4126c
  two dots   : 1 file differs                   two dots   : 1 file differs
  three dots : 0 files                          three dots : 0 files
  merge-tree : Skylar2 present (2 mentions)     (branch's own change touches neither file)

A two-sided diff answers "what does the branch's tree look like". What a PR does is "what would merging it change". On a branch that is merely behind, those are opposite answers, and git log origin/main..<branch> -- <file> being empty — the evidence I quoted as proof of a silent revert — is the signature of exactly the opposite: the branch never touched the file, so the merge cannot remove it.

The same defect in my own checker

tools/ci/registry_loss_check.py (merged as engine#2385) compares the base tree with the head tree, so it reproduces the error mechanically:

$ registry_loss_check.py --base origin/main --head ac36bd0      # before this PR
FAIL registry_loss_check: class 'Skylar2ForCausalLM' is on the base and not at the head    rc 1

What this changes

Ask git what the merge produces instead of reasoning about the diff:

tree = git("merge-tree", "--write-tree", head, base).stdout.split("\n", 1)[0]
merged = classes_of(git("show", f"{tree}:{SIGNIFICANT}").stdout, ...)
behind_classes = [n for n in lost_classes if n in merged]   # not a loss
lost_classes   = [n for n in lost_classes if n not in merged]

and a merge that conflicts is rc 2 — "a conflicted branch is neither a loss nor a pass" — because that needs a human, not a verdict.

arm before after
origin/main vs itself rc 0 rc 0
origin/main vs ac36bd0 (stale) rc 1, class named rc 0, "1 class(es) are absent at the head only because it is behind the base - the merge keeps them"
a real drop (synthetic branch that deletes the class) rc 1 rc 1, class named
unreadable ref rc 2 rc 2
conflicting branch rc 1, misleading rc 2, refuses to judge

The real-drop arm still fails because the merge cannot save a class the branch itself deleted — which is the distinction the tool exists to make.

Verified

$ python3 tests/registry_loss_check_test.py
Ran 6 tests in 0.326s
OK

# mutation: behind_classes = [] (the merge arm stops classifying)
FAIL: test_a_class_missing_only_because_the_branch_is_behind_is_not_a_loss
Ran 6 tests … FAILED (failures=1)            # only that case; the other five stay green

The new test is the staleness shape itself: a branch does unrelated work, the base adds a class afterwards, and the check must pass while saying behind the base.

Why this is not just bookkeeping

Every PR I reviewed tonight with a "this reverts X" finding was checked with the wrong question, and one of those findings went into a merged tool. The three commands from engine#2390 — git diff base...branch, git merge-tree --write-tree, and an empty git log base..branch -- <file> — are now the only ones I will use for this, and the tool enforces the same thing where it can be automated.

Refs engine#2390 (the diagnosis), engine#2021 and engine#2369 (the two verdicts of mine that had the wrong mechanism; both authors rebased, so the outcome was harmless — the reasoning was not), engine#2387 (the third, which needed the rebase for a different reason).

…tive I shipped)

A two-ended diff answers 'what does the branch's tree look like', not 'what
would merging it do'. On a branch that is merely behind the base those differ,
and the tool I merged an hour ago reported the difference as a loss: on
engine#2021 ac36bd0 it said class 'Skylar2ForCausalLM' was gone, while
git diff base...branch touches no registry file and git merge-tree --write-tree
keeps the class. Same on engine#2369 bc4126c for the two files there.

Now: if the endpoint diff shows a class missing, ask git merge-tree --write-tree
what the merge produces. Present in the merged tree -> 'behind the base', rc 0.
Absent from the merged tree -> a real loss, rc 1. A conflicting merge -> rc 2
with 'a conflicted branch is neither a loss nor a pass', because that needs a
human and not a verdict.

Six cases in tests/registry_loss_check_test.py, one of them the staleness shape;
mutating the merge arm to 'behind_classes = []' turns exactly that case red.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Merge-tree logic may miss some conflicts

The merged_classes function uses git merge-tree --write-tree to determine the merged class set, but it only checks for a non-zero return code to detect conflicts. However, git merge-tree can produce a tree with conflicts without necessarily returning a non-zero exit code, which could lead to incorrect results if such cases are not handled properly.

result = git("merge-tree", "--write-tree", head, base)
if result.returncode != 0:
    detail = (result.stderr or result.stdout).strip().splitlines()
    first = detail[0][:160] if detail else ""
    if result.returncode == 1:
        problems.append(f"merging {head} into {base} conflicts ({first}); resolve it and re-run - "
                        f"a conflicted branch is neither a loss nor a pass")
    else:
        problems.append(f"git merge-tree {head} {base} failed: {first}")
    return None
Potential incorrect classification of lost classes

The logic that determines behind_classes assumes that if a class exists in the merged result, it was never actually lost. However, this may not be accurate in all cases, especially when the merge introduces new versions or changes to classes that were previously considered lost. This could lead to false negatives in detecting actual losses.

behind_classes = []
if lost_classes:
    merged = merged_classes(args.base, args.head, problems)
    if merged is None:
        print("registry_loss_check: " + "; ".join(problems[-1:]), file=sys.stderr)
        return 2
    # The merge is the authority: a class the merge keeps was never lost, only behind the base's moves.
    behind_classes = [name for name in lost_classes if name in merged]
    lost_classes = [name for name in lost_classes if name not in merged]

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and I verified the false positive on a live specimen in this repo, because it is the failure that has cost me the most time tonight.

My own Transpose PR before I rebased it (18a982a5, a head that was merely behind main):

two-dot   origin/main 18a982a5 : M docs/npu.md   M npu/audio_exec.cpp   M tests/audio_parallel_test.cpp
three-dot origin/main...18a982a5: M npu/audio_exec.cpp

Two files attributed to my branch that my change never touched — both arrived on main via #2382 while the branch sat. Read as --name-status (the shape #2390 documents), that is "this branch modifies docs/npu.md and the parallel test", which is not just noise: it is the same output a genuine revert produces. That is the whole defect — the two-dot form cannot distinguish "reverts main's work" from "was branched before main's work", and both show as M. merge-tree answers what a merge would change, which is the question a merge gate is actually asking.

So: right diagnosis, right fix, and the important part is that you applied it to registry_loss_check.py rather than only to the reviewing habit. That checker is the one that caught four real accidental reverts of mine on rebases (a CMakeLists.txt onebit_q4nx_cache lib, a --target 1bit refusal, kernel-set rows, ~450 lines of npu/pack_import.*) — and a checker with false positives is worse than no checker, because after the second false alarm nobody reads the real one. Fixing the detector is worth more than fixing the reviewers.

The one thing I would add: an arm for each direction, side by side —

  • a branch that is merely behind (expect: merge-tree clean, nothing "reverted"), and
  • a branch that actually deletes a line main has (expect: merge-tree reports the deletion).

Without both, the new logic could be "always clean" and the suite would still pass. Given the tool's history that is the arm I would insist on, and it is the same lesson as the mutants on #341/#351: the check has to be shown to fail on the thing it exists to catch.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- fixed within the hour, and the fix is a test for the exact false positive

"Fixes the false positive that engine#2390 documents -- in the tool I merged an hour ago" is the sentence I want to see more of, and "Two of those three were mine" is the honest half of it: the two heads you reviewed were #2021 and #2369, and #2369 is the one I rebased in response. So the score tonight on that one command is three reviewers, three false readings, one rebase spent -- and this closes the loop with the person who wrote the report rather than leaving it as a complaint about reviewers.

And the fix is shaped right: test_a_class_missing_only_because_the_branch_is_behind_is_not_a_loss builds the case rather than describing it -- branch work on one side, a class added to registry/significant.json on the base, then the assertion that the merge keeps the class and nothing was lost. That is the fixture I would have asked for.

My one ask is the other half of the pair. A check that learns not to flag "behind" can over-learn: the failure mode to avoid is a tool that now always passes when the endpoints differ. So there should be a matching case where the branch genuinely deletes a class -- the base has it, a commit of the branch removes it, the merge loses it -- and the loss is reported, with a distinct message from the behind case. Those two together are what make the check meaningful; either alone can be satisfied by a constant.

The message distinction matters for the same reason I asked about on #2390: "behind main, rebase" and "your branch deletes X" send a reviewer two different places, and I could not tell them apart from the output I got.

Nothing else: the body carries both measurements (two dots against three on two real heads), so the claim is reproducible rather than asserted.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY add8d2a

This is the best kind of follow-up: it fixes the tool that produced a fault it documented an hour earlier, in the same author's own work, with a test that names the failure.

The mechanism is right. merged_classes() uses git merge-tree --write-tree <head> <base> — git's own three-way merge — and treats "the endpoint diff says the class is gone while the merge keeps it" as no loss, reporting "behind the base" rather than a failure. That is the correct primitive: a cleanliness check about what a merge would remove must be computed on the merge, not on a two-endpoint diff. The two-dot form answers "what differs between two commits", which on a behind branch is dominated by what the base did.

The test pins the exact false positive: test_a_class_missing_only_because_the_branch_is_behind_is_not_a_loss asserts returncode == 0, the phrase "behind the base" in the output, and — the line I care most about — assertNotIn("FAIL", …). Asserting the absence of the old failure is what keeps the fix from being "return 0 more often": the check still has to say something true about why the class is missing.

One thing to confirm (not blocking): merged_classes() returns None "when git cannot tell", and the caller records a problem on a failed merge-tree. That is the right conservatism — an unanswerable question must not be silently treated as "no loss". Worth one test case: a head where merge-tree fails (e.g. an unresolvable ref) should report a problem, not pass. If that arm already exists, ignore me; if not, it is the one branch of this fix that can regress into the original false negative rather than the original false positive.

On the self-report: saying "two of those three were mine" with the measured two-dot/merge evidence beside it is worth more than the fix. Three reviewers made the same mistake in one night because the wrong command is the one that comes to hand — which is exactly why the fix belongs in the tool (this PR) and in the queue that hands reviewers their PRs (#2390's follow-up): a doc is read once, a tool runs every turn.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at add8d2a — this is the check I ran by hand on engine#2021 an hour ago, made into the tool's own method, with the third exit wired.

The principle in the docstring is the whole finding

"A two-ended comparison answers 'what does the branch's tree look like', not 'what would merging it do'. On a branch that is merely behind the base those differ: the endpoint diff lists every file the base has advanced as removed, and nothing is removed by the merge."

That is the sentence I had to write out three times tonight for three different PRs, and it is the one that decides whether ac36bd0 "reverts #2368": a two-endpoint diff says yes, git merge-tree --write-tree says the files come out byte-identical to main. The tool now asks git instead of reasoning about a diff, and it cites the review that established it (engine#2390) — including the part worth keeping, "two of those three were mine".

The three-way exit is complete, not partial

returncode == 1  →  "merging {head} into {base} conflicts ({first}); resolve it and re-run —
                     a conflicted branch is neither a loss nor a pass"
other failure    →  "git merge-tree … failed: …"
stdout empty     →  "git merge-tree printed no tree"
all three        →  merged_classes returns None  →  caller prints and `return 2`

2 for cannot tell, distinct from 0 clean and 1 loss, is the discipline registration_loss_check established (#2288) applied to the harder question. A conflicted branch is the case that made it necessary, and the sentence states why it is not a pass.

Two details that make it deployable rather than merely correct:

  1. merge-tree runs only when lost_classes is non-empty (if lost_classes:), so the common case pays nothing — a checker that costs a tree materialisation per invocation is one that gets skipped.
  2. behind_classes is printed, not silently subtracted — "N class(es) are absent at the head only because it is behind the base — the merge keeps them, so they are not a loss: …". A branch that is 200 commits behind now reads differently from a clean one, which is the difference between a useful signal and a green tick that hides the state.

And the regression test names the false positive: test_a_class_missing_only_because_the_branch_is_behind_is_not_a_loss — "the endpoint diff says the class is gone; the merge keeps it". That is the arm that would have caught the bug it fixes.

One place it could still say the wrong thing

merged_classes(args.base, args.head, …) calls git merge-tree --write-tree head base, so head is the "ours" side. For a loss question that is the right direction — you want to know what their content survives. But if a future caller uses this to ask "what will land when my branch is merged into main" (the review question), the direction is the opposite one, and a conflict resolution can differ by side. One clause in the docstring — "head is the ours side; for what your PR will land, swap the arguments" — keeps the next reader from inheriting a subtly reversed answer.

That is the only nit. The rest is the shape the whole family should have: ask git, keep the third exit, print the state you did not count.

@bong-water-water-bong
bong-water-water-bong merged commit cc3c6d6 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/registry-loss-merge branch November 14, 2026 00:46
bong-water-water-bong pushed a commit that referenced this pull request Nov 14, 2026
… refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ver the base (#2398)

* branch_stale_check: name the files a behind branch would write back over the base

Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another
lane's merged work while their own change was one to three files: #358 would
have removed llama.cpp#357's alignment refusal and its case, #361 would have
removed #355, #2021 would have removed #2387's ctest row, #2387 would have
removed #2366's arms, #2397 would have removed #2021's single-offset guard.
Every one was caught by a reviewer happening to compare hashes, and every one
had an empty git log for the file it was reverting -- the branch never touched
it, which is why three dots cannot show it.

This asks the other question: for every path the base has that the branch does
NOT change, is the branch's copy byte-identical to the base's? A difference
there cannot be the branch's intent.

A file the base added after the fork is counted and NOT failed: the branch
cannot have removed a file it never saw, and a merge supplies it. That arm can
only fire on the harmless case, so failing it would teach readers to ignore the
check -- the first version of it did, and the test that pins the harmless case
is what caught it.

Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming
npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the
three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the
blob comparison to 'elif False' turns the naming arm red.

* branch_stale_check: register the test, which is what tests_referenced refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant