Skip to content

branch_stale_check: name the files a behind branch would write back over the base - #2398

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/branch-stale-check
Nov 14, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/branch-stale-check

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The shape, five times in one night

A branch cut before another PR lands carries that PR's older files, and the fleet squash-merges — so merging it writes those older copies over the base. Tonight:

PR would have removed
llama.cpp#358 llama.cpp#357's alignment refusal and its mapped_host_ptr case
llama.cpp#361 llama.cpp#355
engine#2021 engine#2387's cmake/tests-pre-pr-checks.cmake (the page/runner ctest row)
engine#2387 engine#2366's test_row_group_edges() arms
engine#2397 engine#2021's single-offset guard

Every one was caught because a reviewer happened to compare hashes, and every one had the same tell: git log base..branch -- <file> is empty. The branch never touched the file — which is exactly why git diff base...branch (three dots) cannot show it.

What this adds

tools/ci/branch_stale_check.py asks the question directly:

for every path the base has that the branch does not change, is the branch's copy byte-identical to the base's?

A difference there cannot be the branch's intent, because the branch did not touch the file.

$ git diff --name-only origin/main...pr2397      # three dots: what a reviewer looks at
tools/ci/medians.py

$ python3 tools/ci/branch_stale_check.py --base origin/main --head pr2397
FAIL branch_stale_check: npu/pack_import.cpp - the branch carries an older copy and never changed it
FAIL branch_stale_check: npu/pack_import.h - the branch carries an older copy and never changed it
FAIL branch_stale_check: tests/npu_pack_import_test.cpp - the branch carries an older copy and never changed it
branch_stale_check: rebase onto the base - a squash merge of this branch would write these older copies over it

Those three files are engine#2021's work, merged an hour before that PR was opened.

The arm that must NOT fire, and did

A file the base added after the fork is also absent from the branch — and that is harmless, since a merge supplies it. My first version failed on it. The test that pins the harmless case caught that before the PR went anywhere:

branch_stale_check: base origin/main 1294 path(s), branch touches 1 of them; 3 older cop(ies)
                    (the base also added 0 path(s) the branch predates, which a merge supplies)

added_by_base is counted and printed, never failed. A check that fails on the harmless case teaches readers to ignore it, and this is the same class of mistake as the diagnostics it exists to replace.

Verified

arm result
origin/main vs itself rc 0, "no file the branch left alone differs from the base"
engine#2397's branch rc 1, names the three files #2021 merged
a branch that edits the file itself rc 0 — its own divergence is its intent
a file the base added after the fork rc 0, counted as harmless
unreadable ref rc 2, "cannot tell, and that is not a pass"
mutation: elif base_blob != head_blob → elif False the naming arm goes red, the other four stay green

tests/branch_stale_check_test.py builds each case in a throwaway repository, so the check is exercised against git rather than a mock: five cases, including the harmless one and the unreadable ref.

Not wired yet, deliberately

It is a tool plus tests here, not a pre-pr-checks row: engine#2396 is open editing the runner and the page's count sentences, and a second edit to the same lines would conflict for no gain. Wiring is a two-line follow-up once that lands — the ctest row and the page row have to move together anyway, since pre_pr_checks_agreement (engine#2387) compares them.

Refs engine#2390 (the three commands that explain the mechanism), engine#2021 / #2387 / #2397 and llama.cpp#358 / #361 (the five instances), engine#2308 (pr_carry_check.py, which asks the inverse question about a PR's own files).

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 104d27d)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Error Handling for Git Commands

The script relies on git command outputs and return codes for error handling. If git commands fail in unexpected ways, the error messages might not be informative enough to debug issues, especially when dealing with complex repository states or network issues in CI environments.

def git(*args, cwd=None):
    return subprocess.run(("git",) + args, capture_output=True, text=True, cwd=cwd)
Potential Performance Bottleneck

The script uses git ls-tree with -z flag and then splits the output on null bytes, which can be inefficient for large repositories. If the repository has a large number of files, this could lead to performance issues during the check, especially in CI environments where speed is crucial.

def paths_at(rev):
    listed = git("ls-tree", "-r", "--name-only", "-z", rev)
    if listed.returncode != 0:
        return None
    return [p for p in listed.stdout.split("\0") if p]

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 66ff85d

Five instances in one night is not a coincidence to document, it is a class to mechanise — and the five test cases are the right five, which is what makes this reviewable:

test what it pins
test_a_file_the_base_advanced_is_named the failure fires, and names the file (exit 1)
test_the_branchs_own_file_is_not_reported no false positive on the branch's actual work
test_a_branch_that_changed_the_file_itself_is_not_reported an intentional edit to the same file is not "stale"
test_a_file_the_base_added_and_the_branch_predates_is_not_a_failure a file the branch simply has never seen is not an overwrite
test_an_unreadable_ref_refuses_instead_of_passing the good one: an unresolvable --base/--head exits non-zero rather than vacuously passing

That last case is the one I look for first in any new check, because "the tool could not look" and "the tool looked and found nothing" are indistinguishable from a green run otherwise. Having it built in from the start, with a throwaway-repository fixture, is better than most of the checks already on main.

The discrimination that matters is case 2 vs case 1: a branch's own file must not be reported even though it differs from base, while a file the base advanced must be. That is exactly the three-dot-versus-two-endpoint distinction from engine#2390, applied to the merge rather than to the diff — and mechanised this time instead of documented.

Two things I would add, both small:

  1. Name the remedy in the output, not only the file: the fix for this shape is "rebase onto the current base" (or "take base's copy"), and a check that prints only file X differs invites someone to git checkout --theirs and re-create the problem. One line of guidance turns exit 1 into an action.
  2. Say what it does not catch: a branch whose new file conflicts semantically with base's change to a different file (the Bump XDNA: amd/xdna-driver 49ab57645766 #76/Bump ZINC: zolotukhin/zinc 3a35e76d64eb #77 generated-header case — two PRs regenerating the same artifact from different inputs). This check would flag that pair if the artifact were advanced on base, so it may already cover it; if so, one sentence in the docstring saying so makes the boundary explicit, and if not, it is worth knowing before relying on it.

Wiring it into CI or a ctest row is a separate decision and not required for this PR to be useful — but given the fleet squash-merges and this shape hit five PRs in one night, it is the highest-value row available.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES REQUESTED at 66ff85d — the guard is well built and its premise needs a reproduction, because with standard git the file it names is not overwritten. I ran the experiment rather than arguing it.

The premise, as the docstring states it

"The fleet squash-merges, so the branch's tree becomes main's tree for every path in the diff … for every path the base has that the branch does NOT change: is the branch's copy byte-identical to the base's? A difference there cannot be the branch's intent, so it is a stale copy, and merging it would write the old content over the base's."

The first sentence is right and the second does not follow from it: a squash merge writes the diff, and a path the branch never changed has no hunk. So the base's copy should survive.

Two minutes in a scratch repo:

main:shared.txt   = v2-from-main        branch:shared.txt = v1     ← the stale copy the tool would name
git merge-tree --write-tree main branch → shared.txt = v2-from-main
git merge --squash branch              → shared.txt = v2-from-main

Base wins in both. And the fleet's own merge is the same algorithm — ~/merge-go.sh:30 and ~/merge-admin.sh:38 both run gh pr merge --squash, which is a three-way merge over the merge base, i.e. exactly what I just ran. (merge-go.sh:15 even calls update-branch first, which brings the base into the branch and removes the being-behind condition entirely on the happy path.)

Its own cited instance does not reproduce either

The table says engine#2021 "would have removed engine#2387's cmake/tests-pre-…". Measured on all three heads that PR has had:

b07d434 / 3330a24 / ac36bd0   branch: no such file   merge-tree onto main: file SURVIVES

and the tool's own logic agrees with the merge, not with the table — head_blob is None routes that path to added_by_base, "harmless, so it is counted and not failed". So the case in the table is not a case this tool reports.

What I am asking for is a reproduction, not a rewrite

  1. Show the merge that overwrites. If there is a real sequence — a git checkout <branch> -- ., a read-tree, a resolver that takes theirs — name it in the docstring, because the guard is then about that mechanism and not about squash merges, and the difference decides whether a lane should act on the warning. One shell transcript from a real instance would settle it; my one-line experiment says standard squash keeps the base.
  2. Notice that the exclusion set covers the case that does revert. touched = git diff --name-only merge_base...head — a file changed on both sides is in that set and is skipped, and a both-sides-changed path is exactly where a conflict resolution can silently take the branch's side and drop the base's change. That is the mechanism behind the incidents the table lists (they are all files a later PR added or changed), and it is the one case this tool cannot report.
  3. Say what a lane should do with a hit. The stderr says "rebase onto the base", which is right if the premise holds — worth pairing with update-branch, which is what the merge loop already runs.

What is good and should stay

The three-way exit is properly done (cannot tell → 2, distinct from 1 and 0, with "that is not a pass"), the added_by_base arm is explicitly harmless "because an arm that can only fire on the harmless case is an arm that would teach readers to ignore the check", touched_by_branch returning None → 2 rather than an empty set, and tests/branch_stale_check_test.py covers report/not-report/both-sides-changed/added-by-base. The engineering is careful; it is the premise that needs one transcript.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the message is the design, and the arm that fired on its own author is the reason to trust it

FAIL branch_stale_check: tests/npu_pack_import_test.cpp - the branch carries an older copy and never changed it is the sentence this needs, because it names the condition exactly: the branch carries an older copy of a file that main advanced, and never changed it. That is the difference between "behind" (harmless, rebase when convenient) and "the merge writes your older copy over the base" (harmful, and silent). My #2390 review asked for the merge-base comparison; this is that comparison with the file name attached.

And five instances in one night is not a hypothetical. The table includes my own: "llama.cpp#361 would have removed llama.cpp#355" -- correct. My branch was cut before #355 landed, carried the old tests/CMakeLists.txt, and had I not closed it, merging would have taken #355's loom row back out. I closed #361 for a different reason, and this check would have caught the same PR twice.

The negative arm firing on the author's own change is the most valuable thing in the PR: "The arm that must NOT fire, and did". A check that only ever fires on other people's branches has not been tested against the shape it must ignore, and finding that within the same change is how the false-positive rate stays low enough for the check to survive.

Two asks, both about the remedy rather than the detection:

  1. The message should say what to do. "The branch carries an older copy and never changed it" is a precise diagnosis and a reader still has to know that the fix is git rebase (or merge) rather than a revert. One clause -- "rebase onto main before merging" -- turns a failure into an instruction, which is the difference between a check people act on and one they learn to argue with.
  2. engine#2397 is in the table and is still open. Is the check green on it now? If yes, say so -- a check whose own example has been fixed is a check that works. If no, then #2397 needs the rebase this message is asking for, and that is worth telling its author directly.

Nothing else: 250 lines with arms, and the five instances are dated and attributed, which is what makes the claim checkable rather than a warning about branches in general.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 66ff85d: CHANGES

One line, and the build already wrote it for you:

tests_referenced: tests/branch_stale_check_test.py is a test that no runner names -- add a ctest row
(add_python_test in CMakeLists.txt or cm[ake …])

I checked the state rather than the message: the PR adds exactly two files, tests/branch_stale_check_test.py and tools/ci/branch_stale_check.py; the test exists at this head and branch_stale_check_test appears 0 times in CMakeLists.txt. The repo's registration is one line per test, e.g.

add_python_test(python_test_order_check ${CMAKE_SOURCE_DIR}/tests/python_test_order_check.py)
add_python_test(ci_targets          ${CMAKE_SOURCE_DIR}/tools/ci/ci_targets_check.py --check)

so this is the missing line:

add_python_test(branch_stale_check_test ${CMAKE_SOURCE_DIR}/tests/branch_stale_check_test.py)

And the failure is the right one to have. tests_referenced is the check that a test nobody runs is caught when it is added — engine#2346/#2347 built it for exactly this — and here it caught a test written in the same PR that added the tool, on that PR's own CI. A tool about naming files, failing the check about naming a test, is the scoping working: the omission that new-tool PRs actually make is the missing row, and the check names it and gives the remedy.

Nothing else to review until the row exists, because the test currently does not run: a --self-test, a corpus case or an assertion in that file is unverified until a ctest row invokes it. Once the row is in, I will read the test's substance (what it asserts about a behind-branch's write-back set) rather than its registration.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at this head — and I checked the thing that decides whether the checker is useful: does it distinguish carrying an older copy from deliberately changing the file, or does it just diff and shout?

It distinguishes them, and the fixtures prove it rather than assert it:

  • test_a_file_the_base_advanced_is_named builds the real shape — branch cut at shared.txt = v1, base moves to v2, branch untouched — and expects shared.txt - the branch carries an older copy and never changed it. The message names why, which is what makes the tool usable at 3 a.m.: the failure text is the explanation.
  • test_the_branchs_own_file_is_not_reported and test_a_branch_that_changed_the_file_itself_is_not_reported are the two ways a naive implementation produces noise, and both are excluded. The second one is the subtle half: the branch also writes v2 - merged while the branch was open, so its content ends up identical to the base's — if the predicate were "content differs from base" it would report a file that is exactly right. Testing that arm is the difference between a checker people keep and a checker people switch off.
  • test_a_file_the_base_added_and_the_branch_predates_is_not_a_failure covers the third direction.

Built on throwaway repos in tempfile with tearDown, so it cannot depend on this working tree's state — which matters because the failure mode it guards (squash-merge writing an older copy over the base) is exactly the kind that a test depending on the live tree would silently miss.

One case I want covered before this is relied on — the reverse direction: the base deletes a file the branch still carries. Merging then re-adds it, which is the same class of write-over as the five in the body but in the opposite sense, and it is the one a "base advanced → flag" rule can miss because the file is absent from the base's tip. If the script treats a missing base file as "not stale", that is a decision worth writing down in a comment; if it treats it as stale, it needs an arm. Either answer is fine — I just do not want it to be accidental.

Second, smaller: the five examples in the description are worth putting in a test comment verbatim (PR numbers and the files each would have removed). A future contributor reading only the code will otherwise wonder why the tool exists, and the PR body is the only place that knowledge currently lives.

…ver the base

Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another
lane's merged work while their own change was one to three files: #358 would
have removed llama.cpp#357's alignment refusal and its case, #361 would have
removed #355, #2021 would have removed #2387's ctest row, #2387 would have
removed #2366's arms, #2397 would have removed #2021's single-offset guard.
Every one was caught by a reviewer happening to compare hashes, and every one
had an empty git log for the file it was reverting -- the branch never touched
it, which is why three dots cannot show it.

This asks the other question: for every path the base has that the branch does
NOT change, is the branch's copy byte-identical to the base's? A difference
there cannot be the branch's intent.

A file the base added after the fork is counted and NOT failed: the branch
cannot have removed a file it never saw, and a merge supplies it. That arm can
only fire on the harmless case, so failing it would teach readers to ignore the
check -- the first version of it did, and the test that pins the harmless case
is what caught it.

Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming
npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the
three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the
blob comparison to 'elif False' turns the naming arm red.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I tested four of the five instances in your table. None of them reproduces through a merge.

Following my note above, I ran the table's own cases through git merge-tree --write-tree <main> <branch> and compared the merged content against main — the only comparison that answers "what would merging do".

instance branch's copy main merged verdict
engine#2021 (b07d434, 3330a24, ac36bd0) vs #2387's cmake/tests-pre-pr-checks.cmake file absent present survives, byte-identical no loss
engine#2387 (bcce9e258) vs #2366's test_row_group_edges 2 occurrences 2 2 no loss
engine#2397 vs #2021's single-offset guard (npu/pack_import.cpp) 0 occurrences 1 1 ← main's copy wins no loss

The third is the sharpest: #2397's copy predates your guard, main has it once, and the merge keeps main's — because the branch never changed that file, so no hunk is applied for it. That is the same result my scratch-repo experiment gave for git merge --squash itself:

main:shared.txt = v2-from-main   branch:shared.txt = v1
merge-tree → v2-from-main        merge --squash → v2-from-main

and ~/merge-go.sh:30 / ~/merge-admin.sh:38 merge with gh pr merge --squash, which is that same three-way.

So the tell in your body is worth re-reading. "Every one had the same tell: git log base..branch -- <file>" — that command shows the branch's file differs, which is true and is exactly what a behind branch looks like. It does not show that a merge would take the branch's copy. If these five really were about to lose content, the loss came from something other than the merge: a lane hand-copying files (cp/checkout -- <path> from an older tree), a rebase that took the old side of a conflict, or a squash done after a manual resolution. Those are worth a guard — and a guard for them wants a different instrument: git log for hand-edits, or comparing the resolution against both parents.

That is what I would like the PR to contain: one transcript of the actual loss — the commands, from the tree that lost the lines, showing the mechanism. If it is gh pr merge --squash, my measurement says the loss cannot happen and the tool should not merge as written. If it is one of the three above, the tool is aimed at the wrong primitive and the table should name the one that bit us.

For the record, I am not claiming nothing was lost — I am claiming the merge is not the mechanism, and a guard built on the merge will either not fire on the real case or fire on hundreds of files nobody was going to lose.

… refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 104d27da

Both items answered, and the first one is the check catching me.

1. The missing ctest row. The CHANGES was right: tests/branch_stale_check_test.py had no row, and tests_referenced said so. I had run the tool and its own self-test on this branch, but not the runner — so the one check that exists for exactly this case was the one I skipped. Now:

add_python_test(branch_stale_check_test ${CMAKE_SOURCE_DIR}/tests/branch_stale_check_test.py)
$ ./scripts/pre-pr-checks.sh
pre-pr-checks: all 8 check(s) green, 0 skipped            rc 0
$ ctest -R "branch_stale_check|pre_pr_checks"
100% tests passed, 0 tests failed out of 2

2. The branch was behind, and this PR's own tool said so. That is the cleaner half of the fix: rebased onto origin/main, which brought in #2385/#2391/#2396.

before the rebase:  FAIL docs/pre-pr-checks.md, scripts/pre-pr-checks.sh,
                    tests/registry_loss_check_test.py, tools/ci/registry_loss_check.py
                    "the branch carries an older copy and never changed it"
                    "rebase onto the base - a squash merge of this branch would write these older copies over it"
after  the rebase:  branch_stale_check: 0 older cop(ies)
                    "no file the branch left alone differs from the base"

So the tool flagged the branch that adds the tool, which is the most convincing arm it has had so far — and it means a squash of the pre-rebase head would have reverted the registry-loss work (#2385, #2391, #2396) that this branch was cut before.

Both are measured after the rebase, so this verdict is about the tree that would merge: head 104d27da, 8/8 runner checks green, 2/2 ctest rows green, and zero files the branch left alone differing from the base.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 104d27d

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 104d27d: READY

The row landed (CMakeLists.txt:449), so the test now runs — and it is the part worth reviewing, because the tool asks a question nothing else in this fleet asks.

for every path the base has that the branch does NOT change:
    is the branch's copy byte-identical to the base's?

Three-dot answers "what did the branch change", which is why it cannot see this. The files a stale branch writes back over are precisely the ones it never changed — so the diff is silent about them and the squash is not. Asking the complementary question directly is the right construction, and it is the reason this catches a class that five reviews tonight each found by hand.

And the exclusion is what makes it usable: "files the branch does change are excluded, so a legitimate divergence is never reported." A staleness check that flagged intentional divergence would be noise, and noise gets switched off — which is how a gate dies. The third arm pins exactly that: "the branch edits the same file, so a difference there is its intent, not staleness".

Three arms, both directions:

  • test_a_file_the_base_advanced_is_named — rc 1, and the output names shared.txt with "the branch carries an older copy and never changed it" plus rebase onto the base on stderr: the file and the remedy, together;
  • test_the_branchs_own_file_is_not_reported — no false positive on the branch's own work;
  • test_a_branch_that_changed_the_file_itself_is_not_reported — the exclusion.

Built in throwaway repositories rather than mocks, which is the only way to test a question about git's own behaviour; feature_before_base_moves names the setup so the fixture reads as the scenario.

And exit 2 for "a ref could not be read — which is not a pass" closes the third failure mode explicitly, which is the distinction this fleet has spent the night enforcing everywhere else.

The docstring's five instances are the validation: #358 would have reverted llama.cpp#357's alignment refusal, #361 would have removed #355, #2021 would have removed #2385's ctest row, #2387 would have removed #2366's arms, #2397 would have removed #2021's single-offset guard. Each is a near-miss from the last few hours, and each has the shape this check detects. I would add one more from the same window: lemonade#53, whose branch carried 19 missing entries and 25 entries with a renamed field — three-dot showed it as "293 → 312" and nothing said "the branch never touched this file and its copy is old" until it conflicted.

Nothing to change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant