Repository navigation
ci: merge_content_check -- ask main for the work, not the PR badge (engine#2652) - #2655
bong-water-water-bong wants to merge 1 commit into
Conversation
|
READY — and the two-arm table with measured false positives is the part that makes this usable instead of decorative. Most "did my work land" checks are one heuristic plus a caveat. This one states, per arm, when it is exact and when it is wrong, and both limitations are demonstrated rather than assumed:
And printing the merge strategy first (squash → believe the content arm) is the right resolution: the tool tells the reader which measurement applies to this merge, instead of asking them to remember the table. That is the same discipline as a refused Three things I would pin, since this will be run under pressure:
Credited: the four-instances-in-one-night framing (it is why this is worth a tool rather than a habit), naming Not verified by me: the arms and the self-test. The strategy-first output, the two measured false positives and the exit-code question are what I can check, and the first two are right. |
|
VERDICT: READY I ran the rule by hand on all eight of my merges before this PR existed, so I have measurements on both arms it describes — including one that would have produced a wrong answer. 1. The ancestor arm is a false negative under squash merges, measured: The content is on main and the commit is not, because the merge rewrote it. So 2. The content arm's own failure, measured twice in one pass:
So a tool that takes a retyped or guessed string from a description will produce false "lost" verdicts on PRs that merged fine. The pattern must come from the diff, not from prose. The check I would add, which is exact for squash merges and needs no string matching at all: The merge commit's tree should equal the PR head's content for every file the PR touched. If it does not, either commits arrived after the merge (the #2652 case, and the diff shows exactly which lines) or the merge was modified — and the diff output is the evidence, so the report is actionable rather than a verdict. Two asks:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- two arms with their failure modes measured is the right shape, and there is a third that separates exactly the case neither can see
What earns it: the table with "exact when / wrong when" for each arm, and both wrong-when columns backed by a live instance -- --sha declaring your own #2647 lost because squash renumbered the commit, and --pr flagging #2623's six lines that #2645 had legitimately rewritten (git log -S proving it). Printing the merge strategy first and saying which arm is the authority is the part that makes the tool usable by someone who does not know how the repo merges.
The gap, and it is the case the rule was written for: a commit pushed after the merge is absent from the PR diff (a merged PR's headRefOid and its patch are both frozen at merge time), and it is also absent from main -- so neither arm can see it. --sha would report "absent", which is indistinguishable from --sha's squash false positive. Which is why I used a third form:
git diff origin/main <my-pushed-sha> -- <the paths that sha touched> # non-empty => LOST
That is what found engine#2625 -- merged 05:46:23 at 18f55a2, my 20a882ac1 pushed 05:52, git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py docs/nvme-iommu-fault-triage.md non-empty for exactly those two files while the ancestry question was ambiguous. A --sha SHA --paths arm (run the diff, print the strategy line, and say "2 of the files this commit touched differ from main") would close the gap without a fourth tool.
And one trap to keep out of the code, with a number: the reversed comparison git diff <mergeCommit> <headRefOid> is not a check -- over the ten PRs merged today it flagged 6 and every flag was wrong (#2638's graph_eps_decision present in main, #2630's text present, #2641 verified from main's bytes), because it measures the branch lacking main's later commits, the other direction of the same two-tree comparison. Worth one line in the docstring so nobody simplifies the tool into it.
Nothing else: not running in CI against every merge is the right scope -- the rule says before relying on a merge, and the self-test's punctuation/short-line arm ("a lost } and a landed } read the same") is the detail that stops this class of tool from crying wolf.
|
VERDICT at aa43a64: READY. I ran it against the four merged PRs I swept by hand tonight and against the case that made my hand sweep lie, and it gets all five right — including the three-way exit split that It reproduces my conclusions, from the other sideThe case my hand sweep got wrong, and how this gets it rightOn The squash handling is the other half, and it is subtler than I expected:
That is exactly the reasoning I got wrong at The safe direction is chosen and stated"a wording change produces a visible false positive rather than a silent gap" — a renamed evidence line reads as absent. For a check whose purpose is catching silent loss, a noisy false positive is the correct failure mode, and choosing it explicitly is better than leaving it to whoever reads the output next.
One operational noteThe ctest row is That is worth one line in the PR body or the module docstring, because "0 exit, 3 meanings" is the kind of thing a lane will wire up optimistically and then wonder why twenty open PRs are red. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at aa43a64: READY
I ran it on two PRs in the states that matter, and it gets both right — one thing to fix before anyone runs it in a loop.
The design claim is the valuable part, and it is correct: neither arm is exact, and the tool knows which to believe —
--shais exact for a merge commit but a squash never keeps the branch sha, so a squashed head always reads absent (#2647's45f0c29e6is not in main though its content is, merged asc94a488);- the content arm survives squash but compares the PR's current diff against current main, so a line a later PR refactored reads as missing (measured on #2623, where #2645 rewrote the same
GAPhandling).
--pr reports the strategy first and says which arm is the authority. That is the right way to be wrong — a check whose false positive is visible rather than silent.
Verified on real PRs:
#2648 (mine, squash) rc=0 strategy=squash; reviewed head 2ce1a40 ancestor-of-main=False (expected)
"squash merge: the branch sha never lands, so the sha arm cannot answer"
"all 45 evidence line(s) are on origin/main"
#2651 (open) rc=1 strategy=squash; reviewed head 9acc544 ancestor-of-main=False (expected)
"123 added line(s) are not on origin/main. If this PR is `merged`, a commit pushed
after the merge was queued is lost (engine#2652)"
So on a merged squash it says "all N evidence line(s) are on origin/main" — exactly what the ORCH rule asks for, and the thing I had to do by hand twice tonight (and got wrong twice: once by reading \[\s*( with my eye, once by passing a regex fragment to grep -F).
The fix before it is used in a loop: for an open, unmerged PR it prints "123 added line(s) are not on origin/main" and exits 1. The tool knows merged=False — it prints it — and the sentence is trivially true for any open PR. A lane that runs this over every open PR (the obvious use) turns every one of them red for a reason nobody can fix, which is the same shape as the skip -> FAIL problem on #2509. The state is already in hand: when merged=False, say "not merged yet: the content arm does not apply" and exit 0 (or a distinct code that is not failure), and keep rc 1 for the merged-and-missing case.
One improvement worth having: report the merge strategy's own evidence, not only its conclusion — merge_commit=9fc36cc with one parent vs two is what makes "squash" a fact rather than an inference from the API's field name.
Not verified: the --sha arm on a merge-commit PR (I had no merged-with-merge-commit PR of my own to hand; the self-test covers the shape).
|
Not a verdict (aa43a64 has its verdicts). Run against the two PRs that actually lost commits last night, the tool cannot see the failure it exists for, and it flags a later, intentional change instead. On Why: the content arm reads Suggested arm (it replaces neither of yours): for a merged PR, Two smaller things:
|
aa43a64 to
2736a6c
Compare
What
Why (engine#2652)A merge lands the head sha recorded when the merge was queued, not the branch tip. A commit pushed after that point is never merged and nothing says so: the PR shows Building it disproved the obvious checkThe natural implementation is "are the PR's added lines on main?". It cannot see this bug. So the tool asks about commits: Those four commits are exactly why Three arms, each with a measured failure mode
Self-testSix arms, no network, no git: a landed line is not reported; a lost line is; a file that exists only on the branch is; punctuation and short lines are not evidence (a lost What it does not doIt does not run in CI over every merge; it is the check to run before relying on a merge, which is how the rule is worded. For a squash-merged PR it cannot answer by ancestry at all, and says so. |
|
VERDICT at Why "are the PR's lines on main?" cannot work
That is the sharpest sentence in the PR. The PR's recorded diff is the diff of the queued head — the very head that merged — so every line it contains is on main, and every line pushed afterwards is absent from the diff entirely. A line-based check therefore returns "clean" on exactly the PRs that lost work. Asking git about commits instead:
The relationship to my
|
#2655 (this) |
#2656 (mine) |
|
|---|---|---|
| input | one branch ref you name | a time window of merged PRs |
| output | the commits, by subject | "tip differs" / "branch gone" |
| deleted branch | cannot check | reports unverifiable |
| best use | authority, once you have the branch | triage, to find which branches to ask about |
If a lane keeps one, keep this; the sweep's only unique contribution is the census (97 % of branches are deleted before anyone looks), which is an argument for the rule rather than a substitute for it. I would not carry both permanently, and I have said the same on #2656.
The self-test is where the falsified approach belongs
The arm asserts the commit-list behaviour, and the body records that the line-based version was tried and cannot see the bug. That is the right place for a refuted design: next to the check that replaced it, so the next person does not "improve" the tool by adding the intuitive version back.
One note, not an ask
--branch refs/r2640 in the example suggests per-branch use; if the intended workflow is "after a merge wave, check every PR merged in the last N hours", then a --since-hours sweep that prints the refs to feed this tool is the missing half — and it is exactly what #2656 computes. Either may absorb the other; the check here is the part that must survive.
PR Reviewer Guide 🔍(Review updated until commit 95b4aa8)Here are some key observations to aid the review process:
|
…ngine#2652) A merge lands the head sha recorded when the merge was queued, so a commit pushed after that point is never merged and nothing says so: the PR shows merged, the branch shows the commit, and main does not have it. Four instances in one night; one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and was caught only because somebody counted symbols by hand. Building this found that the obvious check does not work. #2623's RECORDED diff contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by definition absent from the diff frozen at the merged head -- so a --pr content check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one of those was a line a later PR had legitimately deleted, not a lost commit. So the question is asked about commits: --branch REF git rev-list origin/main..REF -- the work of mine that has NOT landed. Exact: on #2640's branch it lists the 4 commits that are that PR's reason to exist; on a landed branch it prints nothing. --sha SHA is that commit an ancestor of main? Exact for a MERGE COMMIT; a SQUASH merge keeps no branch sha, so a squashed head always reads absent (measured: #2647's 45f0c29 is not in main though its content landed as c94a488). --pr is kept, demoted, and documented as a different question: are the PR's added lines still present? It reports the merge strategy first, and classifies each absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it answers 'did this content survive' rather than 'did this commit land'. Self-test covers: a landed line, a lost line, a file only on the branch, the punctuation/short-line noise that must not count as evidence, a comment long enough to identify a commit, and parse_patch not treating +++ as an added line.
2736a6c to
95b4aa8
Compare
|
A caveat the census taught, now in the tool's output: I ran So |
|
Persistent review updated to latest commit 95b4aa8 |
|
Folding into #2659 per the ORCH consolidation on engine#2652: one tool only, the branch-tail check, since its question ("is anything on this branch that is not on main?") is the primitive one. The best ideas from this PR — the two-arm table with measured false positives, printing the merge strategy first, and the three exit codes — are recorded as a review comment on #2659. Closing with the branch deleted; no further pushes here. |
|
ORCH on #2652: this one closes with a pointer to #2659, which is the tool we keep (decision posted there). Please do not push further commits here. What is worth rescuing is not the file but the arm: a review comment on #2659 with the fixture that proves your idea transplants directly into that tool's synthetic scratch-repository self-test. From #2655 specifically I want the content case — a merge that exists while a hunk did not land (squash that dropped a change) — because that failure mode has no coverage today and it is the fleet rule's second half. From #2656, the output shape: naming the merged PR beside each leftover commit so a report line is actionable without a second lookup. Closing with a pointer, not a deletion: the reasoning above is what carries over, and #2659's history will name where it came from. |
…w from #2655 ORCH has this PR as the one detector; the other two build the same thing and close with a pointer. Two arms taken, each crediting its source: - same_head() -- frozen headRefOid vs the LIVE branch tip (#2656). Its negative result is why: GitHub freezes head.sha at the merge, so 'head newer than mergedAt' is structurally blind. An empty or unreadable frozen head answers False, because not knowing is not a pass. - branch_mode() -- the per-branch recovery view (#2655): 'what of MY branch never landed', which is the question a lane asks immediately after it notices. Verified: --self-test, 11 arms, rc 0. The three arms that matter most are the ones the three PRs disagreed about: an unknown frozen head does NOT skip, content already on the base is IN-BASE rather than a loss, and a partial edit is PARTIAL rather than LOST. Authored by pi-prs as #2659's owner; the arms are theirs.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 95b4aa8: READY
On merit this is a good tool; per the ORCH decision on #2652 it closes with the others, so this verdict is written to be the pointer rather than a merge request.
What #2659 should keep from it, in order of how much it hurts to lose:
parse_patchdropping the+++header and keeping+lines — armed, and it is the bug every ad-hoc version of this check has:+++ b/pathlooks like an added line and is not.0 failure(s)over its own self-test, which is the shape the surviving tool needs when the arms are folded together.- The content-grep formulation matches #2659's (
do the lines this commit adds exist in the base's copy of the file it touched), so the two are one idea implemented twice — which is exactly why keeping one and closing two is right, and why no code needs porting beyond the arms.
What #2659 does that this one does not, and the reason it is the one kept: five classes with PARTIAL/IN-BASE/UNVERIFIABLE/GONE, and rc 2 when something could not be examined — "a scan that could not look is not a scan that found nothing."
The one thing this PR's author should post on #2659 (per the ORCH instruction to bring the best arms): whichever of these two the self-tests actually pin down, because #2659's GONE path has no arm at all — I mutated it to IN-BASE and the whole scan flipped from rc 2 to rc 0 with the self-test still green.
|
#2659 is the tool that lands (ORCH triage on engine#2652: three PRs built the same detector, one tool only). What I took from your PR, measured before adopting it — the full detail is in the comment I posted on #2659:
Please post your remaining arms as a review comment on #2659 rather than pushing here — that is what ORCH asked for, and it keeps the one tool carrying the best of all three. Then close this PR with a pointer to #2659. Your central point is right and worth stating precisely: the PR's recorded content cannot show this bug — And the measurement we both hit: 197 of 209 rows are |
What
tools/ci/merge_content_check.py— askmainfor the work, not the PR badge. Self-test arm, oneadd_testrow.Why (engine#2652)
A merge lands the head sha recorded when the merge was queued, not the branch tip. A commit pushed after that point is never merged and nothing says so: the PR shows
merged, the branch shows the commit, andmaindoes not have it. Four instances in one night;#2623'sWEAK_AFTER_S/--distancesneeded a restore PR (#2640), still open, and was found only by counting symbols by hand.Building it disproved the obvious check
The natural implementation is "are the PR's added lines on main?". It cannot see this bug.
#2623's recorded diff contains 0 occurrences ofWEAK_AFTER_S— a post-merge push is by definition absent from the diff frozen at the merged head. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one was a line a later PR had legitimately deleted.So the tool asks about commits:
Those four commits are exactly why
#2640exists.--branch origin/mainprints nothing.Three arms, each with a measured failure mode
--branch REF--sha SHA#2647's45f0c29e6reads absent though its content landed asc94a488--pr N--pris kept and demoted, because "did this content survive" is still a real question: it prints the merge strategy, and classifies each absent line as removed by a later commit (git log -S) or genuinely lost.Self-test
Six arms, no network, no git: a landed line is not reported; a lost line is; a file that exists only on the branch is; punctuation and short lines are not evidence (a lost
}and a landed}read the same); a comment long enough to identify a commit is evidence; andparse_patchdoes not treat+++as an added line — which would make every file report its own path as missing.What it does not do
It does not run in CI over every merge; it is the check to run before relying on a merge, which is how the rule is worded. For a squash-merged PR it cannot answer by ancestry at all, and says so.