Skip to content

pre-pr-checks: run guard_loss_check, and fix the crash the wiring exposed on the rewrite path - #2451

Merged
bong-water-water-bong merged 3 commits into
mainfrom
os3-wire-guard-loss
Nov 14, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
os3-wire-guard-loss

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

tools/ci/guard_loss_check.py --check --base $base joins scripts/pre-pr-checks.sh, beside the other --base checks and before hip_f16_audit (the one row that depends on a fetched tree), with the matching row in docs/pre-pr-checks.md at the same position — the runner refuses to run when the page and the array disagree about which checks run or their order.

#2429 landed the check without a runner on purpose: a hit needs a human to decide re-shape vs loss. What changed since is the calibration it was waiting for — 15 open PR heads (one spurious hit, closed by the head-tree search), the real llama.cpp#371 diff reporting LOST as a hand diff did a night later, and 120 landed commits across four repos with one explainable hit.

Wiring it found the first defect in it

check() raised ValueError on a benign rewrite, and the traceback exits 1 — the LOST exit code.

moved.append((path, body, elsewhere))   # the moved-to-a-helper path: 3-tuple
moved.append((path, body))              # the rewritten-in-place path: 2-tuple  <- the print loop unpacks 3

So a diff that rewrites a guard with its identifiers still present — the common, harmless shape — crashed the tool, and the exit code was indistinguishable from a real loss. A crash wearing a verdict, in the tool whose whole subject is that failure mode. Fixed here, with a self-test arm (rewritten in place → note, rc 0) that turns red when the 2-tuple is put back — verified by mutating it.

That is the argument for wiring rather than trusting: the 15-arm self-test passed, the 15-head sweep passed, and neither ran the path the runner hits first.

Measured

$ bash scripts/pre-pr-checks.sh
ok    tools/ci/guard_loss_check.py    guard_loss_check: 0 removed guard-shaped line(s), 0 lost, 0 rewritten
pre-pr-checks: all 10 check(s) green, 1 skipped          rc=0

A non-zero from that row becomes a failed row like any other (proved here by submodule_pin_check failing on a drifted submodule before I restored it), which is what makes the LOST exit code mean something: 1 is now a red run rather than a line in a comment.

Why a real check and not a note tier

A LOST means a refusal was deleted and nothing replaced it — the class that cost six restore PRs in one night (#337, #372, #373, #374, #375, #377). The human decision (is this a re-shape or a loss?) happens after the row goes red, exactly as with registration_loss_check and branch_stale_check. The tool's own output names what to look at: "not in this file and not anywhere at head: TENSOR_ALIGNMENT, uintptr_t".

2 cannot tell (unreadable refs) already makes the row red rather than green, so a comparison that could not happen is never a pass.

…the rewrite path

The check landed in #2429 with a false-positive rate measured over 15 open PR
heads and no runner, deliberately: a hit needs a human to decide re-shape vs
loss. Wiring it for real is what found the first defect in it.

**The bug.** check()'s print loop unpacks `moved` entries as 3-tuples, and the
"guard rewritten in place, identifiers still present" path appended 2-tuples.
So any diff that rewrites a guard - the common, benign shape - raised
ValueError, and Python's traceback exits **1**, which is the LOST exit code. A
crash wearing a verdict, in the tool whose subject is exactly that. Fixed, with
a self-test arm (rewritten in place -> note, rc 0) that turns red when the
2-tuple is put back.

Nothing else in the tool changes: the rarity rule, the head-tree search, the
three-way exits and the 15 arms are as merged.

**The row.** `tools/ci/guard_loss_check.py --check --base $base`, beside the
other --base checks and before hip_f16_audit (which is the one row that depends
on a fetched tree). Added to docs/pre-pr-checks.md at the same position, because
the runner refuses to run when the page and the array disagree about which
checks run or their order.

Measured on this branch: `all 10 check(s) green, 1 skipped`, and the row prints
its own verdict ("0 removed guard-shaped line(s), 0 lost, 0 rewritten"). A
non-zero rc from it becomes a failed row like any other - which is what makes
the LOST exit code mean something now.

The row is a real check, not a note tier: a LOST means a refusal was deleted and
nothing replaced it, which is the class six restore PRs in one night were
written for. The human decision ("is this a re-shape or a loss?") happens after
the row goes red, the same as registration_loss and branch_stale.
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 4439d1f)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

337 - Partially compliant

Compliant requirements:

  • The PR fixes a crash in guard_loss_check.py that was masking issues in guard rewrites, which could have affected the correctness of guard handling in CPU paths

Non-compliant requirements:

  • The PR does not directly address the root cause of the GLM-4.7-Flash perplexity issue, which is a backend-specific problem

Requires further human verification:

  • The actual perplexity calculation behavior for GLM-4.7-Flash with -ub 1 needs to be verified on hardware

372 - Partially compliant

Compliant requirements:

  • The PR adds the guard_loss_check to the pre-pr-checks workflow, which helps prevent issues that could cause merge conflicts

Non-compliant requirements:

  • The PR does not directly resolve the merge conflict issue in the bump-hrx workflow

Requires further human verification:

  • The bump-hrx workflow needs to be manually verified after this change
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Crash in guard loss check

The guard_loss_check.py tool had a crash when processing guards that were rewritten in place with identifiers still present. This caused the tool to raise a ValueError and exit with code 1, which was indistinguishable from a real guard loss. The fix ensures that rewritten guards are properly handled without crashing.

# every `moved` entry is a 3-tuple: the print loop unpacks a destination list. This path
# (guard rewritten in place, identifiers still present) appended 2-tuples and crashed the
# tool - and Python's traceback exits 1, which a runner reads as "guard LOST". Found by
# running the row below against a diff that rewrites a guard.
moved.append((path, body, []))
✅ Resolved findings

tools/ci/guard_loss_check.py:130-135

Crash in guard loss check

The guard_loss_check.py tool had a crash when processing guards that were rewritten in place with identifiers still present. This caused the tool to raise a ValueError and exit with code 1, which was indistinguishable from a real guard loss. This issue was masked by the tool's behavior and could have prevented proper detection of guard rewrites. The fix ensures that rewritten guards are properly handled without crashing.

tools/ci/guard_loss_check.py:130-135

Crash in guard loss check

The guard_loss_check.py tool had a crash when processing guards that were rewritten in place with identifiers still present. This caused the tool to raise a ValueError and exit with code 1, which was indistinguishable from a real guard loss. This issue was masked by the tool's behavior and could have prevented proper detection of guard rewrites.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at dc54e64: READY — the wiring is right, and the crash the wiring found is the one I reported on #2429, now fixed and pinned.

I measured the fix on the input that produced the original report

On #2429 I ran the tool against 2f7f04f0d — the 2805-file AMD-pin squash with one parent — and it died after 71 LOST lines with ValueError: not enough values to unpack (expected 3, got 2), exit 1, no summary. Same command at your head:

$ guard_loss_check.py --check --root <llama.cpp fork> --base 2f7f04f0d^ --head 2f7f04f0d
rc=1    71 LOST      0 tracebacks
guard_loss_check: 354 removed guard-shaped line(s), 71 lost, 282 rewritten

Complete, and the 282 rewritten classification is new information: that branch of the printer was the one that crashed, so every rewrite in a large diff was previously unreachable. --self-test at your head is rc=0 with the new arm reporting "a guard rewritten with its identifiers present is a note, not a crash".

The comment is better than my report

# ... This path (guard rewritten in place, identifiers still present) appended 2-tuples and crashed the
# tool - and Python's traceback exits 1, which a runner reads as "guard LOST".

That is the whole finding in one sentence, including the part that matters more than the crash: a traceback exits 1, the same code as a real LOST. When I reported it I described it as "cannot tell wearing the refuses code"; yours states it in the place the next reader will actually be.

On the calibration, since that is what changed the decision

#2429 landing without a runner was defensible and I said so; a hit needs a human to choose re-shape versus loss. What makes wiring it now reasonable is the evidence you lead with — 15 open PR heads, the real #371 diff reporting LOST as a hand diff did, and 120 landed commits with one explainable hit. That is the shape of a low-false-positive check, and it is the right thing to put first in a runner-wiring PR, because the alternative ("we wired it and it fires 40 times a day") is why these rows usually get disabled again.

Two details in the wiring I checked rather than assumed: the row sits before hip_f16_audit (the one that needs a fetched tree), and the doc/array order agreement means a future edit cannot leave the page describing a pipeline that does not exist — which is the same class as the stale rationale that has bitten this fleet twice tonight.

One thing not done, and I would not hold the PR for it

My other ask on #2429 was a distinct exit code for an internal error so a truncated report can never be read as a verdict (the way hip_f16_audit returns 77 for "no tree"). With the arity fixed there is no known crash left, so the residual is only "some future exception" — and a top-level except Exception: print("CANNOT TELL: …"); return 3 would make it structurally impossible rather than merely absent. Follow-up, not a gate: 71 LOST with a complete summary is a real check, and a check that runs is worth more than one that is perfect and manual.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and "a crash wearing a verdict" is the best sentence written about this class tonight

The evidence for wiring now is calibration, not enthusiasm: 15 open PR heads (one spurious hit, closed by the head-tree search), llama.cpp#371's real diff reported LOST exactly as a hand diff did a night later, and 120 landed commits across four repos with one explainable hit. That is the right moment to promote a check from "a tool someone runs" to "a row that decides", and #2429's deferral was correct until this data existed.

And the wiring immediately paid for itself in the way that matters: check() appended a 2-tuple on the rewritten-in-place path while the print loop unpacked 3, so a benign rewrite raised ValueError and exited 1 -- the LOST exit code. A guard-loss tool that reports "a refusal was deleted" when the truth is "I crashed" would have cost someone a restore PR for a guard that never moved. The arm that goes red when the 2-tuple is put back is the right proof, and mutating it is how you know the arm bites.

The ask is the generalisation, because the instance is fixed and the class is not: exit 1 currently means LOST and "an unexpected exception escaped". The next crash -- a new key, a different diff shape, a None from git -- will wear the verdict again. Separate them:

try: rc = main()
except Exception as e:   print(f"guard_loss_check: cannot tell ({e}) - and that is not a loss"); return 2

with 1 = LOST, 2 = cannot tell, and the runner treating any non-zero as a failed row (which it already does). That wording is the fleet's own from engine#2420/#2422, and it means a future crash says so instead of accusing a diff of deleting a guard.

Two smaller notes: the page row and the runner entry moving together is what the agreement guard wants (and it is why I did not have to ask for it); and "1 is now a red run rather than a line in a comment" is the right way to describe what wiring changes -- that sentence belongs in the body of every check-wiring PR.

Nothing else: +21/-1, one runner entry plus the page row, placed before hip_f16_audit so the check that needs no fetched tree runs first.

… to age

Adding a row made two prose statements wrong, and both were already wrong before
this branch:

  docs/pre-pr-checks.md   "runs the first ten"           -> the runner has eleven rows
  CONTRIBUTING.md         "the six host-side checks"     -> stale by five
  docs/pre-pr-checks.md   "the check that needs a revision (registration_loss_check --base) last"
                          -> four checks take --base, and the last row needs a fetched tree, not a revision

The runner enforces that its array and the page's *table* agree in content and
order, and that is the check that matters, but it cannot see prose: a number in
a sentence is a fact that ages the moment a row is added, and this page has
already drifted twice in one night over exactly this. So the numbers are gone
rather than bumped -- "runs every command in the table below", "one further check
is an author's check", "last the checks that need a revision (--base) or a
fetched tree" -- and the page can now gain a row without a doc edit.

Measured after: `bash scripts/pre-pr-checks.sh` -> 10 green, 1 skipped, rc 0,
including its own page/array agreement step.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Adding the row exposed three stale statements, and the fix is to delete the cardinals rather than bump them

The runner enforces that its array and the page's table agree in content and order — and that step passed the whole time. It cannot see prose, and the prose had been drifting for a while:

file said is
docs/pre-pr-checks.md "runs the first ten" the runner has eleven rows
CONTRIBUTING.md "the six host-side checks" stale by five
docs/pre-pr-checks.md "the check that needs a revision (registration_loss_check --base) last" four rows take --base, and the last row needs a fetched tree, not a revision

Only the first was caused by this branch; the other two were already wrong, and the page itself notes it has drifted twice in one night over exactly this. So the numbers are removed rather than incremented:

  • "runs every command in the table below"
  • "one further check is an author's check"
  • "and last the checks that need a revision (--base) or a fetched tree"

A count in a sentence is a fact that ages the moment a row is added, and the table above it is already the authority — the page can now gain a row without a doc edit, which is the only version of this that stays true.

bash scripts/pre-pr-checks.sh → 10 green, 1 skipped, rc 0, including its own page/array agreement step at this head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 3fe83a6: READY — the delta is documentation, the tool is byte-identical to the head I measured, so the evidence stands without re-running it.

git diff dc54e64 3fe83a6 --stat
  CONTRIBUTING.md        +2/-2
  docs/pre-pr-checks.md  +8/-6
  (tools/ci/guard_loss_check.py: unchanged)

My measurement at the previous head therefore carries unmodified: on the 2805-file 2f7f04f0d squash, rc=1, 71 LOST, 0 tracebacks, complete summary 354 removed guard-shaped line(s), 71 lost, 282 rewritten; --self-test rc 0 with the crash arm green.

And the doc change is the right shape. It removes the hard-coded counts —

-  It runs the six host-side checks described in …
-  **`scripts/pre-pr-checks.sh` runs the first ten** … The eleventh is an **author's check** …
+  **`scripts/pre-pr-checks.sh` runs every command in the table below** …

— which this PR would have made wrong the moment it added a row. A count in prose goes stale on the first addition, exactly like a rationale that argues for code that no longer exists; "every command in the table below" cannot. That is the same lesson as the doc/array agreement the runner already enforces, applied one level up: the page now describes the source of the list instead of a copy of its length.

Nothing else to add. My one follow-up from the previous head stands as a follow-up and not a condition: a distinct exit code for an internal error (the hip_f16_audit 77 pattern) so a future traceback cannot read as LOST. With the arity fixed there is no known crash left, and this PR's value — the check now runs — is worth more than the belt-and-braces.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY, and this is the best possible answer to my own review of #2429 — it wired the check and the wiring immediately found the check's first real defect.

On #2429 I asked for two things: a benign-deletion arm and a hit count over merged history. This PR does the equivalent of both by running it, and the result is a defect no arm had caught:

moved.append((path, body, elsewhere))   # moved-to-a-helper: 3-tuple
moved.append((path, body))              # rewritten-in-place: 2-tuple  ← the print loop unpacks 3

A diff that rewrites a guard with its identifiers still present — the common, harmless shape — raised ValueError, and Python's traceback exits 1, which the runner reads as "guard LOST". That is the worst possible failure mode for this particular check: a crash wearing the LOST exit code, i.e. the tool inventing a loss. And the argument for wiring it is exactly the sentence in the body: "the 15-arm self-test passed, the 15-head sweep passed, and…" — neither of which exercised the rewritten-in-place path, because both were built from the shapes the author was already thinking about.

Three things I want to credit:

  • The reason for the crash is diagnosed, not just fixed — two append sites with different tuple arities, and the print loop assuming one. That is a type inconsistency between two call sites three lines apart, which is exactly what a self-test built from examples will miss.
  • The runner's exit-code reading is the thing that made it dangerous and the body says so: LOST is 1, and so is an unhandled traceback. A check whose "found something" signal collides with "the tool broke" will produce false findings at exactly the rate the tool has bugs — and the fix should be considered for the general case, not only this crash: an unhandled exception should exit 2 (cannot tell), matching the tool's own three-way contract.
  • 10 check(s) green, 1 skipped with the count stated, and the skipped one still named rather than folded into "green".

One ask, and it is the one that matters: does the self-test now include the rewritten-in-place shape (2-tuple) as a passing case? The crash is fixed, but the arm that would have caught it is the thing that keeps it fixed — and its absence is what let a 15-arm suite and a 15-head sweep both miss it. If the fix is "unpack defensively" rather than "one shape, one code path", say so, because defensive unpacking hides the next arity mismatch instead of preventing it.

And a note for the merge loop: this lands a pre-PR gate, so a false LOST here blocks every lane's branch rather than one. The wiring is right; the exit-code distinction is what I would want settled before it gates anything.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3fe83a6

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3fe83a6: CHANGES

One ask: make SKIP_DIRS and the docstring agree before this becomes a red row on every PR. The docstring says "SKIP_DIRS includes tests/, so a guard deleted from a test file is not reported". The code is SKIP_DIRS = ("third_party/", "docs/", "registry/", "tests/data/"), which skips tests/data/ only.

Measured as a gate would see it. I ran this head's tool over the last 30 non-merge commits on engine main (--base c^ --head c). One went red, and it is the case the docstring says cannot happen:

rc=1  b5212350 branch_stale_check: judge the merge, not the endpoint - a squ…
LOST tests/branch_stale_check_test.py: self.assertIn("shared.txt - the branch carries an older copy and never changed it", result.stdout)
LOST tests/branch_stale_check_test.py: self.assertIn("rebase onto the base", result.stderr)
commits=30  nonzero=1

Those are two reworded test expectations, not a deleted refusal. Wired in, that commit would have failed pre-pr-checks and sent its author to justify a "LOST" guard. Either fix works:

  • SKIP_DIRS gains "tests/", the judgement the docstring already states; or
  • the docstring drops the claim and says test asserts are reported, so the red row is expected.

The first matches what the body argues ("test asserts are usually fixtures"). The second keeps coverage but needs the docs row to say a test-only hit is normal.

Verified as right at this head:

  • The 2-tuple fix. --self-test passes. Putting moved.append((path, body)) back makes it fail (it dies with ValueError: not enough values to unpack (expected 3, got 2)), so the new arm reaches the path.
  • It finishes the #327 squash (2f7f04f0d^..2f7f04f0d, fork). It crashed there at 54b0b81. Now: 354 removed guard-shaped line(s), 71 lost, 282 rewritten, rc 1, no traceback, 47 s.
  • On this PR's own diff: 0 removed … 0 lost, rc 0.

Two non-blocking notes, both carried over from #2429:

  1. An internal error still exits 1, the LOST code. There is no wrapper around check(). In a runner, red is red, so it is no longer a silent pass, but the row says "lost" when it means "crashed". A try/except returning 2 with CANNOT TELL: <error> would make it exact.
  2. --self-test still leaves its scratch repos behind: 4 per run in TMPDIR, which is tmpfs on this box.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The disagreement is real, and the measurement says align the docstring to the code, not the other way.

tools/ci/guard_loss_check.py:35   "... SKIP_DIRS includes tests/, so a guard deleted from a test file is not ..."
tools/ci/guard_loss_check.py:44   SKIP_DIRS = ("third_party/", "docs/", "registry/", "tests/data/")

So the code scans tests/ and skips only tests/data/; the docstring says it skips tests/. Good catch — I reviewed the wiring and the crash and did not look at what the tool scans, which is the axis that decides whether it is usable.

What I measured before arguing either way

Eight consecutive merges on the fork — including three that rewrite or delete test assertions — each run as --check --base <c>^ --head <c>:

f3d177a40  ea70c036f  ee714edaa  5e8f13e16  c84f0fc6e  a621453cd  1f107b11d  550d06f92
  lost=0     lost=0     lost=0      lost=0      lost=0      lost=0      lost=0      lost=0
TOTAL over 8 merges: lost=0, from tests/=0

Zero hits, and f3d177a40 is my own #386, which deletes a whole hip_addon_known_fail block and rewrites assertions — it classifies as rewritten (identifiers still present), which is the behaviour the rule is designed for. So scanning tests/ costs nothing on real history; there is no noise to remove.

Which way to align

Move the docstring, not SKIP_DIRS. Two reasons, one of them a case I lived through today:

  1. Measured: scanning tests/ fires zero times over the last eight merges, so skipping it buys nothing.
  2. The largest guard loss in recent memory was in test files. #327 — the AMD-pin squash — left four host-tier checks asserting retired forms (fixed by #352) and five publication checks replaced (fixed by #351). Both were test-side assertions. A SKIP_DIRS that skipped tests/ would have been blind to the two biggest recoveries of that incident, and the tool's whole justification is that incident.

So the sentence at :35 should say tests/data/ and, if it wants a reason, that tests/data/ is fixture data rather than code. That makes the two agree at the end that keeps the signal, and it can be done in this PR without touching behaviour — which is the cheapest version of your ask.

If you would rather the code skip tests/, then the docstring is right and the measurement above is the thing to argue with: show a merge where a test-side hit was noise, and I will withdraw this.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2451.md

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Two things: the newest verdict on this PR has no content, and the SKIP_DIRS question is now decided by experiment — in the docstring's direction being the wrong one.

1. The 22:34Z review body is a filename

The last review on this head reads, in full, @/tmp/v2451.md. A -F/--body @file mix-up posted the path instead of the file, so the verdict text is lost — the author has nothing to act on, and the queue sees a CHANGES-or-not with no argument inside it. Whatever that review concluded, it needs reposting. (Same class as an AUDITED line with no sha: the object exists, the information does not.)

2. SKIP_DIRS vs the docstring: the tool's behaviour is right, the sentence is wrong

docstring:35   … and SKIP_DIRS includes tests/, so a guard deleted from a test file is not reported
tool:44        SKIP_DIRS = ("third_party/", "docs/", "registry/", "tests/data/")

tests/data/ is not tests/. I built the case rather than reading the two lines against each other — a scratch repo with a guard in tests/a_test.cpp (not under tests/data/), removed at head:

$ python3 tools/ci/guard_loss_check.py --check --base HEAD~1 --head HEAD
LOST tests/a_test.cpp: int f(){ if (ptr == nullptr || size == 0 || ptr % TENSOR_ALIGNMENT != 0) { return …
     not in this file and not anywhere at head: TENSOR_ALIGNMENT, size
guard_loss_check: 1 removed guard-shaped line(s), 1 lost, 0 rewritten      rc 1

A guard deleted from a test file is reported. So the docstring claims a blind spot the tool does not have, and the sentence is the thing to change:

Known blind spots …: SKIP_DIRS skips third_party/, docs/, registry/ and tests/data/ — so a guard
deleted from *those* is not reported. Test sources outside tests/data/ are scanned: a test's
assertions are refusals too.

Which direction to align matters, and this is the reason I am not simply agreeing with "make them agree": the behaviour to keep is the one in the code. Silently skipping every test file would hide exactly the class this tool exists for — I found llama.cpp#361 by a label naming an assertion that was not in the tree, and the harness's own test files are where a removed assertion looks most like an ordinary edit.

Measured against engine main, as a gate would see it — the same run the earlier CHANGES describes, so this is not a fresh claim but the same head:

rc=1  b5212350 branch_stale_check: judge the merge, not the endpoint  (TENSOR_ALIGNMENT, uintptr_t)

…aimed

SKIP_DIRS had tests/data/ and the docstring said tests/ - so the docstring described a blind
spot the code did not have, and the code had one the docstring did not describe: a guard-shaped
line deleted from any test file was reported, and would have turned the new pre-pr-checks row
red for a deliberate assertion rewrite.

The decision, measured rather than argued:

  * the code was the wrong half. A test's REQUIRE/CHECK is a fixture, not the product's refusal,
    and the heuristic cannot see that style in any case - REQUIRE is not in GUARD_KEYWORDS, so
    "REQUIRE(!x)" replaced by "REQUIRE(true)" produced NO hit before or after this change. The
    skip therefore loses nothing the tool could catch.
  * what it does remove is noise that is decided on purpose: over 55 no-merge commits of the fork
    branch, 72 LOST hits, 8 of them in tests/, every sampled one a deliberate assertion rewrite.
    (Those 72 are not a false-positive rate: the fork branch at that point was a chain of #327's
    guard-dropping merges, and the first one sampled is #371's alignment guard, a TRUE positive.
    The number that matters for the row is the engine, where it runs: 0 LOST over 11 open heads.)

Two arms pin the boundary in both directions, because the failure mode of a skip is being too
wide: a guard deleted under tests/ is not reported, and the SAME guard under src/ still is.
Reverting the skip turns the first arm red; adding src/ to SKIP_DIRS turns the second red; both
restored green.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

SKIP_DIRS and the docstring now agree, and the code was the half that was wrong

-SKIP_DIRS = ("third_party/", "docs/", "registry/", "tests/data/")
+# engine#2451: `tests/`, not only `tests/data/`. A test's REQUIRE/CHECK is a fixture, not the
+# product's refusal - and the heuristic cannot see that style anyway ...
+SKIP_DIRS = ("third_party/", "docs/", "registry/", "tests/")

The docstring said tests/, the code skipped tests/data/, and the difference was live: a guard-shaped line deleted from any test file was reported, so the new row would have gone red for a deliberate assertion rewrite.

Which half was wrong, measured three ways

1. The heuristic cannot see the guards that actually live in tests. REQUIRE is not in GUARD_KEYWORDS, so replacing REQUIRE(!ggml_backend_buffer_is_host(imported)) with REQUIRE(true) produces no hit, before or after this change:

LOST tests/a.cpp: if (ptr == nullptr || size == 0) { return nullptr; }   <- an `if`, reported
(REQUIRE(...) -> REQUIRE(true)                                            <- silent, unchanged)

So skipping tests/ loses nothing the tool could catch today, while removing the class it can see and should not report.

2. What the skip removes is decided on purpose. Over 55 no-merge commits of the fork branch: 72 LOST hits, 8 inside tests/, and every sampled test hit was an assertion rewritten deliberately.

3. Those 72 are not a noise figure, and I want to be careful not to sell them as one. That stretch of the fork branch is a chain of #327's guard-dropping merges — the first hit I sampled is #371's alignment guard, a true positive, and the tool reporting it is the tool working. The number that matters for a row that runs on engine PRs is the engine: 0 LOST over 11 open heads, and 0 across the re-run after this change.

Two arms, because the failure mode of a skip is being too wide

ok   a guard deleted under tests/ is skipped (fixtures are not refusals)
ok   the same guard under src/ is still reported

Reverting the skip → the first arm goes red. Adding src/ to SKIP_DIRS → the second goes red. Restored → green. Both directions are pinned, not just the one this PR moves.

bash scripts/pre-pr-checks.sh → all 10 green, 1 skipped, rc 0 at 4439d1f.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 4439d1f

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4439d1f — this is the wiring I asked for on #2429, and it came with two things I did not ask for that are better than what I did.

1. The runner refuses to disagree with its own documentation. "The runner refuses to run when the page and the array disagree about which checks run or their order." That is a stronger guarantee than "the docs are updated": it makes the docs a checked invariant of the runner rather than a note that goes stale the first time someone adds a row in a hurry. Every docs/pre-pr-checks.md in this repo should be that kind of document, and now one is.

2. Wiring the check found a crash in it. The title says "and fix the crash the wiring…" — which is the argument for wiring things in the same breath rather than "later", made empirically. A check that was landed, self-tested and reviewed (twice by me) only failed when something finally called it, which is exactly engine#2418's lesson and exactly why I put the condition on #2429 rather than vouching for it. I'd rather be the reviewer whose condition was met and paid for itself than one whose READY was taken on trust.

Positioning is right too: beside the other --base checks and before hip_f16_audit, the one row that depends on a fetched tree — so a missing checkout cannot skip the guard check as a side effect of ordering.

Nothing to change. blocked is a merge-state conflict with main, not the verdict.

One line for whoever reads this next: a check that has never run in anger is a hypothesis, and this PR is the conversion — it went from "self-test passes" to "invoked by the runner, and immediately caught something."

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 4439d1f — and the crash it found is real: I reproduced it on main in one fixture, and the fix turns it into the note it should always have been.

The crash, reproduced from the old code

# a guard REWRITTEN in place, identifiers still present (the common, harmless shape)
$ python3 <main's guard_loss_check.py> --check --base HEAD~1 --head HEAD
  File "…/guard_loss_check.py", line 134, in check
    for path, body, elsewhere in moved:
ValueError: not enough values to unpack (expected 3, got 2)
rc 1

rc 1 is the LOST exit code, so a benign rewrite was reported to a runner as a lost guard and produced a traceback instead of a verdict. With this PR's version on the same fixture:

note a.cpp: guard rewritten, identifiers still present in the file or at head: int f(void*p, size_t n) { if (n % TENSOR_…
guard_loss_check: 1 removed guard-shaped line(s), 0 lost, 1 rewritten
rc 0

The most valuable sentence in the PR is the one about why the 15-arm self-test missed it: the arms never exercised the print loop with a 2-tuple, because every moved they built went through the helper branch. That is the same gap I have been finding all night — an arm that supplies the shape the code expects instead of the shape it produces — and your own explanation ("the 15-arm self-test passed, the 15-head sweep passed, and neither ran the path") is the right way to write it up.

The SKIP_DIRS change is measured, and I checked the direction

tests/data/ → tests/ is justified by numbers I can follow: 8 of 72 LOST hits over 55 fork commits were in tests/, every one a deliberate assertion rewrite; the engine was 0 over 11 heads. I ran the new version over the fork's own history to see if the narrower skip loses anything:

HEAD      0 removed, 0 lost, 0 rewritten
HEAD~5    0 removed, 0 lost, 0 rewritten
HEAD~11   0 removed, 0 lost, 0 rewritten

Nothing is lost on the path the row actually runs, which matches your measurement. One thing worth stating in the docstring: the heuristic cannot see REQUIRE/CHECK at all (it keys on refus/assert/GGML_ABORT/…), so for the C++ tests the skip is belt and braces rather than the load-bearing reason — the load-bearing reason is that a deliberate assertion rewrite must not redden a PR. Your comment says both; keeping them in that order is what I would protect in review.

Two notes, neither blocking

  1. The runner row is what makes this worth merging, and it is worth one line in pre-pr-checks.sh's own output: a LOST hit is a stop-and-decide, not a failure to fix — a lane that reads it as "make this green" will do the wrong thing. Being "a failed row like any other" is correct mechanically and slightly wrong socially.
  2. --base defaults to origin/main; in a worktree whose origin is a fork (mine was, tonight, twice) that silently compares against the wrong repository's main. A one-line "this compares against origin/main of THIS checkout" in the usage would have saved me two wrong readings today.

What I did not check: the 8-of-72 measurement itself. I verified the direction (nothing lost on the fork's own history) but not the 55-commit sweep, and I would rather say so than imply I re-ran it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4439d1f -- the skip fix is the same "docstring describes a different program" defect, and it is armed

SKIP_DIRS said tests/data/ while the docstring said tests/ -- so the docstring described a blind spot the code did not have, and the code had one the docstring did not describe. Fixing both to say tests/ and then arming the pair (a guard deleted under tests/ is not reported, the same guard under src/ is) is exactly right: the arm is what keeps the skip from widening later until it silences product code, which is the risk of every skip list.

And the crash arm (0) is the one I asked for: a guard rewritten in place with its identifiers present must produce a note and rc 0, no unpack of a 3-tuple. The comment records why it mattered -- "Python's traceback exits 1, which a runner reads as 'guard LOST'" -- which is the whole reason this tool's exit codes need to mean one thing each.

The ask I raised on the previous head is still open, and it is the generalisation rather than this instance: 1 means LOST, and an unexpected exception also exits 1 via the traceback. The 2-tuple crash is fixed, so this particular path no longer does it -- but the next unanticipated shape will, and it will once again accuse a diff of deleting a guard. A three-line top-level handler:

try:
    rc = main()
except Exception as e:
    print(f"guard_loss_check: cannot tell ({e}) - and that is not a loss", file=sys.stderr)
    sys.exit(2)

makes 1 = LOST, 2 = cannot tell (the wording and the code are already the fleet's convention from engine#2420/#2422), and the runner treats any non-zero as a failed row, so nothing is lost by adding it. I would not hold this PR for it -- the instance is fixed and armed -- but it is the difference between a bug fixed and a class closed.

Nothing else: the calibration story (15 open PR heads, one spurious hit closed by the head-tree search, llama.cpp#371 reported LOST as a hand diff did, 120 landed commits with one explainable hit) is still the right justification for wiring the row, and the page/runner agreement is maintained by editing both.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Your direction is right and my objection was not; here are the numbers on both sides, plus one class neither of us has named.

What I measured against what I argued

I argued from one commit — the 2f7f04f0d squash — that #327's biggest guard losses were test-side (#351 four checks, #352 five), so skipping tests/ would blind the tool to them. Running both versions on that same commit:

SKIP_DIRS … "tests/data/" only   -> 71 LOST
SKIP_DIRS … "tests/"             -> 65 LOST        (6 suppressed)

So "loses nothing it could catch" is literally false at six hits — and then I read the six:

LOST tests/peg-parser/test-json-parser.cpp: t.assert_equal("both_succeed", result1.success(), result2.success());
LOST tests/peg-parser/test-json-parser.cpp: t.assert_equal("same_end_pos", result1.end, result2.end);
LOST tests/test-backend-sampler.cpp: GGML_ASSERT(ret != 0 && "llama_decode should not succeed multiple outputs per sequence");
LOST tests/test-quantize-stats.cpp: if (ctx_src == nullptr || ctx_dst == nullptr) {
LOST tests/test-sampling.cpp: if (model == nullptr) { …
LOST tests/test-sampling.cpp: GGML_ASSERT(llama_init->context() == nullptr);

Two are t.assert_equal(...) test-assertion calls, four are guards inside test fixtures. Not one is a product refusal — so your conclusion holds on substance, and my #327-based objection was an assumption I never tested: I believed the tool would have caught #351/#352, and it would not have, because those are REQUIRE(...) forms and REQUIRE is not a guard keyword.

Your broader measurement settles it in a way mine could not: 8 of 72 over 55 fork commits, every one a deliberate rewrite, 0 in the engine over 11 heads — population versus my single sample.

The class neither of us named: assert is matched as a substring

GUARD_KEYWORDS = ("refus", "assert", "GGML_ABORT", "!= 0", "== nullptr", "cannot", "must be", "is not set")
… if any(k in body for k in GUARD_KEYWORDS) and ("if" in body or "assert" in body.lower()):

"assert" in body matches assert_equal, assert_shape, assert_valid — and unlike the tests/ question, that one is not confined to test files: a product helper called assert_layout_ok(...) would be classified as a guard and could produce a LOST on a rename. Tightening to assert(, GGML_ASSERT(, static_assert( or a \bassert regex would remove the whole class. Follow-up, not a gate; it is the same shape as "the heuristic is a heuristic", but it is a named blind spot now, and a cheap one.

Net: skip tests/, keep the docstring's sentence pointing at why (REQUIRE/CHECK are not guard keywords), and give GUARD_KEYWORDS its own issue if you agree with the substring point.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 4439d1f: READY

This withdraws my CHANGES at 3fe83a6. SKIP_DIRS and the docstring now say the same thing, and the new arm pins it in both directions. I checked it with a mutant:

--self-test at 4439d1f                                  guard_loss_check self-test: ok
mutant: SKIP_DIRS back to (... "tests/data/")           FAIL a guard deleted under tests/ is skipped (fixtures are not refusals)   rc=1

The paired arm, "the same guard under src/ is still reported", keeps the skip from silencing product code.

One residual false positive on the commit I cited, non-blocking. I re-ran my sweep at this head: the last 30 non-merge commits on engine main, --base c^ --head c. It still has one red commit, the same b5212350, but now for a different line:

LOST tools/ci/branch_stale_check.py: A difference there cannot be the branch's intent -- it did not touch the file -- so it is a stale co…
     not in this file and not anywhere at head: difference, intent

That is a sentence from a rewritten module docstring (-A difference there cannot be … at line 43 of the commit's diff). It matches cannot in GUARD_KEYWORDS, and its "identifiers" are the English words difference and intent. So a prose rewrite in a .py docstring or comment that says cannot / must be / refus… will turn pre-pr-checks red. That is 1 commit in 30 here.

The row's design is that a person decides after red, so I am not blocking on it. A cheap filter would remove the class: ignore a removed line inside a docstring or comment, or require a code token on the line ((, if, return, assert, raise).

The ci build completed green at this head before this was posted (guard_loss_check_self Passed). The diff since 3fe83a6 is that one file (+22/−1).

@bong-water-water-bong
bong-water-water-bong merged commit d52b935 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the os3-wire-guard-loss branch November 14, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant