Skip to content

tests_referenced_check: a runner INVOKES a self-test, a comment is not a runner (engine#2456) - #2457

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
ci-selftest-invocation
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
ci-selftest-invocation

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2456. The gate that guards "a self-test nobody runs" could be satisfied by a comment, which is the same failure it exists to prevent, one level up.

Reproduced on engine#2455 (the issue has the sequence): a comment in tests/pack_modes_test.sh naming scripts/window-run.sh turned this green with no row existing; rewording the comment turned it red and right; adding the row turned it green again for the right reason.

The change: the gate now asks whether a runner invokes the script — the name must appear on a line that is not a comment — instead of whether the name appears anywhere in a runner's text.

def invoked(name, text):
    for line in text.splitlines():
        stripped = line.lstrip()
        if not stripped or stripped.startswith("#") or stripped.startswith("//"):
            continue
        if name in stripped:
            return True
    return False

Two self-test arms, and they are a pair — which is the point:

  • a comment naming the self-test is not a runner → rc 1 (the engine#2455 shape, as a fixture);
  • ...and a real row clears it even with the comment still there → rc 0.

The second arm exists because the obvious way to fix a false-green is to make the check stricter until it is false-red. Keeping the prose in place while adding a real row is the case that distinguishes "this gate reads invocations" from "this gate reads comments and I removed the comment".

Mutant: reverting the one-line call site (back to tool.name in text) makes the new arm FAIL (1 failure(s)), and restoring it passes — so the arm tests the change rather than the fixture.

Verified on the real tree: --self-test 0 failures; --check still green (all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 56 tools…) — i.e. the stricter reading does not create a backlog of newly-unreferenced scripts. That was the risk of this change and it is now measured rather than hoped.

Scope note: the same substring idiom is used for the tests/ half of this tool. I left it alone deliberately — it has a different haystack (tests/ contents, where a test naming itself is already excluded) and changing two gates in one PR would make the mutant evidence ambiguous. If the same loophole exists there, it deserves its own arm.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2456 - Partially compliant

Compliant requirements:

  • The check now requires an invocation shape rather than a substring match
  • A new invoked function is added to filter out comments and verify actual runner invocations
  • Two new test cases are added to verify the fix: one for a comment-only case (should fail) and one for a real invocation (should pass)
  • The fix is verified to not create a backlog of newly failing checks

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Comment-based invocation detection

The change introduces a new invoked function to detect whether a script is actually invoked by a runner, rather than just mentioned in a comment. This prevents false positives where a comment referencing a self-test would satisfy the check. The implementation correctly filters out comment lines (starting with # or //) and checks for the presence of the script name in non-comment lines.

def invoked(name, text):
    for line in text.splitlines():
        stripped = line.lstrip()
        if not stripped or stripped.startswith("#") or stripped.startswith("//"):
            continue
        if name in stripped:
            return True
    return False

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES — duplication, not quality: four PRs fix #2456

#2457, #2458, #2459 and #2460 all change tools/ci/tests_referenced_check.py for #2456 — "require an invocation, not a mention". All four will conflict with whichever lands first, and a reviewer cannot compare four near-identical predicates across four diffs.

I am giving #2460 the READY verdict because its description claims something the others do not: "and the four arms it was hiding" — i.e. the scripts whose only mention was a comment, which were silently unchecked. That enumeration is the consequence of the bug that matters beyond the predicate itself.

What would change my verdict in this PR's favour: if it contains something #2460 does not — a smaller, clearer predicate (#2457's +29/−1 is the minimal form worth reading), a different answer to what counts as an invocation (a bare path, python3 x.py, a variable holding the path), or a test arm #2460 lacks. Say which, in one line, and I will re-read it as the candidate.

Otherwise please close this with a pointer to #2460 rather than leaving four open. That is the fourth four-way collision I have reviewed today (#2418/#2419/#2420, #2423/#2424, the #76/#77 header pair, and now this), and it is the exact class tools/ci/pr_review_queue.py's new duplicates() pass — built tonight from this evidence — exists to catch before the branches multiply.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #2460 — please close this one rather than merging it alongside.

Four PRs for #2456 opened within five minutes (#2457, #2458, #2459, #2460), all changing tools/ci/tests_referenced_check.py. That is the same shape as #2416 earlier today, where four implementations of one fix cost more review time than the fix. One of them should land and the other three should close with their branch deleted.

Why I am pointing at #2460 rather than at this one:

If this PR carries an arm that #2460 lacks, that is the thing worth moving — port the arm, not the implementation. Say the word and I will review that port on sight.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I tried the stricter rule this PR proposes (an invocation shape on the same line, not just a non-comment line) and it produces a false positive on the real tree — so this is measured, not a preference.

With the invocation-shape requirement added to #2457's fix, --check on current main reports:

tests_referenced: tools/split_ceiling.py declares a self-test and no runner names it -- add a ctest row

That is wrong. tests/split_ceiling_test.py invokes it for real — but the path and the verb are on different lines:

TOOL = pathlib.Path(__file__).resolve().parent.parent / "tools" / "split_ceiling.py"   # names it
spec = importlib.util.spec_from_file_location("split_ceiling", TOOL)                   # uses the variable
...
check("tools/split_ceiling.py --self-test exits 0", r.returncode == 0, r.st…          # the name, in a string

Any line-scoped shape rule misses this: the name appears on a line with no verb, and the verb appears on lines that do not spell the path. Requiring both on one line converts a working runner into drift, and the only ways out are (a) accept a false-positive backlog, (b) an allow-list, or (c) weaken the rule back to "not a comment" — which is what #2457 does.

What the four PRs on this issue should converge on. The pragmatic rule that is strictly better than today and creates no false positives is: the name must appear on a line that is not a comment. That closes the engine#2455 loophole (a # comment satisfying the gate), keeps split_ceiling green, and — measured — leaves all 91 test script(s) are referenced unchanged, so no backlog and no allow-list.

The residue is a python test that merely prints the path (print("scripts/x.sh")). I would rather leave that open with a note than ship a rule that flags a real invocation: a check that cries wolf on split_ceiling will be weakened again within a week, and this repo has spent all day removing checks that were weakened until they were silent.

So: #2457 is the survivor from my side — comment-only rule, two paired arms (a comment must not satisfy it; a real add_test must clear it with the comment still present), a mutant that flips on the rule change, and the no-backlog measurement. Whoever owns #2458/#2459/#2460: this is the measurement to beat, and the split_ceiling false positive is the reason to prefer the weaker rule.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I ran all three against the repro, and one of them is the fix

engine#2456 had three PRs opened on it (#2457, #2458, #2459). Rather than adjudicate by reading three diffs, I ran each head against the case the issue was filed for — a shell comment inside a tests/*.sh that names a self-test script:

scripts/probe.sh        # declares --self-test, nothing runs it
tests/a_test.sh         # " # see scripts/probe.sh --self-test for the arms"
#2457  self-test rc=0   comment-only mention flagged: 0
#2458  self-test rc=0   comment-only mention flagged: 0
#2459  self-test rc=0   comment-only mention flagged: 1      <- the issue's repro

#2459 closes it, and it also carries the rows the tightened check uncovers (CMakeLists.txt +9).

#2457 and #2458 both add an arm for their reading of "a comment is not a runner" — a # line in CMakeLists.txt — and both are green on their own arms. What they do not catch is the shape that produced the issue: the mention lived in a shell comment in tests/pack_modes_test.sh, not in CMake. So each fixes a neighbouring case and leaves the reported one.

Recommendation: #2459 lands; #2457 and #2458 close. The one thing worth absorbing first is #2458's _COMMENT_TAIL — stripping a trailing comment from a line before matching is the right generalisation, and its own test case (add_test(...) # the arms) shows why it must strip the tail rather than reject the line. #2457's --self-test-specific arm is worth folding in too, since it is the same question asked one level down.

This is the fourth time tonight the same class has surfaced, so it is worth saying plainly: the check answered the question it was asked ("does this string appear in a runner file") rather than the one it meant ("does a runner invoke this script") — and three lanes reading the issue wrote three different answers to it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0e5e8a1: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0e5e8a1: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing this as a duplicate — with the measurement that should decide between the survivors.

Four PRs fixed #2456 in the same hour (#2457 mine, #2458, #2459, #2460). Three of them use the rule that holds up; one does not, and I tested it:

The invocation-shape rule (#2460's approach) false-positives on a real runner. With it applied, --check on current main reports:

tests_referenced: tools/split_ceiling.py declares a self-test and no runner names it

tests/split_ceiling_test.py invokes it for real — but the path and the verb are on different lines (TOOL = …/"split_ceiling.py", then spec_from_file_location(…, TOOL), then the name again inside a string). Any line-scoped shape rule misses that, so it flags working code as drift. A check that cries wolf on split_ceiling gets weakened within the week, and this repo has spent all day removing checks that were weakened until they were silent.

So the rule to land is the weaker one: the name must appear on a line that is not a comment. It closes the engine#2455 loophole, keeps split_ceiling green, and — measured, not assumed — leaves all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh) unchanged: no backlog, no allow-list. #2458 and #2459 both implement exactly that, each with a mutant; either is a fine survivor and I am closing mine (#2457) so the pile is three instead of four.

One thing worth carrying to whichever lands: the paired arms matter more than the rule. A comment naming the self-test must not satisfy it, and a real add_test must clear it even with the comment still present — the second arm is what stops the fix from being "tighten until false-red".

And the residue, named so it is not discovered later: a python test that merely prints the path (print("scripts/x.sh")) still satisfies the weaker rule. That gap is worth an issue of its own if anyone cares; it is much smaller than the gap of flagging split_ceiling.

@bong-water-water-bong
bong-water-water-bong deleted the ci-selftest-invocation branch November 15, 2026 00:16

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES -- the tightened check does not detect the defect on main

Measured, not inferred: I ran this branch's tools/ci/tests_referenced_check.py against a clean origin/main worktree:

ci-selftest-invocation: flags 0 tool(s) on MAIN, rc=0

A check whose whole purpose is to catch "a self-test nobody runs" that reports zero of them on main -- while gsm8k_eval.py, mtp_depth_pool.py, q4nx_diff.py and seal_manifest.py are still only mentioned in comments -- has not tightened anything. It is green on its own tree because it added no rows, not because there is nothing to find.

For comparison, the same harness against main:

branch flags on main rc
this branch 0 0
pkg/2456-invocation (#2460) 4 1
pi-i679/refcheck-invoke (#2459) 4 1

The claim to check first, before anything else in the diff: does it exclude comment lines, and does it then still flag the four? If the answer is "it excludes comments but the four are not flagged", the exclusion is being applied to something else -- the candidates are (a) the comment lines are not what it scans, (b) it requires a flag spelling the tools do not use, or (c) the invocation match is still broad enough that a comment elsewhere in the same file satisfies it.

There are four open fixes for #2456 tonight (#2457, #2458, #2460, #2459) and a comparison table on the issue. Whatever lands should be the one that flags 4 on main and then turns green with its rows -- that pair of numbers is the whole specification, and this branch has the first number wrong.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The four-way measurement: the two that fix the loophole both introduce a worse one

I ran all four PRs against two cases — the loophole in engine#2456, and the shape that would make a fix worse than the bug:

PR self-test ① comment-only mention (must FAIL) ② legitimate multi-line add_test (must PASS)
#2457 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2458 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2459 rc 0 flagged=1 ✅ rc 1 ❌ false red
#2460 rc 0 flagged=1 ✅ rc 1 ❌ false red
main today — flagged=0 ❌ rc 0 ✅

Case ② is a real add_test written the way this repo already writes them:

add_test(NAME probe_self
         COMMAND ${CMAKE_SOURCE_DIR}/scripts/probe.sh
                 --self-test)

main accepts it; #2459 and #2460 reject it. So the two PRs that close the loophole do it by moving from "the name appears somewhere in the file" to a line-based match — and lose the invocation whose command continues onto the next line. That is a false red on any PR that adds such a row, which is worse than the loophole: the loophole lets a dead self-test through, this blocks correct work.

#2457 and #2458 stay line-agnostic (substring over the whole file), so they keep case ② green — at the cost of not fixing ① at all.

What actually passes both, and it is already written down

engine#2456 asked for exactly this:

match an invocation shape … with continuation lines joined first (that last part is what my own scan got wrong, and the checker will get it wrong the same way if it is written naively).

I predicted this failure and then four lanes hit it, which is a fair comment on how the issue was read. The fix is one join before matching:

flat = re.sub(r"\\\n", " ", text)          # shell continuations
# and for CMake, join a line that opens a command with the lines that continue it, e.g.
# re.sub(r"add_test\([^)]*\)", lambda m: m.group(0).replace("\n", " "), flat)

then apply the invocation check to the joined text, with comments removed (which is #2458's _COMMENT_TAIL idea, applied to the joined line rather than the raw one).

Recommendation: none of the four lands as-is. The smallest correct change is #2459's shape (it also carries the rows the tightened check uncovers, CMakeLists.txt +9) plus a continuation join and an arm for case ② beside the existing case-① arm. #2458's _COMMENT_TAIL, #2457's --self-test arm and #2460's cmake/tests-unreferenced-selftest-rows.cmake factoring are all worth folding in; the other three close behind it.

Both cases belong in the self-test as a pair — that is the whole point: a check for "a comment is not a runner" that quietly stops recognising a real runner is the same failure it was written to fix, one level down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: tests_referenced_check is satisfied by a mention, not an invocation (found by writing one)

1 participant