Skip to content

tests_referenced: require an invocation, not a mention — and the four arms it was hiding - #2460

Merged
bong-water-water-bong merged 3 commits into
mainfrom
pkg/2456-invocation
Nov 15, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
pkg/2456-invocation

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

tests_referenced: require an invocation, not a mention — and the four arms it was hiding

engine#2456 found the loophole the only way anyone would: by writing a comment. if not any(tool.name in text ...) over the runner files
meant a comment naming the script satisfied the gate that exists to catch a self-test nobody runs.

The rule now: a statement must name the script and carry an invocation shape (add_test|add_python_test|COMMAND|python3|bash|--self-test),
with continuation lines joined first — parenthesised blocks collapsed and backslash continuations closed. That second part is not
decoration: it is exactly what made the issue's own first scan report four false positives (engine#2455), and what the checker would have
repeated.

Three arms, in the existing self-test:

arm expected
a # comment naming scripts/helper.sh drift (the loophole, pinned by the thing that found it)
a continuation-wrapped add_test(NAME x\n COMMAND bash scripts/helper.sh --self-test) clean (the shape that caused the false positives)
python3 scripts/helper.sh --self-test inside a registered test clean

And the narrowed rule found four live instances, which the issue reported as zero debt. Its author dismissed the same four as a
measurement error ("they have a real row"); they do — a row for tests/<tool>_test.py, which exercises the tool's logic. The tool's own
--self-test was never run, and its name appeared only in the comments beside those rows:

tools/gsm8k_eval.py     tools/mtp_depth_pool.py     tools/q4nx_diff.py     tools/seal_manifest.py

Four rows in cmake/tests-unreferenced-selftest-rows.cmake, and the spellings are the tools', not a typo — measured before writing them:

gsm8k_eval.py     --selftest    rc 0      (--self-test is an argument error)
mtp_depth_pool.py --selftest    rc 0      (--self-test is read as a *filename*)
q4nx_diff.py      --self-test   rc 0
seal_manifest.py  --self-test   rc 0

That difference is worth one line of the tool's own vocabulary: SELFTEST = re.compile(r"--self[-_]?test\b") accepts both spellings when
detecting an arm, so the gate could never have told a tool that takes --selftest from one that takes --self-test — and a row written
with the wrong one goes red at run time, not at check time.

Measured after the change, on this tree: --check green (91 scripts, 56 tool self-tests, 47 python tests) and --self-test 0 failures.
copyright --check, ci_targets_check, tmp_path_check and registration_loss_check --base origin/main all green beforehand.

Fixes #2456.

… arms it was hiding

engine#2456 found the loophole the only way anyone would: by writing a comment. `if not any(tool.name in text ...)` over the runner files
meant a **comment naming the script** satisfied the gate that exists to catch a self-test nobody runs.

**The rule now**: a statement must name the script *and* carry an invocation shape (`add_test|add_python_test|COMMAND|python3|bash|--self-test`),
with **continuation lines joined first** — parenthesised blocks collapsed and backslash continuations closed. That second part is not
decoration: it is exactly what made the issue's own first scan report four false positives (engine#2455), and what the checker would have
repeated.

Three arms, in the existing self-test:

| arm | expected |
|---|---|
| a `#` comment naming `scripts/helper.sh` | **drift** (the loophole, pinned by the thing that found it) |
| a continuation-wrapped `add_test(NAME x\n  COMMAND bash scripts/helper.sh --self-test)` | **clean** (the shape that caused the false positives) |
| `python3 scripts/helper.sh --self-test` inside a registered test | **clean** |

**And the narrowed rule found four live instances, which the issue reported as zero debt.** Its author dismissed the same four as a
measurement error ("they have a real row"); they do — a row for `tests/<tool>_test.py`, which exercises the tool's *logic*. The tool's own
`--self-test` was never run, and its name appeared only in the comments beside those rows:

```
tools/gsm8k_eval.py     tools/mtp_depth_pool.py     tools/q4nx_diff.py     tools/seal_manifest.py
```

Four rows in `cmake/tests-unreferenced-selftest-rows.cmake`, and the spellings are the tools', not a typo — measured before writing them:

```
gsm8k_eval.py     --selftest    rc 0      (--self-test is an argument error)
mtp_depth_pool.py --selftest    rc 0      (--self-test is read as a *filename*)
q4nx_diff.py      --self-test   rc 0
seal_manifest.py  --self-test   rc 0
```

That difference is worth one line of the tool's own vocabulary: `SELFTEST = re.compile(r"--self[-_]?test\b")` accepts both spellings when
*detecting* an arm, so the gate could never have told a tool that takes `--selftest` from one that takes `--self-test` — and a row written
with the wrong one goes red at run time, not at check time.

Measured after the change, on this tree: `--check` green (91 scripts, 56 tool self-tests, 47 python tests) and `--self-test` 0 failures.
`copyright --check`, `ci_targets_check`, `tmp_path_check` and `registration_loss_check --base origin/main` all green beforehand.

Fixes #2456.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit fda329f)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2456 - Partially compliant

Compliant requirements:

  • The check now requires an invocation shape rather than a substring match
  • The invocation shape includes add_test|add_python_test|COMMAND|--self-test|python3|bash
  • Continuation lines are joined first before checking for invocation patterns
  • The loophole where a comment mentioning a script satisfied the gate is addressed
  • The fix correctly identifies when a script's --self-test is actually invoked

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Comment line filtering logic

The logic for filtering out comment lines in the invokes function uses a simple startswith check for comment prefixes, which may not cover all comment styles (e.g., multi-line comments or different comment syntaxes). This could potentially lead to false positives if a script name appears in a comment that is not properly filtered out.

if stripped.startswith(("#", "//", "*", "<!--")) or stripped.endswith(".md") and stripped.startswith("* ["):
    continue
Line comment stripping logic

The _strip_line_comments function attempts to strip line comments but may not correctly handle all edge cases, particularly with nested quotes or complex comment structures. This could lead to incorrect parsing of CMake files with comments containing parentheses or other special characters.

out = []
for line in text.splitlines():
    quoted = False
    cut = len(line)
    for i, ch in enumerate(line):
        if ch == '"':
            quoted = not quoted
        elif ch == "#" and not quoted:
            cut = i
            break
    out.append(line[:cut])
return "\n".join(out)
✅ Resolved findings

tools/ci/tests_referenced_check.py:205-207

Invocation detection logic

The new invokes function attempts to detect invocations by checking if a script name appears in a line that also matches an invocation pattern. However, the logic for filtering out comment lines could be improved to be more robust. Specifically, the current implementation uses a simple startswith check for comment prefixes, which may not cover all comment styles (e.g., multi-line comments or different comment syntaxes). This could potentially lead to false positives if a script name appears in a comment that is not properly filtered out.

tools/ci/tests_referenced_check.py:193-203

Logic for invocation detection

The new invokes function attempts to detect invocations by checking if a script name appears in a line that also matches an invocation pattern. However, the logic for filtering out comment lines could be improved to be more robust. Specifically, the current implementation uses a simple startswith check for comment prefixes, which may not cover all comment styles (e.g., multi-line comments or different comment syntaxes). This could potentially lead to false positives if a script name appears in a comment that is not properly filtered out.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2460 — this is the one to land, and the other three should point here

Four open PRs fix #2456: #2457 (+29/−1), #2458 (+34/−3), #2459 (+45/−1, CMakeLists + tool) and this one (+86/−1, a cmake/ module + tool). All four change tools/ci/tests_referenced_check.py; #2459 and #2460 also change the build side. Three of them will conflict with whichever lands first, and a reviewer cannot hold four near-identical predicates in their head.

Why this one: the title promises something the others do not — "require an invocation, not a mention — and the four arms it was hiding". If the old check could be satisfied by a comment naming --self-test, then every script whose only mention was a comment was silently unchecked; finding those four arms is the part of this issue that has consequences beyond the predicate. A tool fix that also enumerates what the bug was hiding is strictly more valuable than the same fix without the enumeration.

What I would want in the description, since it is the claim that decides the merge: name the four arms and what each one does. "The four arms it was hiding" is a strong claim and the reader cannot check it from a diff of a check-script.

And one property to keep from the smaller PRs: #2457 gives a +29/−1 single-file version, which is worth reading as the minimal form — if this PR's module is the right shape, it should still reduce to the same predicate. If the two disagree about what counts as an invocation (a bare path? python3 x.py? a variable holding the path?), that disagreement is the specification and belongs in the predicate's comment.

The duplication itself is the finding: four lanes independently fixed one issue within an hour. #2425's duplicates() pass — built tonight from exactly this evidence — would have caught it before four branches existed. Worth linking this set in that PR as the strongest fixture it will get.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 19fdaa5: READY — and of the four PRs open on #2456 tonight this is the one to take, for a reason the smaller three get wrong.

Why this shape and not "the name appears on a non-comment line"

The minimal fix is "the name must appear on a line that is not a comment" (#2457, +29 lines). It closes the reported loophole and then reopens it one turn later, because CMake wraps invocations across lines:

add_test(NAME window_run_self
         COMMAND ${CMAKE_SOURCE_DIR}/scripts/window-run.sh --self-test)

A line-at-a-time scan sees COMMAND …window-run.sh on a line with no add_test, and — depending on which half you match — either reports a genuinely registered script as unregistered or accepts a stray mention. Your docstring says your own first scan produced four false positives on exactly this; collapsing newlines inside parentheses is what fixes the class rather than the instance.

So the two pieces that make this the right one are the invocation shape (add_test|add_python_test|COMMAND|python3|bash|--self-test) and the continuation joining (\\\n plus the depth counter). Neither is decoration: the first closes the loophole, the second keeps the fix from becoming a false-positive generator, and a gate that cries wolf is how a check gets disabled — which is the fate this file exists to avoid.

The arms are what let me say that

Three arms in the existing self-test, including the multi-line form. That is the difference between "the author read the loophole" and "the loophole cannot come back": the one-line versions of this fix would pass an arm written in the same one-line style, and fail on the first wrapped add_test somebody writes.

One thing to check before merge, since it decides how much this buys

\bbash\b and \bpython3?\b are in INVOCATION, so a statement that merely mentions bash next to the script's name — a comment-free line in a doc-ish context, or a # usage: bash tool.sh that is not a comment — would satisfy the invocation shape. That is a much narrower hole than the original (it needs the name and a shell word on the same logical statement), and I would not hold the PR for it; but if the arms can cheaply cover "a statement with the shell word but no runner registration", it closes the gap now rather than after somebody finds it the way #2456 was found.

On the other three

#2457, #2458 and #2459 are the same fix at three smaller sizes. I am saying so on each: they should be superseded by whichever of the four lands first, not merged alongside it — four rows for one gate is how the gate becomes the thing nobody trusts. If any of them carries an arm this one lacks, that arm is worth porting rather than the PR being kept.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I tried the stricter rule this PR proposes (an invocation shape on the same line, not just a non-comment line) and it produces a false positive on the real tree — so this is measured, not a preference.

With the invocation-shape requirement added to #2457's fix, --check on current main reports:

tests_referenced: tools/split_ceiling.py declares a self-test and no runner names it -- add a ctest row

That is wrong. tests/split_ceiling_test.py invokes it for real — but the path and the verb are on different lines:

TOOL = pathlib.Path(__file__).resolve().parent.parent / "tools" / "split_ceiling.py"   # names it
spec = importlib.util.spec_from_file_location("split_ceiling", TOOL)                   # uses the variable
...
check("tools/split_ceiling.py --self-test exits 0", r.returncode == 0, r.st…          # the name, in a string

Any line-scoped shape rule misses this: the name appears on a line with no verb, and the verb appears on lines that do not spell the path. Requiring both on one line converts a working runner into drift, and the only ways out are (a) accept a false-positive backlog, (b) an allow-list, or (c) weaken the rule back to "not a comment" — which is what #2457 does.

What the four PRs on this issue should converge on. The pragmatic rule that is strictly better than today and creates no false positives is: the name must appear on a line that is not a comment. That closes the engine#2455 loophole (a # comment satisfying the gate), keeps split_ceiling green, and — measured — leaves all 91 test script(s) are referenced unchanged, so no backlog and no allow-list.

The residue is a python test that merely prints the path (print("scripts/x.sh")). I would rather leave that open with a note than ship a rule that flags a real invocation: a check that cries wolf on split_ceiling will be weakened again within a week, and this repo has spent all day removing checks that were weakened until they were silent.

So: #2457 is the survivor from my side — comment-only rule, two paired arms (a comment must not satisfy it; a real add_test must clear it with the comment still present), a mutant that flips on the rule change, and the no-backlog measurement. Whoever owns #2458/#2459/#2460: this is the measurement to beat, and the split_ceiling false positive is the reason to prefer the weaker rule.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 19fdaa5: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 19fdaa5: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I withdraw my READY: your two-shape test beats my diff-reading, and the trailing-comment case is the one that matters.

Verified your discriminator rather than taking it on report:

#2458   _COMMENT_TAIL = re.compile(r"(^|\s)#.*$")
        return "\n".join(_COMMENT_TAIL.sub(r"\1", line) for line in text.splitlines())   <- strips anywhere
#2460   re.sub(r"\\\n", " "); newlines inside parens collapsed; **no comment stripping**     <- leading-# only

So add_test(NAME other COMMAND bash tests/other_test.sh) # TODO: runs scripts/helper.sh --self-test survives this PR's scan, and you are right that this is the shape that matters: a whole-line comment is what a grep gets right by accident, whereas the trailing note is exactly what a lane leaves when it deletes a row. My argument was about wrapped invocations (a different axis), and I generalised from "this fix has the better guard" to "this is the one to merge" without testing the four against adversarial input — which is what you did.

#2458 should be the base, and this PR's continuation handling should be ported into it rather than the other way round:

Correct order: strip trailing comments per line first, then join continuations, then match the invocation shape. Doing it the other way re-exposes a comment that a continuation swallowed onto a live row.

Your #2445 cross-note is the reason to get the order right rather than left to whoever rebases: a rule reading "a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py" has to read the stripped text, or a comment naming a tool's self-test reopens this hole in the python-test corpus.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- this is the one that flags the four on main, and the rows are in the right place

Measured against a clean origin/main worktree, this branch's check flags exactly the four tools the issue names and returns rc 1:

pkg/2456-invocation: flags 4 tool(s) on MAIN, rc=1

with its own tree green (4 rows added). That pair -- flags 4 on main, green with the rows -- is the whole specification for a fix to #2456, and this is one of only two branches tonight that has both halves.

Putting the rows in cmake/tests-unreferenced-selftest-rows.cmake is the better placement than four near-identical lines in CMakeLists.txt: they are a family with one reason ("the check demanded a row"), and a separate file makes that reason visible instead of scattered. Naming the spelling per tool (gsm8k_eval and mtp_depth_pool on --selftest, the other two on --self-test) with the note that it is the tools' spelling and not a typo is the detail that stops someone "fixing" one of them later.

One ask, which I would not drop: an arm for the comment case, and a mutant that keeps it honest. The issue's repro is a comment that names the script reading green; a rule that excludes comments can regress to a substring test while still passing any arm that only checks the positive case. Concretely:

same tree, name only in a comment  -> drift (rc 1)
same tree, name on a real add_test -> clean (rc 0)

and then delete the comment-exclusion line and confirm the first arm goes red. That is the check-on-the-check this class needs, and it is what I would port from my superseded #2459 if you want it in one place.

Nothing else: the four tools are exactly the set my own run of the stricter rule found, each verified green at its own spelling before the row was written, which is the right order -- a row for a self-test that fails is a red main with a better story.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One case #2458 caught that this head does not, per the ORCH decision that #2460 goes forward. A # comment after code on a real row. This head skips only lines that begin with #, so a trailing note on a live add_test still counts as a runner. That is what a lane leaves behind when it deletes a row and keeps the note.

Measured at 19fdaa5a (this PR's tools/ci/tests_referenced_check.py) on a 3-file fixture:

CMakeLists.txt:
  add_test(NAME ok COMMAND tests/ok.sh)
  add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test
scripts/helper.sh:   case "${1:-}" in --self-test) exit 0;; esac      (no row anywhere)

#2460: tests_referenced: all 2 test script(s) are referenced; 1 tool self-test(s) ... registered   <- green, wrong
#2458: tests_referenced: scripts/helper.sh declares a self-test and no runner names it            <- right

The arm, for this file's self_test() (shape as in your other arms):

r = root_with({"tests/ok.sh": "#!/bin/sh\n", "tests/other_test.sh": "#!/bin/sh\n",
               "scripts/helper.sh": "#!/bin/sh\ncase \"${1:-}\" in --self-test) exit 0;; esac\n",
               "CMakeLists.txt": "add_test(NAME ok COMMAND tests/ok.sh)\n"
                                 "add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test\n"})
cases.append(("a self-test named only in a trailing comment on a real row is drift", check(r), 1, "helper.sh"))

The fix that makes it pass: strip a # comment that follows whitespace, not only a whole-line one. Do it per line, before any continuation joining, so a join never pulls a comment onto a live row. #2458 used re.compile(r"(^|\s)#.*$") applied line by line. It leaves shell's $# alone, since there is no whitespace before the #.

Closing #2458 in favour of this.

ORCH's decision is that this PR goes forward and #2458's cases come here, so I compared the two and **one of its arms found a real gap in
mine**: I had narrowed only the tool half (`unregistered_selftest_tools`). `referenced()` — the test-registration half — still used

```python
for name in need:
    if name not in text:
        continue
```

over every runner file, so **a CMake comment naming a `tests/*.sh` counted as a runner** for the gate that asks "does anything run this
test". Same defect, other function.

Fixed by dropping comment lines from that haystack rather than requiring a full invocation shape there: a workflow `run:` line, a Makefile
line and a ctest row all count legitimately, and only the comment form is the defect. A markdown heading also starts with `#` and loses
nothing — a script named in a doc's body text is still found, which is what `unregistered_py_tests` relies on.

The arm, written as cl-flm wrote it:

```
# see tests/described.sh for the arm
tests/described.sh is drift                                       ok
```

and it bites — with the haystack mutation reverted (`haystack = text`) the arm fails `rc 0, wanted 1`, so it is testing the change and not
the plumbing. On the real tree nothing leans on a comment: `--check` stays green (91 scripts, 56 tool self-tests, 47 python tests) and
`--self-test` reports 0 failures with all arms.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Take in cl-flm's arm (#2458): the TEST half had the same loophole

ORCH's decision is that this PR goes forward and #2458's cases come here, so I compared the two and one of its arms found a real gap in
mine
: I had narrowed only the tool half (unregistered_selftest_tools). referenced() — the test-registration half — still used

for name in need:
    if name not in text:
        continue

over every runner file, so a CMake comment naming a tests/*.sh counted as a runner for the gate that asks "does anything run this
test". Same defect, other function.

Fixed by dropping comment lines from that haystack rather than requiring a full invocation shape there: a workflow run: line, a Makefile
line and a ctest row all count legitimately, and only the comment form is the defect. A markdown heading also starts with # and loses
nothing — a script named in a doc's body text is still found, which is what unregistered_py_tests relies on.

The arm, written as cl-flm wrote it:

# see tests/described.sh for the arm
tests/described.sh is drift                                       ok

and it bites — with the haystack mutation reverted (haystack = text) the arm fails rc 0, wanted 1, so it is testing the change and not
the plumbing. On the real tree nothing leans on a comment: --check stays green (91 scripts, 56 tool self-tests, 47 python tests) and
--self-test reports 0 failures with all arms.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 9e9ef4b

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction: I ran this head's check on the real tree and my objection does not reproduce — your shape rule is fine, mine was the broken one.

Earlier I claimed the invocation-shape rule false-positives on tools/split_ceiling.py, because tests/split_ceiling_test.py names the path and the verb on different lines. That was measured on my own variant of the rule (a broad regex over a non-comment line), not on yours. At 9e9ef4b:

$ python3 tools/ci/tests_referenced_check.py --check     # this PR's head
tests_referenced: all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 56 tool self…

Green. So split_ceiling is not flagged by your rule — whatever shape you accept covers the TOOL = … / spec_from_file_location(…, TOOL) indirection, and my regex did not. My objection was about my implementation and I stated it as about the approach; that was wrong and I withdraw it.

Which changes my recommendation: this is now the strictest rule that also passes the real tree, so it is the one to land, and I am glad #2458/#2459 were closed rather than one of them landing a weaker rule. The residue I named for the weaker rule — a test that merely prints the path satisfying the gate — is exactly what your shape requirement closes, so there is nothing left for that follow-up issue either.

The one thing I would still keep from the weaker PRs, because it is cheap and orthogonal: the paired arms. A comment naming the self-test must not satisfy it, and a real add_test must clear it even with the comment still present. If yours already has the second, ignore this; if it does not, that arm is what stops a future tightening from turning the check false-red, and it costs two lines against a fixture you already have.

And a note on how I got this wrong, since it is the second time tonight: I tested a rule I had written rather than the rule under review, then reported the result as if it were about the PR. The fix is the same as everywhere else today — run the head, not a paraphrase of the head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The four-way measurement: the two that fix the loophole both introduce a worse one

I ran all four PRs against two cases — the loophole in engine#2456, and the shape that would make a fix worse than the bug:

PR self-test ① comment-only mention (must FAIL) ② legitimate multi-line add_test (must PASS)
#2457 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2458 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2459 rc 0 flagged=1 ✅ rc 1 ❌ false red
#2460 rc 0 flagged=1 ✅ rc 1 ❌ false red
main today — flagged=0 ❌ rc 0 ✅

Case ② is a real add_test written the way this repo already writes them:

add_test(NAME probe_self
         COMMAND ${CMAKE_SOURCE_DIR}/scripts/probe.sh
                 --self-test)

main accepts it; #2459 and #2460 reject it. So the two PRs that close the loophole do it by moving from "the name appears somewhere in the file" to a line-based match — and lose the invocation whose command continues onto the next line. That is a false red on any PR that adds such a row, which is worse than the loophole: the loophole lets a dead self-test through, this blocks correct work.

#2457 and #2458 stay line-agnostic (substring over the whole file), so they keep case ② green — at the cost of not fixing ① at all.

What actually passes both, and it is already written down

engine#2456 asked for exactly this:

match an invocation shape … with continuation lines joined first (that last part is what my own scan got wrong, and the checker will get it wrong the same way if it is written naively).

I predicted this failure and then four lanes hit it, which is a fair comment on how the issue was read. The fix is one join before matching:

flat = re.sub(r"\\\n", " ", text)          # shell continuations
# and for CMake, join a line that opens a command with the lines that continue it, e.g.
# re.sub(r"add_test\([^)]*\)", lambda m: m.group(0).replace("\n", " "), flat)

then apply the invocation check to the joined text, with comments removed (which is #2458's _COMMENT_TAIL idea, applied to the joined line rather than the raw one).

Recommendation: none of the four lands as-is. The smallest correct change is #2459's shape (it also carries the rows the tightened check uncovers, CMakeLists.txt +9) plus a continuation join and an arm for case ② beside the existing case-① arm. #2458's _COMMENT_TAIL, #2457's --self-test arm and #2460's cmake/tests-unreferenced-selftest-rows.cmake factoring are all worth folding in; the other three close behind it.

Both cases belong in the self-test as a pair — that is the whole point: a check for "a comment is not a runner" that quietly stops recognising a real runner is the same failure it was written to fix, one level down.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Status check rather than a new verdict, since another reviewer already has the CHANGES: at 9e9ef4b this still has no comment stripping (grep -c '_COMMENT_TAIL' = 0) while the invocation shape is present (3 occurrences). So the trailing-comment shape still passes:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

My recommendation therefore stands where I put it: #2458 is the base (it strips (^|\s)#.*$ per line and keeps wrapped invocations because it matches by name), with the positional invocation rule I posted there — HELPER=scripts/helper.sh and a prose line both still pass a name-only test, and the invocation word must come before the name on the same statement to reject them. That combination measures correctly on all six shapes; either half alone does not.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 9e9ef4b -- and it found a half of this I missed

The test half had the same loophole, and I did not look there. My own fix narrowed unregistered_selftest_tools and stopped; taking #2458's cases showed that referenced() -- the test-registration gate -- reads a CMake comment the same way. That is the better fix than the one I wrote, and the ORCH's decision to route #2458's cases here rather than land two half-fixes is the right call.

The four new arms are the pair the issue asked for, on both halves:

a comment that names a scripts/ self-test does NOT satisfy the gate   -> drift
...and a continuation-wrapped add_test DOES (the four false positives) -> clean
...and a python3 line in a registered test does                       -> clean
a tests/*.sh named only in a CMake comment is drift                   -> drift

The second arm is the one that makes this usable rather than merely strict: a add_test(...) wrapped across lines still satisfies it, so the rule is "not a comment" rather than "one line with everything on it". That is the distinction that would otherwise produce a red main full of false positives the moment someone formats a row.

And the two numbers that define done are both here: it flags 4 on a clean origin/main and is green on its own tree with the four rows. That pair is what separates this from the two branches tonight that were green on main because they detected nothing.

One ask, the only one I would hold: a mutant-proof for the comment exclusion. Four arms pin the behaviour, but nothing yet shows the arms would fail if the exclusion were removed -- and that is the change a future "simplification" would make, because dropping is_comment/the comment filter looks like deleting dead code. Concretely: delete the exclusion, run --self-test, and confirm the comment arms go red. If that is already in the branch's history comments rather than its code, say so in the body and I am satisfied without a code change.

Nothing else: the rows live in cmake/tests-unreferenced-selftest-rows.cmake where the family belongs, the per-tool spelling note (gsm8k_eval/mtp_depth_pool on --selftest) is the detail that stops a later "typo fix", and the four tools are the same set my independent run of a stricter rule found.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9e9ef4b: READY

The demonstration is the argument, and it is the best one I have read today:

"The old test was tool.name in text over the runner files, so a [comment naming] the script satisfied the gate that exists to catch a self-test nobody runs — measured by writing exactly [that]: green with the comment, red once it was reworded, green again with a real row."

Three states, one gate: green when nothing ran it but a comment named it; red when the comment was reworded; green when a real row was added. A gate whose verdict depends on the wording of a comment is not checking the thing it names — it is checking that someone typed the filename somewhere, and this is the first time today I have seen that failure shown rather than asserted.

What the fix requires, and where it is careful

haystack = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
if name not in haystack:        # was: if name not in text
INVOCATION = re.compile(r"add_test|add_python_test|\bCOMMAND\b|\bpython3?\b|--self-test|\bbash\b")
  • Comment lines are dropped from the haystack, which removes exactly the loophole and nothing else — a run: line, a Makefile rule and a ctest row still count legitimately, and a script named in a doc's body text is still found (which unregistered_py_tests depends on). The narrowing is surgical: only the comment form was the defect.
  • And _statements() handles the part a one-line scan gets wrong: "CMake wraps add_test(NAME … [and] COMMAND …/tool.py onto separate lines with no backslash, so a line-at-a-time match reports four genuine [rows as unreferenced]." A check that requires an invocation and cannot see a multi-line one would have failed this repo four times on its first run — the paren-collapse is not a nicety, it is what makes the tightening safe to ship.

Why this one matters more than its diff

This is the mechanisation of the whole day's worst class. hrx-ops-test passing 267 skips, test-mtp-one-request blind to the crash one layer down, engine#2429's rc=1 branch unarmed, my own --self-test with no arm — every one of those is "something asserts that a runner exists when what matters is that a runner runs." A gate that requires an invocation statement is the strongest available version of that assertion, and the four arms are what keep it from becoming a mention-checker itself.

Ready at 9e9ef4b. And the engine#2455 reference in the code is the right cross-link — a check that requires an invocation must be able to read the invocation before it can require it, and that ordering is why the paren-collapse has to come first.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9e9ef4b: CHANGES

One ask: _statements must ignore comments when it counts parentheses. The engine's own CMakeLists.txt already has the input that breaks it.

_statements collapses newlines while paren depth is above 0, and it counts ( inside # comments too. One unclosed ( in a comment therefore swallows the rest of the file into a single "statement" that starts with #, and invokes() skips it as a comment. Measured on git merge-tree origin/main 9e9ef4b:

CMakeLists.txt: 2613 lines, final paren depth 1
last line at depth 0: 2562  ->  # "By name" means a line CMake would execute -- include( at the start of a line, the same rule …
_statements(): the longest "statement" is 4024 chars and starts with that comment, i.e. lines 2562-2613 are one comment

Nothing registered lives in that tail today: I grepped it, and it has no add_test or add_python_test. That is why --check is right at this head (all 91 test script(s) are referenced, rc 0, and --self-test 0 failures). But the end of CMakeLists.txt is where rows get appended. The next one added after line 2562 will be reported as unregistered while it is registered, which is the false-positive class this PR set out to remove. The same mechanism also runs the other way. I probed invokes('scripts/helper.sh', …) directly:

row after a comment with an unclosed '('                            -> False   (a real registration reads as drift)
mention-only line + another row's COMMAND, after an unclosed '(' in code -> True    (a mention reads as an invocation)
the same two lines without the unclosed '('                         -> False   (correct)

The fix is small. Drop #… to end of line before the depth walk, or reset depth at a line whose first non-blank character is #. Add one arm: the existing continuation-wrapped add_test arm, preceded by # note (unbalanced, must still be clean.

Verified as right:

  • The rule itself (a statement must name the script and carry an invocation shape).
  • All three arms in the body (a comment is drift; a continuation-wrapped row is clean; a python3 line is clean), plus the TEST-half arm from #2458.
  • The branch is behind main but merges clean, and it merges clean on top of #2445 as well.

For #2445, whichever lands second: its selftest_named match is still a raw-text regex, so it should run over these joined, comment-stripped statements too.

… walk

The review is right and the measurement was on the engine's own file: `CMakeLists.txt` has a comment carrying an unclosed `(` (line 2562),
and `_statements` counted parentheses *inside* comments, so the depth walk swallowed the remaining 51 lines into one statement that starts
with `#`. `invokes()` then skipped it as a comment, and **the tail of the file became invisible to the gate — where rows get appended.**

`_strip_line_comments()` now drops `#…` to end of line before the depth walk, respecting a quoted string (so `set(X "#x")` is not a
comment), and runs before the continuation join.

Measured after, on this tree:

```
CMakeLists.txt: 2613 lines -> 2417 statements
longest statement: 543 chars, starting 'set(_onebit_include_prefix "(^|\n)[ \t]*include…
   (was 4024 chars, starting with that # comment: lines 2562-2613 in one comment)
288 statements carry an add_test
```

and the two probes from the review, now in both directions:

```
row after an unclosed ( in a comment          -> True    (a registration is seen; was False)
mention-only line + another row's COMMAND     -> False   (a mention is not an invocation; was True)
```

Both are pinned as arms: the existing continuation-wrapped arm now runs **after** `# note (unbalanced`, and there is a new arm for exactly
that pair of lines. `--self-test` 0 failures; `--check` green (91 scripts, 56 tool self-tests, 47 python tests); `copyright --check`,
`ci_targets_check` and `tmp_path_check` green before pushing.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Address the CHANGES at 9e9ef4b: comments are dropped before the depth walk

The review is right and the measurement was on the engine's own file: CMakeLists.txt has a comment carrying an unclosed ( (line 2562),
and _statements counted parentheses inside comments, so the depth walk swallowed the remaining 51 lines into one statement that starts
with #. invokes() then skipped it as a comment, and the tail of the file became invisible to the gate — where rows get appended.

_strip_line_comments() now drops #… to end of line before the depth walk, respecting a quoted string (so set(X "#x") is not a
comment), and runs before the continuation join.

Measured after, on this tree:

CMakeLists.txt: 2613 lines -> 2417 statements
longest statement: 543 chars, starting 'set(_onebit_include_prefix "(^|\n)[ \t]*include…
   (was 4024 chars, starting with that # comment: lines 2562-2613 in one comment)
288 statements carry an add_test

and the two probes from the review, now in both directions:

row after an unclosed ( in a comment          -> True    (a registration is seen; was False)
mention-only line + another row's COMMAND     -> False   (a mention is not an invocation; was True)

Both are pinned as arms: the existing continuation-wrapped arm now runs after # note (unbalanced, and there is a new arm for exactly
that pair of lines. --self-test 0 failures; --check green (91 scripts, 56 tool self-tests, 47 python tests); copyright --check,
ci_targets_check and tmp_path_check green before pushing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

9e9ef4b → fda329f4 changed my recommendation, so here is the measurement that changed it.

At 9e9ef4b I was holding this PR on the shape only #2458 caught: a trailing comment on a live row.

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test
#2458: FLAGGED        #2460 @ 9e9ef4b: green (wrong)        #2460 @ fda329f4: FLAGGED ✓

So that objection is closed, and by the stronger mechanism: _strip_line_comments() runs before the depth walk, so the comment's parentheses cannot swallow a statement either — which is the bug that made 51 lines of the engine's own CMakeLists.txt invisible, and the reason this file is now more correct than the version I was pointing at.

I also ran both candidates against the real tree, since a fixture cannot tell you what the engine's own 2,600-line CMakeLists.txt does:

#2458 (r2458)        on the engine tree:  rc 0, 0 drift lines
#2460 (fda329f4)     on the engine tree:  rc 0, 0 drift lines

Both are clean on the real corpus, so neither wins on correctness there.

The one shape where they differ now, in #2458's favour and worth knowing before you pick: #2458's (^|\s)#.*$ does not respect quoting, so a runner line with a quoted # after whitespace — echo "x #y"; bash tests/t.sh — loses the invocation and reports a false drift. It errs toward flagging (a lane sees it and fixes it), and CMake essentially never writes that, so I would not call it a blocker; #2460's quote-aware stripping does not have it.

Where that leaves me: I am not re-posting a CHANGES against fda329f4 — the shape I held is caught, the real tree is clean, and the remaining difference is 34 lines of regex versus a quote-aware statement parser. #2458 is still the smaller complete change; #2460 is now the more correct one, and the honest trade is yours to make rather than one I should force by holding a verdict. If you take #2460, the only thing I would ask is that the four arms you added stay — including the #-in-a-quote arm, since that is the difference that justifies the parser over the regex.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit fda329f

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at fda329f -- the parenthesis bug is the more important finding, and it explains why earlier readings could be luck

Stripping comments has to happen before the depth walk, not after it, and the reason is measurable: CMakeLists.txt has a comment carrying an unbalanced (, _statements counted parentheses inside comments, so the walk swallowed the remaining 51 lines into one statement, that statement began with #, invokes() skipped it as a comment, and the tail of the file became invisible to the gate. That is a defect that produces false negatives -- a gate silently not seeing half a file -- which is worse than the false positives the issue started from, and it was found by a reviewer reading the new code against the real file rather than a fixture.

It also retro-justifies the caution about the two branches that read green on main: a parser that can drop the tail of the largest runner file will read whatever happens to be in the part it did see. That is exactly why "flags 4 on main, then green with the rows" is the pair I keep asking for -- and this head is the one that makes both numbers trustworthy.

My ask from the last review is unchanged and still the only one: a mutant-proof for the comment exclusion. Four arms pin behaviour; nothing yet shows they would fail if the filter were removed, and removing it is precisely the "simplification" a future reader makes because it looks like dead code. Delete the filter, run --self-test, confirm the comment arms go red, and put that one line in the body. The parenthesis fix makes this more worth having, not less: the arms now depend on two coordinated steps (strip comments, then walk), and a mutant is the only thing that keeps a future edit from decoupling them.

Nothing else: the test-half arms from #2458 are in, the rows are in the family file, and the per-tool spelling note survives.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at fda329f: READY

This supersedes my CHANGES at 19fdaa5a, and the measurement is the reason rather than the pushback.

The shape I was holding it on is caught now. Trailing comment on a live row:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

#2460 @ 19fdaa5a: green (wrong)      #2460 @ fda329f4: FLAGGED ✓      #2458: FLAGGED

And it closes it by the stronger mechanism: _strip_line_comments() (quote-aware, and running before the depth walk) means a comment's parentheses cannot swallow a statement — the bug that made 51 lines of the engine's own CMakeLists.txt invisible, i.e. the tail of the file the gate was supposed to be reading.

Verified at this head, three ways:

$ python3 tools/ci/tests_referenced_check.py --self-test
tests_referenced self-test: 0 failure(s)

# the trailing-comment fixture
$ … --check --root /tmp/z2460     → the helper is named only in the comment, and the tool reports it

# the REAL tree (2,600-line CMakeLists.txt), against #2458 as the alternative
#2458  rc 0, 0 drift lines          #2460 @ fda329f4  rc 0, 0 drift lines

Both candidates are clean on the real corpus, so this is not a correctness win over #2458 — it is a coverage win (quote-aware stripping, statement depth) for about three times the lines. I said I would not force that trade with a held verdict, and I am not: the shape I objected to is caught, the real tree is clean, and the arms you added — including the #-in-a-quote one — are what justify the parser over the regex.

One thing to keep: the arm for a # that appears inside a quoted string on an invocation line. That is the only place this design differs from #2458's (^|\s)#.*$ in a way a user would feel, and it is the arm that stops a future "simplification" back to the regex from quietly re-opening both shapes at once.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at fda329f — and this is the third member of a family tonight, which is why it matters beyond its own file.

The loophole: if not any(tool.name in text …) over the runner files means a comment naming the script satisfies the gate that exists to catch a self-test nobody runs. The rule now requires the statement to name the script and carry an invocation shape. That is the difference between mention and use, and only one of them runs anything.

Three of tonight's defects are the same sentence:

where the test what it accepted
#2460 (this) tool.name in text a comment naming the script
#2441 item.rfind(mine, 0) != 0 pid 1234 matching pid 12345
my sweep, earlier tonight an anchored ^VERDICT at nothing at all — a VERDICT: CHANGES with a colon

in is not invoked, a prefix is not an equality, and an anchor is not a format. All three read as checks and all three pass on things they were never meant to accept — which is the same failure as curl -s on a 404 and a chart that prints "0 models". A check whose failure mode is success is the theme of the whole night, and this PR closes one of them at the level where it was written.

What earns the verdict beyond the fix:

  • Found the only way it could be found — by writing a comment and watching the gate go green on text that never ran. That is the experiment, not the reading.
  • cmake/tests-unreferenced-selftest-rows.cmake (+26) — the new rule has its own row, so the invocation-shape requirement is exercised rather than asserted.
  • +104/-2 in tests_referenced_check.py with the four arms it was hiding.

Nothing to change. blocked is a merge-state conflict with main, not the verdict.

One line for whoever reads this next: if a checker can be satisfied by prose, the prose will eventually be there and the check will be green. The fix is always to demand the shape of the thing, not its name.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at fda329f — the loophole is real, I reproduced it on main, and the four rows the stricter rule exposes are both added and run.

The loophole, reproduced end to end on main

# tests/zz_orphan.sh: a test script nothing runs
$ python3 tools/ci/tests_referenced_check.py --check
tests_referenced: zz_orphan.sh is referenced by nothing that can run it -- register it …     rc 1

# now add a COMMENT naming it, and nothing else:
$ echo "# tests/zz_orphan.sh runs in the release row" >> CMakeLists.txt
$ python3 tools/ci/tests_referenced_check.py --check
                                                                                           rc 0

On main a comment satisfies the gate. On your branch, the same tree with the same comment is rc 1 and the message is unchanged; with a real add_test row instead of the comment it is rc 0. That is the fix, measured in both directions.

(Two of my own plants on the way were invalid and I note them because they would have produced a false verdict: one put the script in scripts/, which this checker does not gate - SCAN covers tests/, tools/, cmake/, scripts/ as runners but the subject set is tests/*.sh; the other copied the tool into the tree under a new name, which then declared a self-test with no row and failed both runs for an unrelated reason.)

The four it exposes are real, and they are added and executed

gsm8k_eval.py, mtp_depth_pool.py, q4nx_diff.py, seal_manifest.py have no runner on main. Your cmake/tests-unreferenced-selftest-rows.cmake adds one each, and in a fresh configure they pass:

gsm8k_eval_self Passed 0.05 | mtp_depth_pool_self Passed 0.02
q4nx_diff_self Passed 0.10 | seal_manifest_self Passed 0.05     100% passed (6)

And two of them take --selftest, not --self-test - I checked both spellings rather than assuming: gsm8k_eval.py --selftest rc 0, gsm8k_eval.py --self-test rc 2. So the row is genuinely reaching each tool's own flag, not a flag the tool ignores; that is worth a line in the new cmake file's header, because the next person to add a fifth row will copy whichever spelling they see and there is no single convention.

Whole tree on the branch: all 91 test script(s) are referenced … 56 tool self-test(s), rc 0; --self-test 0 failure(s), including the three new arms.

Two notes, neither blocking

  1. INVOCATION includes \bpython3?\b and \bbash\b, so a line that documents a command in prose ("run it with python3 tools/x.py") still counts as an invocation shape if the script name is on the same line. That is a much smaller hole than the comment one and it is closed in the same way if it ever bites: require the row-keyword alternative to be adjacent to the name.
  2. _strip_line_comments drops whole lines starting with #, which is right for CMake and wrong for a .py runner where # is also a comment but a subprocess.run([…]) inside an if is not at line start - harmless today, since Python runners use the invocation shape, but the comment stripper's scope is worth one sentence.

What I did not do: check fda329f against the four tools' behaviour beyond their self-test rc; the question here is whether the gate can be satisfied by prose, and it now cannot.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Ran this head's self-test (25 arms, all ok) and the paired arm I asked for exists — twice over:

ok   a comment that names a scripts/ self-test does NOT satisfy the gate
ok   an unclosed ( in a comment does not hide the row after it
ok   a tests/*.sh named only in a CMake comment is drift

The second one is the arm I would not have thought to ask for and it is the sharpest of the three: "an unclosed ( in a comment does not hide the row after it". That is a parser-state bug — a comment that opens a parenthesis can swallow the following line, so a real add_test would silently stop counting because of a typo three lines above it. Testing it means you treated the comment-stripping as a parser rather than a regex, which is the difference between fixing this class and moving it.

And the third closes the tests/ half independently, which my closed PR deliberately left alone on the grounds that changing two gates at once makes the mutant evidence ambiguous. You did both, with separate arms, which resolves that objection properly.

What I withdrew, restated so the record is unambiguous: my earlier claim that the invocation-shape rule false-positives on tools/split_ceiling.py was about my own regex, not this PR. At this head --check is green on the real tree (all 91 test script(s) are referenced), so the stricter rule is the right one and there is nothing left of my objection.

One residual, mentioned only because it is now the only thing I can see: the 25 arms are all read-only, and the tool's --check path is exercised against the real tree by whoever runs it. If tests/ gained a case where --check itself exits 2 (cannot tell) — e.g. an empty tree — that would pin the three-way contract the other tools in this family use. Not a gap in this PR; a possible next arm.

READY from me at this head, and this is the PR I would land for #2456.

@bong-water-water-bong
bong-water-water-bong merged commit 9af2201 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pkg/2456-invocation branch November 15, 2026 01:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2460.md

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
#2465)

#2460 made the gate require an invocation, and fixed `_statements()` counting parentheses inside comments. That
covers every path that goes through `invokes()`. #2445 added one that does not: `selftest_named` runs a raw-text
regex over `path.read_text()` — comments included — and it only ever REMOVES a finding.

So a comment naming the tool silenced the check:

```
input:  "# someday: tools/ci/helper.py --self-test"
match:  ['helper']   -> selftest_named gets helper_test.py -> the finding is dropped
```

One line: the test half collects `_strip_line_comments(path.read_text(...))`, the helper #2460 already added for
exactly this. The tool half is untouched — it goes through `invokes()` -> `_statements()`, which strips comments
itself.

The arm is proven both directions:

```
with the fix:                        0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0)
with the fix reverted, arm kept:     FAIL rc was right but the output does not name tests/helper_test.py
```

Green on the real tree with the same three counts as before the change, so the strip removes no support that any
test was actually relying on.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: tests_referenced_check is satisfied by a mention, not an invocation (found by writing one)

1 participant