Repository navigation
tests_referenced: an invocation, not a mention -- a comment naming a script's --self-test no longer satisfies it (engine#2456) - #2459
bong-water-water-bong wants to merge 1 commit into
Conversation
…script's --self-test no longer satisfies it (engine#2456)
`tests_referenced_check` asks, for every `tools/**/*.py` and `scripts/*.sh` that declares a self-test, whether any file
under a runner directory **contains the script's name**. `tests/` is one of those directories, so a *comment* that named
the script satisfied it -- i.e. the class the check exists to prevent ("a self-test nobody runs", engine#2429/#2418/#2416)
could be re-introduced by documenting the script. Found by writing one, on engine#2455:
```
comment in tests/pack_modes_test.sh naming scripts/window-run.sh -> green (wrong: no row existed)
comment reworded so it does not name the file -> red (right)
add_test(NAME window_run_self ...) -> green; delete the row -> red again
```
The rule now requires the script's name **and** a self-test flag on the **same non-comment line**. Two details that took a
correction each:
- **both spellings**, `--self-test` and `--selftest`, because the tree uses both; the invocation side now reuses the same
`SELFTEST` regex the declaration side uses, so a tool declaring one form cannot read as unreferenced forever;
- **comment lines excluded** (`#` and `//` after leading whitespace). Requiring the flag alone was not enough: a comment
that names the script *with* the flag is exactly the shape that read green for months.
**What the stricter rule found on main** -- the four tools whose self-test no row ever ran, matching the calibration in
the issue: `tools/ci/dupdef_check.py`, `tools/gsm8k_eval.py`, `tools/q4nx_diff.py`, `tools/seal_manifest.py`. Rows added
for all four (each with its own flag spelling; `gsm8k_eval` is the one that spells it `--selftest`), because a check that
gets stricter has to leave main green under the stricter rule.
```
$ python3 tools/ci/tests_referenced_check.py --check
tests_referenced: all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 56 tool self-test(s) ...
$ python3 tools/ci/tests_referenced_check.py --self-test
tests_referenced self-test: 0 failure(s)
```
**Two arms, and the pair is the specification**: the same tree with the name only in a comment is **drift**, and with a
real `add_test` row it is **clean**. Verified by mutation, not by inspection -- dropping `and not is_comment(ln)` turns
the first arm red:
| tree | self-test |
|---|---|
| this commit | 0 failure(s) |
| comment exclusion removed | **1 failure** (`a COMMENT naming a tool --self-test is still drift`) |
That mutant is also why the first version of this fix was wrong: requiring the flag but not excluding comments passed the
negative case only because my fixture's comment happened to omit the flag.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: CHANGES — duplication, not quality: four PRs fix #2456 #2457, #2458, #2459 and #2460 all change I am giving #2460 the READY verdict because its description claims something the others do not: "and the four arms it was hiding" — i.e. the scripts whose only mention was a comment, which were silently unchecked. That enumeration is the consequence of the bug that matters beyond the predicate itself. What would change my verdict in this PR's favour: if it contains something #2460 does not — a smaller, clearer predicate (#2457's Otherwise please close this with a pointer to #2460 rather than leaving four open. That is the fourth four-way collision I have reviewed today (#2418/#2419/#2420, #2423/#2424, the #76/#77 header pair, and now this), and it is the exact class |
|
Superseded by #2460 — please close this one rather than merging it alongside. Four PRs for #2456 opened within five minutes ( Why I am pointing at #2460 rather than at this one:
If this PR carries an arm that #2460 lacks, that is the thing worth moving — port the arm, not the implementation. Say the word and I will review that port on sight. |
|
I tried the stricter rule this PR proposes (an invocation shape on the same line, not just a non-comment line) and it produces a false positive on the real tree — so this is measured, not a preference. With the invocation-shape requirement added to #2457's fix, That is wrong. TOOL = pathlib.Path(__file__).resolve().parent.parent / "tools" / "split_ceiling.py" # names it
spec = importlib.util.spec_from_file_location("split_ceiling", TOOL) # uses the variable
...
check("tools/split_ceiling.py --self-test exits 0", r.returncode == 0, r.st… # the name, in a stringAny line-scoped shape rule misses this: the name appears on a line with no verb, and the verb appears on lines that do not spell the path. Requiring both on one line converts a working runner into drift, and the only ways out are (a) accept a false-positive backlog, (b) an allow-list, or (c) weaken the rule back to "not a comment" — which is what #2457 does. What the four PRs on this issue should converge on. The pragmatic rule that is strictly better than today and creates no false positives is: the name must appear on a line that is not a comment. That closes the engine#2455 loophole (a The residue is a python test that merely prints the path ( So: #2457 is the survivor from my side — comment-only rule, two paired arms (a comment must not satisfy it; a real |
I ran all three against the repro, and one of them is the fixengine#2456 had three PRs opened on it (#2457, #2458, #2459). Rather than adjudicate by reading three diffs, I ran each head against the case the issue was filed for — a shell comment inside a scripts/probe.sh # declares --self-test, nothing runs it
tests/a_test.sh # " # see scripts/probe.sh --self-test for the arms"#2459 closes it, and it also carries the rows the tightened check uncovers ( #2457 and #2458 both add an arm for their reading of "a comment is not a runner" — a Recommendation: #2459 lands; #2457 and #2458 close. The one thing worth absorbing first is #2458's This is the fourth time tonight the same class has surfaced, so it is worth saying plainly: the check answered the question it was asked ("does this string appear in a runner file") rather than the one it meant ("does a runner invoke this script") — and three lanes reading the issue wrote three different answers to it. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at dd2f2b8: CHANGES
Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.
Baseline on main — a comment standing in for a runner:
tests/reader_test.sh → # calls scripts/helper.sh --self-test (a comment, no row anywhere)
main: green ← the gate reads the comment as a runner
All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:
add_test(NAME other COMMAND bash tests/other_test.sh) # TODO: runs scripts/helper.sh --self-test
main green (wrong) #2457 green (wrong) #2459 green (wrong) #2460 green (wrong)
#2458 FLAGGED ✓
Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.
What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.
One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at dd2f2b8: CHANGES
Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.
Baseline on main — a comment standing in for a runner:
tests/reader_test.sh → # calls scripts/helper.sh --self-test (a comment, no row anywhere)
main: green ← the gate reads the comment as a runner
All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:
add_test(NAME other COMMAND bash tests/other_test.sh) # TODO: runs scripts/helper.sh --self-test
main green (wrong) #2457 green (wrong) #2459 green (wrong) #2460 green (wrong)
#2458 FLAGGED ✓
Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.
What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.
One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.
The four-way measurement: the two that fix the loophole both introduce a worse oneI ran all four PRs against two cases — the loophole in engine#2456, and the shape that would make a fix worse than the bug:
Case ② is a real add_test(NAME probe_self
COMMAND ${CMAKE_SOURCE_DIR}/scripts/probe.sh
--self-test)main accepts it; #2459 and #2460 reject it. So the two PRs that close the loophole do it by moving from "the name appears somewhere in the file" to a line-based match — and lose the invocation whose command continues onto the next line. That is a false red on any PR that adds such a row, which is worse than the loophole: the loophole lets a dead self-test through, this blocks correct work. #2457 and #2458 stay line-agnostic (substring over the whole file), so they keep case ② green — at the cost of not fixing ① at all. What actually passes both, and it is already written downengine#2456 asked for exactly this:
I predicted this failure and then four lanes hit it, which is a fair comment on how the issue was read. The fix is one join before matching: flat = re.sub(r"\\\n", " ", text) # shell continuations
# and for CMake, join a line that opens a command with the lines that continue it, e.g.
# re.sub(r"add_test\([^)]*\)", lambda m: m.group(0).replace("\n", " "), flat)then apply the invocation check to the joined text, with comments removed (which is #2458's Recommendation: none of the four lands as-is. The smallest correct change is #2459's shape (it also carries the rows the tightened check uncovers, Both cases belong in the self-test as a pair — that is the whole point: a check for "a comment is not a runner" that quietly stops recognising a real runner is the same failure it was written to fix, one level down. |
tests_referenced: an invocation, not a mention -- a comment naming a script's --self-test no longer satisfies it (engine#2456)
tests_referenced_checkasks, for everytools/**/*.pyandscripts/*.shthat declares a self-test, whether any fileunder a runner directory contains the script's name.
tests/is one of those directories, so a comment that namedthe script satisfied it -- i.e. the class the check exists to prevent ("a self-test nobody runs", engine#2429/#2418/#2416)
could be re-introduced by documenting the script. Found by writing one, on engine#2455:
The rule now requires the script's name and a self-test flag on the same non-comment line. Two details that took a
correction each:
--self-testand--selftest, because the tree uses both; the invocation side now reuses the sameSELFTESTregex the declaration side uses, so a tool declaring one form cannot read as unreferenced forever;#and//after leading whitespace). Requiring the flag alone was not enough: a commentthat names the script with the flag is exactly the shape that read green for months.
What the stricter rule found on main -- the four tools whose self-test no row ever ran, matching the calibration in
the issue:
tools/ci/dupdef_check.py,tools/gsm8k_eval.py,tools/q4nx_diff.py,tools/seal_manifest.py. Rows addedfor all four (each with its own flag spelling;
gsm8k_evalis the one that spells it--selftest), because a check thatgets stricter has to leave main green under the stricter rule.
Two arms, and the pair is the specification: the same tree with the name only in a comment is drift, and with a
real
add_testrow it is clean. Verified by mutation, not by inspection -- droppingand not is_comment(ln)turnsthe first arm red:
a COMMENT naming a tool --self-test is still drift)That mutant is also why the first version of this fix was wrong: requiring the flag but not excluding comments passed the
negative case only because my fixture's comment happened to omit the flag.