Skip to content

tests_referenced: an invocation, not a mention -- a comment naming a script's --self-test no longer satisfies it (engine#2456) - #2459

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
pi-i679/refcheck-invoke
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
pi-i679/refcheck-invoke

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

tests_referenced: an invocation, not a mention -- a comment naming a script's --self-test no longer satisfies it (engine#2456)

tests_referenced_check asks, for every tools/**/*.py and scripts/*.sh that declares a self-test, whether any file
under a runner directory contains the script's name. tests/ is one of those directories, so a comment that named
the script satisfied it -- i.e. the class the check exists to prevent ("a self-test nobody runs", engine#2429/#2418/#2416)
could be re-introduced by documenting the script. Found by writing one, on engine#2455:

comment in tests/pack_modes_test.sh naming scripts/window-run.sh   -> green (wrong: no row existed)
comment reworded so it does not name the file                       -> red   (right)
add_test(NAME window_run_self ...)                                  -> green; delete the row -> red again

The rule now requires the script's name and a self-test flag on the same non-comment line. Two details that took a
correction each:

  • both spellings, --self-test and --selftest, because the tree uses both; the invocation side now reuses the same
    SELFTEST regex the declaration side uses, so a tool declaring one form cannot read as unreferenced forever;
  • comment lines excluded (# and // after leading whitespace). Requiring the flag alone was not enough: a comment
    that names the script with the flag is exactly the shape that read green for months.

What the stricter rule found on main -- the four tools whose self-test no row ever ran, matching the calibration in
the issue: tools/ci/dupdef_check.py, tools/gsm8k_eval.py, tools/q4nx_diff.py, tools/seal_manifest.py. Rows added
for all four (each with its own flag spelling; gsm8k_eval is the one that spells it --selftest), because a check that
gets stricter has to leave main green under the stricter rule.

$ python3 tools/ci/tests_referenced_check.py --check
tests_referenced: all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 56 tool self-test(s) ...

$ python3 tools/ci/tests_referenced_check.py --self-test
tests_referenced self-test: 0 failure(s)

Two arms, and the pair is the specification: the same tree with the name only in a comment is drift, and with a
real add_test row it is clean. Verified by mutation, not by inspection -- dropping and not is_comment(ln) turns
the first arm red:

tree self-test
this commit 0 failure(s)
comment exclusion removed 1 failure (a COMMENT naming a tool --self-test is still drift)

That mutant is also why the first version of this fix was wrong: requiring the flag but not excluding comments passed the
negative case only because my fixture's comment happened to omit the flag.

…script's --self-test no longer satisfies it (engine#2456)

`tests_referenced_check` asks, for every `tools/**/*.py` and `scripts/*.sh` that declares a self-test, whether any file
under a runner directory **contains the script's name**. `tests/` is one of those directories, so a *comment* that named
the script satisfied it -- i.e. the class the check exists to prevent ("a self-test nobody runs", engine#2429/#2418/#2416)
could be re-introduced by documenting the script. Found by writing one, on engine#2455:

```
comment in tests/pack_modes_test.sh naming scripts/window-run.sh   -> green (wrong: no row existed)
comment reworded so it does not name the file                       -> red   (right)
add_test(NAME window_run_self ...)                                  -> green; delete the row -> red again
```

The rule now requires the script's name **and** a self-test flag on the **same non-comment line**. Two details that took a
correction each:

- **both spellings**, `--self-test` and `--selftest`, because the tree uses both; the invocation side now reuses the same
  `SELFTEST` regex the declaration side uses, so a tool declaring one form cannot read as unreferenced forever;
- **comment lines excluded** (`#` and `//` after leading whitespace). Requiring the flag alone was not enough: a comment
  that names the script *with* the flag is exactly the shape that read green for months.

**What the stricter rule found on main** -- the four tools whose self-test no row ever ran, matching the calibration in
the issue: `tools/ci/dupdef_check.py`, `tools/gsm8k_eval.py`, `tools/q4nx_diff.py`, `tools/seal_manifest.py`. Rows added
for all four (each with its own flag spelling; `gsm8k_eval` is the one that spells it `--selftest`), because a check that
gets stricter has to leave main green under the stricter rule.

```
$ python3 tools/ci/tests_referenced_check.py --check
tests_referenced: all 91 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh); 56 tool self-test(s) ...

$ python3 tools/ci/tests_referenced_check.py --self-test
tests_referenced self-test: 0 failure(s)
```

**Two arms, and the pair is the specification**: the same tree with the name only in a comment is **drift**, and with a
real `add_test` row it is **clean**. Verified by mutation, not by inspection -- dropping `and not is_comment(ln)` turns
the first arm red:

| tree | self-test |
|---|---|
| this commit | 0 failure(s) |
| comment exclusion removed | **1 failure** (`a COMMENT naming a tool --self-test is still drift`) |

That mutant is also why the first version of this fix was wrong: requiring the flag but not excluding comments passed the
negative case only because my fixture's comment happened to omit the flag.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2416 - Partially compliant

Compliant requirements:

  • The CI now includes a new test hip_f16_audit_pinned that will invoke the audit when the gpu-kernels tree is checked out
  • The audit is gated to run only on the correct pinned commit via --expect-sha
  • The CI skips the audit with a clear message when the gpu-kernels tree is not checked out

Non-compliant requirements:

  • The PR does not include the actual CI job or hook that invokes the audit (this is noted as "the open question" in the ticket)

Requires further human verification:

  • Verification that the CI job actually runs the audit when the pinned gpu-kernels tree is checked out
  • Verification that the audit correctly gates on the pinned commit
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Stricter Self-test Reference Check

The updated tests_referenced_check.py now requires that a self-test is not only mentioned but actually invoked on the same line, excluding comment lines. This prevents a regression where a comment in a test file naming a script with --self-test would incorrectly satisfy the check, which was the root cause of the "a self-test nobody runs" issue.

# So collect LINES, not blobs, and count only a line that names the script AND passes --self-test.
runner_text = []
for sub in TOOL_RUNNERS:
    p = root / sub
    files = [p] if p.is_file() else (sorted(q for q in p.rglob("*") if q.is_file()) if p.is_dir() else [])
    for f in files:
        try:
            runner_text.append(f.read_text(errors="replace"))
        except OSError:
            pass
runner_lines = [ln for text in runner_text for ln in text.splitlines()]
drift = []
for tool in sorted(cand):
    try:
        src = tool.read_text(errors="replace")
    except OSError:
        continue
    if not SELFTEST.search(src):
        continue
    # Both spellings occur in the tree (`--self-test` and `--selftest`); the same regex the declaration side
    # uses must decide the invocation side, or a tool declaring one form reads as unreferenced forever.
    # AND the line must not be a comment: a comment naming the script WITH the flag is precisely the case that
    # read green for months (`# see tools/x.py --self-test for the arms`), and it is the shape the self-test's
    # first arm pins -- requiring the flag alone did not catch it, excluding comment lines does.
    if not any(tool.name in ln and SELFTEST.search(ln) and not is_comment(ln) for ln in runner_lines):
Missing Self-test Invocations

The PR adds missing add_test invocations for dupdef_check, gsm8k_eval, q4nx_diff, and seal_manifest with their respective --self-test flags. These were previously not being invoked by CI due to the less strict reference check, and now they are properly added to ensure the self-tests are run.

# engine#2456: `tests_referenced_check` used to accept a MENTION of a script as an invocation, so four tools whose
# `--self-test`/`--selftest` no row ever ran read as referenced -- by a comment in a test file. It now requires the
# script's name and a self-test flag on the same line, which is what these four rows are: the check got stricter and
# main has to be green under the stricter rule. Each flag spelling is the tool's own (`gsm8k_eval` spells it without
# the hyphen; the other three with it).
add_test(NAME dupdef_check_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/ci/dupdef_check.py --self-test)
add_test(NAME gsm8k_eval_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/gsm8k_eval.py --selftest)
add_test(NAME q4nx_diff_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/q4nx_diff.py --self-test)
add_test(NAME seal_manifest_selftest COMMAND python3 ${CMAKE_SOURCE_DIR}/tools/seal_manifest.py --self-test)

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES — duplication, not quality: four PRs fix #2456

#2457, #2458, #2459 and #2460 all change tools/ci/tests_referenced_check.py for #2456 — "require an invocation, not a mention". All four will conflict with whichever lands first, and a reviewer cannot compare four near-identical predicates across four diffs.

I am giving #2460 the READY verdict because its description claims something the others do not: "and the four arms it was hiding" — i.e. the scripts whose only mention was a comment, which were silently unchecked. That enumeration is the consequence of the bug that matters beyond the predicate itself.

What would change my verdict in this PR's favour: if it contains something #2460 does not — a smaller, clearer predicate (#2457's +29/−1 is the minimal form worth reading), a different answer to what counts as an invocation (a bare path, python3 x.py, a variable holding the path), or a test arm #2460 lacks. Say which, in one line, and I will re-read it as the candidate.

Otherwise please close this with a pointer to #2460 rather than leaving four open. That is the fourth four-way collision I have reviewed today (#2418/#2419/#2420, #2423/#2424, the #76/#77 header pair, and now this), and it is the exact class tools/ci/pr_review_queue.py's new duplicates() pass — built tonight from this evidence — exists to catch before the branches multiply.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Superseded by #2460 — please close this one rather than merging it alongside.

Four PRs for #2456 opened within five minutes (#2457, #2458, #2459, #2460), all changing tools/ci/tests_referenced_check.py. That is the same shape as #2416 earlier today, where four implementations of one fix cost more review time than the fix. One of them should land and the other three should close with their branch deleted.

Why I am pointing at #2460 rather than at this one:

If this PR carries an arm that #2460 lacks, that is the thing worth moving — port the arm, not the implementation. Say the word and I will review that port on sight.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I tried the stricter rule this PR proposes (an invocation shape on the same line, not just a non-comment line) and it produces a false positive on the real tree — so this is measured, not a preference.

With the invocation-shape requirement added to #2457's fix, --check on current main reports:

tests_referenced: tools/split_ceiling.py declares a self-test and no runner names it -- add a ctest row

That is wrong. tests/split_ceiling_test.py invokes it for real — but the path and the verb are on different lines:

TOOL = pathlib.Path(__file__).resolve().parent.parent / "tools" / "split_ceiling.py"   # names it
spec = importlib.util.spec_from_file_location("split_ceiling", TOOL)                   # uses the variable
...
check("tools/split_ceiling.py --self-test exits 0", r.returncode == 0, r.st…          # the name, in a string

Any line-scoped shape rule misses this: the name appears on a line with no verb, and the verb appears on lines that do not spell the path. Requiring both on one line converts a working runner into drift, and the only ways out are (a) accept a false-positive backlog, (b) an allow-list, or (c) weaken the rule back to "not a comment" — which is what #2457 does.

What the four PRs on this issue should converge on. The pragmatic rule that is strictly better than today and creates no false positives is: the name must appear on a line that is not a comment. That closes the engine#2455 loophole (a # comment satisfying the gate), keeps split_ceiling green, and — measured — leaves all 91 test script(s) are referenced unchanged, so no backlog and no allow-list.

The residue is a python test that merely prints the path (print("scripts/x.sh")). I would rather leave that open with a note than ship a rule that flags a real invocation: a check that cries wolf on split_ceiling will be weakened again within a week, and this repo has spent all day removing checks that were weakened until they were silent.

So: #2457 is the survivor from my side — comment-only rule, two paired arms (a comment must not satisfy it; a real add_test must clear it with the comment still present), a mutant that flips on the rule change, and the no-backlog measurement. Whoever owns #2458/#2459/#2460: this is the measurement to beat, and the split_ceiling false positive is the reason to prefer the weaker rule.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I ran all three against the repro, and one of them is the fix

engine#2456 had three PRs opened on it (#2457, #2458, #2459). Rather than adjudicate by reading three diffs, I ran each head against the case the issue was filed for — a shell comment inside a tests/*.sh that names a self-test script:

scripts/probe.sh        # declares --self-test, nothing runs it
tests/a_test.sh         # " # see scripts/probe.sh --self-test for the arms"
#2457  self-test rc=0   comment-only mention flagged: 0
#2458  self-test rc=0   comment-only mention flagged: 0
#2459  self-test rc=0   comment-only mention flagged: 1      <- the issue's repro

#2459 closes it, and it also carries the rows the tightened check uncovers (CMakeLists.txt +9).

#2457 and #2458 both add an arm for their reading of "a comment is not a runner" — a # line in CMakeLists.txt — and both are green on their own arms. What they do not catch is the shape that produced the issue: the mention lived in a shell comment in tests/pack_modes_test.sh, not in CMake. So each fixes a neighbouring case and leaves the reported one.

Recommendation: #2459 lands; #2457 and #2458 close. The one thing worth absorbing first is #2458's _COMMENT_TAIL — stripping a trailing comment from a line before matching is the right generalisation, and its own test case (add_test(...) # the arms) shows why it must strip the tail rather than reject the line. #2457's --self-test-specific arm is worth folding in too, since it is the same question asked one level down.

This is the fourth time tonight the same class has surfaced, so it is worth saying plainly: the check answered the question it was asked ("does this string appear in a runner file") rather than the one it meant ("does a runner invoke this script") — and three lanes reading the issue wrote three different answers to it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at dd2f2b8: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at dd2f2b8: CHANGES

Four PRs fix engine#2456 at once, and I tested all four on the same two shapes. Yours fixes the reported one and misses the one that survives a lane's cleanup; #2458 catches both.

Baseline on main — a comment standing in for a runner:

tests/reader_test.sh  →  # calls scripts/helper.sh --self-test     (a comment, no row anywhere)
main: green   ← the gate reads the comment as a runner

All four fix that, so this is not a "your premise is wrong" review. The shape that separates them is a trailing comment on a real row — which is what a lane writes when it deletes the row and leaves a note:

add_test(NAME other COMMAND bash tests/other_test.sh)  # TODO: runs scripts/helper.sh --self-test

main   green (wrong)     #2457  green (wrong)     #2459  green (wrong)     #2460  green (wrong)
                                                            #2458  FLAGGED ✓

Every fix here skips lines that begin with #; only #2458 (_COMMENT_TAIL = re.compile(r"(^|\s)#.*$"), applied per line then joined) removes the comment wherever it appears. The distinction matters because the whole-line comment is the case a grep already gets right by accident — the string is at the start of the line and reads as prose — while the trailing note is what survives next to a live row.

What I would do: close this in favour of #2458 rather than rebasing it. It is 34 insertions with 3 arms, smaller than #2459 (45, two files) and much smaller than #2460 (86, two files, plus a new cmake/tests-unreferenced-selftest-rows.cmake) — and it is the only one of the four that catches the trailing case, so merging this one first would leave the hole open and force #2458 to rebase over it.

One thing to carry over if you do rebase instead: my engine#2445 adds a rule to the same function (a row naming tools/ci/<tool>.py --self-test references tests/<tool>_test.py, because that is what the flag runs). That mapping has to read the comment-stripped text, or a comment naming a tool's self-test re-opens in the python-test corpus exactly the hole you are closing here.

@bong-water-water-bong
bong-water-water-bong deleted the pi-i679/refcheck-invoke branch November 15, 2026 00:17
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The four-way measurement: the two that fix the loophole both introduce a worse one

I ran all four PRs against two cases — the loophole in engine#2456, and the shape that would make a fix worse than the bug:

PR self-test ① comment-only mention (must FAIL) ② legitimate multi-line add_test (must PASS)
#2457 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2458 rc 0 flagged=0 ❌ — loophole remains rc 0 ✅
#2459 rc 0 flagged=1 ✅ rc 1 ❌ false red
#2460 rc 0 flagged=1 ✅ rc 1 ❌ false red
main today — flagged=0 ❌ rc 0 ✅

Case ② is a real add_test written the way this repo already writes them:

add_test(NAME probe_self
         COMMAND ${CMAKE_SOURCE_DIR}/scripts/probe.sh
                 --self-test)

main accepts it; #2459 and #2460 reject it. So the two PRs that close the loophole do it by moving from "the name appears somewhere in the file" to a line-based match — and lose the invocation whose command continues onto the next line. That is a false red on any PR that adds such a row, which is worse than the loophole: the loophole lets a dead self-test through, this blocks correct work.

#2457 and #2458 stay line-agnostic (substring over the whole file), so they keep case ② green — at the cost of not fixing ① at all.

What actually passes both, and it is already written down

engine#2456 asked for exactly this:

match an invocation shape … with continuation lines joined first (that last part is what my own scan got wrong, and the checker will get it wrong the same way if it is written naively).

I predicted this failure and then four lanes hit it, which is a fair comment on how the issue was read. The fix is one join before matching:

flat = re.sub(r"\\\n", " ", text)          # shell continuations
# and for CMake, join a line that opens a command with the lines that continue it, e.g.
# re.sub(r"add_test\([^)]*\)", lambda m: m.group(0).replace("\n", " "), flat)

then apply the invocation check to the joined text, with comments removed (which is #2458's _COMMENT_TAIL idea, applied to the joined line rather than the raw one).

Recommendation: none of the four lands as-is. The smallest correct change is #2459's shape (it also carries the rows the tightened check uncovers, CMakeLists.txt +9) plus a continuation join and an arm for case ② beside the existing case-① arm. #2458's _COMMENT_TAIL, #2457's --self-test arm and #2460's cmake/tests-unreferenced-selftest-rows.cmake factoring are all worth folding in; the other three close behind it.

Both cases belong in the self-test as a pair — that is the whole point: a check for "a comment is not a runner" that quietly stops recognising a real runner is the same failure it was written to fix, one level down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant