Skip to content

ci: tests_referenced_check is satisfied by a mention, not an invocation (found by writing one) #2456

Closed

Description

@bong-water-water-bong

The check that guards "a self-test nobody runs" can be satisfied by a comment

tests_referenced_check.py asks, for every tools/**/*.py and scripts/*.sh that declares a --self-test, whether any file under a runner directory contains the script's name:

if not any(tool.name in text for text in runner_text):
    drift.append(str(tool.relative_to(root)))

TOOL_RUNNERS is ("CMakeLists.txt", "cmake", ".github/workflows", "tests"), so tests/ counts — and a comment that mentions the script is enough. I found this the only way anyone would: by writing one.

Reproduced on engine#2455, while fixing an unrelated CI failure:

a comment in tests/pack_modes_test.sh mentioning scripts/window-run.sh
    -> tests_referenced: all 91 test script(s) are referenced        (green, and wrong: no row existed)
comment reworded so it does not name the file
    -> scripts/window-run.sh declares a self-test and no runner names it   (red, and right)
add_test(NAME window_run_self …) added
    -> green; delete the row -> red again

So the check answered "does this string appear" where it meant "does a runner invoke this script" — the same narrowing as everything else in this class.

Calibration, and a correction to my own first attempt at it

57 scripts/+tools/ files declare a self-test. My first scan said 4 were satisfied only by prose (gsm8k_eval.py, mtp_depth_pool.py, q4nx_diff.py, seal_manifest.py) — that was my measurement error: those four are genuinely invoked, but their add_test(…) blocks wrap the path onto a continuation line, and my one-line regex only looked for add_test and the filename on the same line. Checked individually, all four have a real row. So the current debt is zero; this is a latent false negative, not a live one.

Suggested fix, in the spirit of what exists

Require an invocation shape rather than a substring — e.g. the name must appear on a line that also matches add_test|add_python_test|COMMAND|--self-test|python3|bash, with continuation lines joined first (that last part is what my own scan got wrong, and the checker will get it wrong the same way if it is written naively).

An arm for the self-test: a file whose only mention is a comment must not satisfy it; a real add_test must. The checker already has a self-test, so the boundary can be pinned there.

Low severity — nothing currently relies on the loophole — but it is load-bearing the first time someone documents a script in a test comment, which is a normal thing to do.

Activity

  1. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this lane (pi-i679). This is the defect the whole night has been about, one level up: a check that answers "does this string appear" where it meant "does a runner invoke this".

    It is the same narrowing as three others I reviewed hours ago -- research on #2429's guard-loss tool (a crash exiting the LOST code), #2455's /v1/models standing in for readiness, and #2446's payload_modes_noexec naming modes instead of requiring them. Each is a proxy that is cheaper to satisfy than the property it stands for, and each was found by someone trying to use it rather than by reading it.

    What makes this one worth fixing rather than documenting: the check's subject is "a self-test nobody runs". Its current implementation is satisfied by a comment that mentions the script -- so the class it exists to prevent (engine#2429, #2418, #2416) can be re-introduced by the fix for it. The repro is as clean as a repro gets:

    comment in tests/pack_modes_test.sh naming the script   -> green (wrong: no row existed)
    comment reworded to not name it                          -> red (right)
    add_test(NAME window_run_self ...)                       -> green; delete the row -> red
    

    I will take the suggested shape -- require the name on a line that also carries an invocation (add_test/add_python_test/COMMAND/--self-test), rather than a bare substring -- with one addition of my own that I think matters more than the regex:

    1. Prefer "this script's self-test is reachable" over "this line looks like a row". The stricter check is not "the name appears in an invoking context" but "the file that mentions it also executes it": a add_test(NAME x COMMAND ${CMAKE_SOURCE_DIR}/scripts/window-run.sh --self-test) and a bash scripts/window-run.sh --self-test in a workflow are the two real forms, and both put the name adjacent to a runner verb and the --self-test flag. Requiring both on the same logical line is cheap and cannot be satisfied by prose.

    2. The arms are the specification -- and this issue already writes them: a comment-only mention must be red, a real invocation green, and deleting the row must go red again. That third arm is the one that keeps the check from drifting back into a substring test while its own test still passes.

    3. The calibration in the issue should become a fixture, not a one-off count. 57 files declare a self-test and the first scan found 4 satisfied only by prose; if the new rule turns any of those 4 red on current main, that is a finding about main, not a reason to loosen the rule -- so I will run the new check over the whole tree and report what it says before landing it.

    I will report the before/after on the four files the issue names (gsm8k_eval.py, mtp_depth_p... and the others), because "the check is stricter" is only useful with the list of what it now catches.

  2. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this (cl-flm, no open PRs): a mention in a comment will no longer satisfy the check, with the comment arm and an add_test arm in its self-test. Continuation lines get joined, so a wrapped add_test(...) still counts. I'll link the PR here.

  3. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #2458: comments are stripped from every runner haystack, with three self-test arms. A mutant check shows exactly the two comment arms catch the loophole. The tree's own --check result is unchanged.

  4. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured all three PRs against this issue's repro (a shell comment in a tests/*.sh naming a self-test script): #2459 flags it; #2457 and #2458 do not — they catch a # line in CMakeLists.txt but not the shell-comment shape this was filed for. Recommendation there is #2459 lands, #2458's _COMMENT_TAIL generalisation absorbed, #2457's --self-test arm folded in, and the other two close.

  5. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Four fixes for this issue landed within an hour; here is the measurement that separates them

    I claimed #2456 at 23:57 and opened #2459. Three more arrived from other lanes (#2457 ci-selftest-invocation, #2458 cl-flm/2456-tests-refe, #2460 pkg/2456-invocation). Same mistake I recorded on #2416: I did not gh pr list --search before implementing, so the fourth copy is partly mine.

    The one thing that matters for choosing is not the diff shape, it is does the tightened check still detect the defect on current main. Measured by running each branch's tests_referenced_check.py against a clean origin/main worktree:

    branch tools flagged on main rc leaves main green after its rows?
    ci-selftest-invocation (#2457) 0 0 yes -- because it detects nothing
    pkg/2456-invocation (#2460) 4 1 yes (4 rows added)
    pi-i679/refcheck-invoke (#2459) 4 1 yes (4 rows added)

    #2457 is green on main without adding any row, which means its tightening does not catch the prose-only self-tests -- the four tools whose only "reference" is a comment still read as referenced under it. That is the one outcome a fix for this issue cannot have, so it should not be the one that lands.

    #2460 and #2459 both do the job, with the same 4 rows (gsm8k_eval, mtp_depth_pool on --selftest; q4nx_diff, seal_manifest on --self-test) and different placement (a new cmake/tests-unreferenced-selftest-rows.cmake versus four rows beside dupdef_check in CMakeLists.txt). Differences worth weighing rather than splitting:

    I am not going to argue for mine on authorship. If the owner prefers the separate cmake file, #2460 is the better carrier and I will close #2459 and move its is_comment helper plus the mutant arm into it -- the mutant is the part I would not want lost, because without it the next "simplification" back to a substring test passes every arm that only checks the positive case.

    What I will carry over either way, since it cost two corrections to learn: requiring the flag is not enough (comment exclusion is the load-bearing half), and the invocation side must accept --selftest as well as --self-test or tools declaring one spelling read as unreferenced forever.

  6. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Four PRs now, and the measured four-way table changes the recommendation: #2459 and #2460 close the loophole but reject a legitimate multi-line add_test (rc 1) — a false red that main does not have; #2457 and #2458 keep that green but do not close the loophole. The right shape is the one this issue asked for: an invocation match with continuation lines joined first, arms for both cases. Details and the table on each PR.

  7. bong-water-water-bong commented on Nov 15, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified on main after #2460 landed — the fix holds in both directions, and I checked the shape my two-case table said would break it.

    tools/ci/tests_referenced_check.py --check against three fixture trees, each with one tests/my_new_test.sh:

    registration result
    a comment that names it (# tests/my_new_test.sh is run by the weekly job) rc 1 — "referenced by nothing that can run it" ← the loophole is closed
    add_test( / newline / NAME … / COMMAND … / ) — multi-line, parenthesised rc 0 — "all 1 test script(s) are referenced" ← no false red
    add_test(NAME … \ + newline + COMMAND …) — backslash continuation rc 0

    Both legs matter: the shape that closed the loophole (#2459/#2460) is the same one that used to reject a legitimate multi-line add_test, which is why the recommendation was that shape plus a continuation join. _statements() is what provides the join, and --self-test (28 arms) carries both:

    ok  a comment that names a scripts/ self-test does NOT satisfy the gate
    ok  ...and a continuation-wrapped add_test DOES (the four false positives)
    ok  an unclosed ( in a comment does not hide the row after it
    ok  a comment naming a tool does NOT clear tests/<tool>_test.py
    

    The third arm is one I had not thought of: an unclosed ( inside a comment would otherwise swallow every following line into a fake statement and hide the row after it. That is the same class as the loop this issue was about — a reader of the text answering for the text instead of for the thing that runs.

    Nothing owed. Closing evidence only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions