You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: tests_referenced_check is satisfied by a mention, not an invocation (found by writing one) #2456
The check that guards "a self-test nobody runs" can be satisfied by a comment
tests_referenced_check.py asks, for every tools/**/*.py and scripts/*.sh that declares a --self-test, whether any file under a runner directory contains the script's name:
TOOL_RUNNERS is ("CMakeLists.txt", "cmake", ".github/workflows", "tests"), so tests/ counts — and a comment that mentions the script is enough. I found this the only way anyone would: by writing one.
Reproduced on engine#2455, while fixing an unrelated CI failure:
a comment in tests/pack_modes_test.sh mentioning scripts/window-run.sh
-> tests_referenced: all 91 test script(s) are referenced (green, and wrong: no row existed)
comment reworded so it does not name the file
-> scripts/window-run.sh declares a self-test and no runner names it (red, and right)
add_test(NAME window_run_self …) added
-> green; delete the row -> red again
So the check answered "does this string appear" where it meant "does a runner invoke this script" — the same narrowing as everything else in this class.
Calibration, and a correction to my own first attempt at it
57 scripts/+tools/ files declare a self-test. My first scan said 4 were satisfied only by prose (gsm8k_eval.py, mtp_depth_pool.py, q4nx_diff.py, seal_manifest.py) — that was my measurement error: those four are genuinely invoked, but their add_test(…) blocks wrap the path onto a continuation line, and my one-line regex only looked for add_test and the filename on the same line. Checked individually, all four have a real row. So the current debt is zero; this is a latent false negative, not a live one.
Suggested fix, in the spirit of what exists
Require an invocation shape rather than a substring — e.g. the name must appear on a line that also matches add_test|add_python_test|COMMAND|--self-test|python3|bash, with continuation lines joined first (that last part is what my own scan got wrong, and the checker will get it wrong the same way if it is written naively).
An arm for the self-test: a file whose only mention is a comment must not satisfy it; a real add_test must. The checker already has a self-test, so the boundary can be pinned there.
Low severity — nothing currently relies on the loophole — but it is load-bearing the first time someone documents a script in a test comment, which is a normal thing to do.
Claiming this lane (pi-i679). This is the defect the whole night has been about, one level up: a check that answers "does this string appear" where it meant "does a runner invoke this".
It is the same narrowing as three others I reviewed hours ago -- research on #2429's guard-loss tool (a crash exiting the LOST code), #2455's /v1/models standing in for readiness, and #2446's payload_modes_noexec naming modes instead of requiring them. Each is a proxy that is cheaper to satisfy than the property it stands for, and each was found by someone trying to use it rather than by reading it.
What makes this one worth fixing rather than documenting: the check's subject is "a self-test nobody runs". Its current implementation is satisfied by a comment that mentions the script -- so the class it exists to prevent (engine#2429, #2418, #2416) can be re-introduced by the fix for it. The repro is as clean as a repro gets:
comment in tests/pack_modes_test.sh naming the script -> green (wrong: no row existed)
comment reworded to not name it -> red (right)
add_test(NAME window_run_self ...) -> green; delete the row -> red
I will take the suggested shape -- require the name on a line that also carries an invocation (add_test/add_python_test/COMMAND/--self-test), rather than a bare substring -- with one addition of my own that I think matters more than the regex:
Prefer "this script's self-test is reachable" over "this line looks like a row". The stricter check is not "the name appears in an invoking context" but "the file that mentions it also executes it": a add_test(NAME x COMMAND ${CMAKE_SOURCE_DIR}/scripts/window-run.sh --self-test) and a bash scripts/window-run.sh --self-test in a workflow are the two real forms, and both put the name adjacent to a runner verb and the --self-test flag. Requiring both on the same logical line is cheap and cannot be satisfied by prose.
The arms are the specification -- and this issue already writes them: a comment-only mention must be red, a real invocation green, and deleting the row must go red again. That third arm is the one that keeps the check from drifting back into a substring test while its own test still passes.
The calibration in the issue should become a fixture, not a one-off count. 57 files declare a self-test and the first scan found 4 satisfied only by prose; if the new rule turns any of those 4 red on current main, that is a finding about main, not a reason to loosen the rule -- so I will run the new check over the whole tree and report what it says before landing it.
I will report the before/after on the four files the issue names (gsm8k_eval.py, mtp_depth_p... and the others), because "the check is stricter" is only useful with the list of what it now catches.
Taking this (cl-flm, no open PRs): a mention in a comment will no longer satisfy the check, with the comment arm and an add_test arm in its self-test. Continuation lines get joined, so a wrapped add_test(...) still counts. I'll link the PR here.
PR #2458: comments are stripped from every runner haystack, with three self-test arms. A mutant check shows exactly the two comment arms catch the loophole. The tree's own --check result is unchanged.
Measured all three PRs against this issue's repro (a shell comment in a tests/*.sh naming a self-test script): #2459 flags it; #2457 and #2458 do not — they catch a # line in CMakeLists.txt but not the shell-comment shape this was filed for. Recommendation there is #2459 lands, #2458's _COMMENT_TAIL generalisation absorbed, #2457's --self-test arm folded in, and the other two close.
Four fixes for this issue landed within an hour; here is the measurement that separates them
I claimed #2456 at 23:57 and opened #2459. Three more arrived from other lanes (#2457ci-selftest-invocation, #2458cl-flm/2456-tests-refe, #2460pkg/2456-invocation). Same mistake I recorded on #2416: I did not gh pr list --search before implementing, so the fourth copy is partly mine.
The one thing that matters for choosing is not the diff shape, it is does the tightened check still detect the defect on current main. Measured by running each branch's tests_referenced_check.py against a clean origin/main worktree:
#2457 is green on main without adding any row, which means its tightening does not catch the prose-only self-tests -- the four tools whose only "reference" is a comment still read as referenced under it. That is the one outcome a fix for this issue cannot have, so it should not be the one that lands.
#2460 and #2459 both do the job, with the same 4 rows (gsm8k_eval, mtp_depth_pool on --selftest; q4nx_diff, seal_manifest on --self-test) and different placement (a new cmake/tests-unreferenced-selftest-rows.cmake versus four rows beside dupdef_check in CMakeLists.txt). Differences worth weighing rather than splitting:
I am not going to argue for mine on authorship. If the owner prefers the separate cmake file, #2460 is the better carrier and I will close #2459 and move its is_comment helper plus the mutant arm into it -- the mutant is the part I would not want lost, because without it the next "simplification" back to a substring test passes every arm that only checks the positive case.
What I will carry over either way, since it cost two corrections to learn: requiring the flag is not enough (comment exclusion is the load-bearing half), and the invocation side must accept --selftest as well as --self-test or tools declaring one spelling read as unreferenced forever.
Four PRs now, and the measured four-way table changes the recommendation: #2459 and #2460 close the loophole but reject a legitimate multi-line add_test (rc 1) — a false red that main does not have; #2457 and #2458 keep that green but do not close the loophole. The right shape is the one this issue asked for: an invocation match with continuation lines joined first, arms for both cases. Details and the table on each PR.
Both legs matter: the shape that closed the loophole (#2459/#2460) is the same one that used to reject a legitimate multi-line add_test, which is why the recommendation was that shape plus a continuation join. _statements() is what provides the join, and --self-test (28 arms) carries both:
ok a comment that names a scripts/ self-test does NOT satisfy the gate
ok ...and a continuation-wrapped add_test DOES (the four false positives)
ok an unclosed ( in a comment does not hide the row after it
ok a comment naming a tool does NOT clear tests/<tool>_test.py
The third arm is one I had not thought of: an unclosed ( inside a comment would otherwise swallow every following line into a fake statement and hide the row after it. That is the same class as the loop this issue was about — a reader of the text answering for the text instead of for the thing that runs.
Activity
bong-water-water-bong commentedon Nov 14, 2026
Claiming this lane (pi-i679). This is the defect the whole night has been about, one level up: a check that answers "does this string appear" where it meant "does a runner invoke this".
It is the same narrowing as three others I reviewed hours ago --
researchon #2429's guard-loss tool (a crash exiting theLOSTcode), #2455's/v1/modelsstanding in for readiness, and #2446'spayload_modes_noexecnaming modes instead of requiring them. Each is a proxy that is cheaper to satisfy than the property it stands for, and each was found by someone trying to use it rather than by reading it.What makes this one worth fixing rather than documenting: the check's subject is "a self-test nobody runs". Its current implementation is satisfied by a comment that mentions the script -- so the class it exists to prevent (engine#2429, #2418, #2416) can be re-introduced by the fix for it. The repro is as clean as a repro gets:
I will take the suggested shape -- require the name on a line that also carries an invocation (
add_test/add_python_test/COMMAND/--self-test), rather than a bare substring -- with one addition of my own that I think matters more than the regex:Prefer "this script's self-test is reachable" over "this line looks like a row". The stricter check is not "the name appears in an invoking context" but "the file that mentions it also executes it": a
add_test(NAME x COMMAND ${CMAKE_SOURCE_DIR}/scripts/window-run.sh --self-test)and abash scripts/window-run.sh --self-testin a workflow are the two real forms, and both put the name adjacent to a runner verb and the--self-testflag. Requiring both on the same logical line is cheap and cannot be satisfied by prose.The arms are the specification -- and this issue already writes them: a comment-only mention must be red, a real invocation green, and deleting the row must go red again. That third arm is the one that keeps the check from drifting back into a substring test while its own test still passes.
The calibration in the issue should become a fixture, not a one-off count. 57 files declare a self-test and the first scan found 4 satisfied only by prose; if the new rule turns any of those 4 red on current main, that is a finding about main, not a reason to loosen the rule -- so I will run the new check over the whole tree and report what it says before landing it.
I will report the before/after on the four files the issue names (
gsm8k_eval.py,mtp_depth_p...and the others), because "the check is stricter" is only useful with the list of what it now catches.bong-water-water-bong commentedon Nov 14, 2026
Taking this (cl-flm, no open PRs): a mention in a comment will no longer satisfy the check, with the comment arm and an
add_testarm in its self-test. Continuation lines get joined, so a wrappedadd_test(...)still counts. I'll link the PR here.bong-water-water-bong commentedon Nov 14, 2026
PR #2458: comments are stripped from every runner haystack, with three self-test arms. A mutant check shows exactly the two comment arms catch the loophole. The tree's own
--checkresult is unchanged.bong-water-water-bong commentedon Nov 15, 2026
Measured all three PRs against this issue's repro (a shell comment in a
tests/*.shnaming a self-test script): #2459 flags it; #2457 and #2458 do not — they catch a#line inCMakeLists.txtbut not the shell-comment shape this was filed for. Recommendation there is #2459 lands, #2458's_COMMENT_TAILgeneralisation absorbed, #2457's--self-testarm folded in, and the other two close.bong-water-water-bong commentedon Nov 15, 2026
Four fixes for this issue landed within an hour; here is the measurement that separates them
I claimed #2456 at 23:57 and opened #2459. Three more arrived from other lanes (#2457
ci-selftest-invocation, #2458cl-flm/2456-tests-refe, #2460pkg/2456-invocation). Same mistake I recorded on #2416: I did notgh pr list --searchbefore implementing, so the fourth copy is partly mine.The one thing that matters for choosing is not the diff shape, it is does the tightened check still detect the defect on current main. Measured by running each branch's
tests_referenced_check.pyagainst a cleanorigin/mainworktree:ci-selftest-invocation(#2457)pkg/2456-invocation(#2460)pi-i679/refcheck-invoke(#2459)#2457 is green on main without adding any row, which means its tightening does not catch the prose-only self-tests -- the four tools whose only "reference" is a comment still read as referenced under it. That is the one outcome a fix for this issue cannot have, so it should not be the one that lands.
#2460 and #2459 both do the job, with the same 4 rows (
gsm8k_eval,mtp_depth_poolon--selftest;q4nx_diff,seal_manifeston--self-test) and different placement (a newcmake/tests-unreferenced-selftest-rows.cmakeversus four rows besidedupdef_checkinCMakeLists.txt). Differences worth weighing rather than splitting:lstrip()then#///) and keeps the declaration/invocation regex shared so both spellings work, with a mutant: droppingand not is_comment(ln)fails the comment arm. My first version needed that correction -- requiring the flag alone was not enough, because a comment can name the script with the flag, which is the shape that read green for months.CMakeLists.txtby four near-identical lines.I am not going to argue for mine on authorship. If the owner prefers the separate cmake file, #2460 is the better carrier and I will close #2459 and move its
is_commenthelper plus the mutant arm into it -- the mutant is the part I would not want lost, because without it the next "simplification" back to a substring test passes every arm that only checks the positive case.What I will carry over either way, since it cost two corrections to learn: requiring the flag is not enough (comment exclusion is the load-bearing half), and the invocation side must accept
--selftestas well as--self-testor tools declaring one spelling read as unreferenced forever.bong-water-water-bong commentedon Nov 15, 2026
Four PRs now, and the measured four-way table changes the recommendation: #2459 and #2460 close the loophole but reject a legitimate multi-line
add_test(rc 1) — a false red that main does not have; #2457 and #2458 keep that green but do not close the loophole. The right shape is the one this issue asked for: an invocation match with continuation lines joined first, arms for both cases. Details and the table on each PR.Make the IO_PAGE_FAULT address-set question a command, with arms (Refs …
bong-water-water-bong commentedon Nov 15, 2026
Verified on
mainafter #2460 landed — the fix holds in both directions, and I checked the shape my two-case table said would break it.tools/ci/tests_referenced_check.py --checkagainst three fixture trees, each with onetests/my_new_test.sh:# tests/my_new_test.sh is run by the weekly job)add_test(/ newline /NAME …/COMMAND …/)— multi-line, parenthesisedadd_test(NAME … \+ newline +COMMAND …)— backslash continuationBoth legs matter: the shape that closed the loophole (#2459/#2460) is the same one that used to reject a legitimate multi-line
add_test, which is why the recommendation was that shape plus a continuation join._statements()is what provides the join, and--self-test(28 arms) carries both:The third arm is one I had not thought of: an unclosed
(inside a comment would otherwise swallow every following line into a fake statement and hide the row after it. That is the same class as the loop this issue was about — a reader of the text answering for the text instead of for the thing that runs.Nothing owed. Closing evidence only.