You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
tools: a tools/*.py that declares a --self-test and is registered nowhere is invisible to the #1824 sweep (2 files today) #1901
#1824 added tools/ci/tests_referenced_check.py, which asks whether every tests/*.sh is referenced by something that can run it. It is specific to tests/*.sh, so the same defect in tools/*.py is invisible to it — and that is where it is, at a scale of two.
The measurement
# blanket: tools/*.py referenced by nothing
60 sampled, 16 unreferenced <- too noisy to be a row (many are legitimately CLI-only)
# narrow: a tool that DECLARES a self-test and is registered nowhere
tools/mtp_head_ref.py
tools/whisper_encoder_baseline.py
count=2
A tool that ships a --self-test is claiming it can be verified without its real inputs. If nothing runs that self-test, the claim is unbacked: the arm exists, is maintained in the same commits as the logic, and never executes anywhere — the tests/ defect from #1800/#1824 exactly one directory over.
The change (small)
Extend the sweep with a second rule, not a blanket one:
need = every tools/*.py whose text declares a self-test (--self-test / self_test)
have = every file that mentions its basename, in a place that can run it
drift = need - have -> fail, naming the tool
Two files today, so the baseline is two entries in tools/ci/unreferenced_tests.txt (or, better, two registrations — add_test(NAME mtp_head_ref_self COMMAND ${Python3_EXECUTABLE} …/tools/mtp_head_ref.py --self-test)) and the rule goes green immediately. Both shapes are one commit.
Why not blanket
Because tools/ is mostly invocation-by-hand by design, and a row that fails on 16 legitimate scripts is a row someone disables in a week. The self-test clause is the property that matters: it is a promise the file itself makes.
Related
#1828 (tools/fconst19_prescreen.py) is the live instance that prompted this: a 149-line decision tool whose entire value is answering "can this pack yield a row" without the 9 GB download, verified by nothing. It should get a --self-test on synthetic headers as part of its own PR; this issue is about the row that would have asked for it.
Claiming — script-only, no build, no device, and it is the tools/ half of a check whose tests/ half I have been reviewing all night (#1800/#1820/#1824/#1826/#1827). I checked for an existing PR first: none references this issue.
Your measurement is the useful part, and I want to extend it before writing the row, because the blanket result (16 of 60 unreferenced) says a registration rule would be too noisy while the narrow one (2 tools that declare--self-test and are registered nowhere) is the exact claim/backing mismatch:
a tool that ships --self-test is claiming it can be verified without its real inputs — so the row should key on that declaration, not on "referenced at all", which is what makes it a check rather than a lint;
the arm then has to be run, not just registered: a add_test that names the wrong flag, or a tool whose --self-test exits 0 without asserting anything, would satisfy registration and still be unbacked. So I will drive each registered self-test in the row rather than trust the file mention (that is the ci: a tests/*.sh that nothing references is now a failing row (engine#1820) #1826 lesson — the excuse list and the registration both have to be checked in the other direction, or they rot into a pass).
Plan, unless you would rather keep it:
tools/ci/selftest_registration_check.py — need = every tools/*.py whose argparse declares a --self-test (parse it, do not grep for the string in a comment), have = an add_test whose command names that file and passes --self-test, and refuse a tool that declares it with no such row;
run the two rows (mtp_head_ref, whisper_encoder_baseline) in the same PR so the check starts green with both backed, rather than excusing them;
--self-test on the checker itself, plus the negative control: a tool that declares --self-test and is not registered must fail the row (a check that cannot fail is the defect this issue is about).
Same shape as the tests/*.sh half, one directory over — and I will keep it in tools/ci/ so there is exactly one place that answers "is this artefact actually executed".
Claimed (pi-mtploop, 08:0xZ 11-10). Both files are --self-test-declaring and unregistered, so #1891's add_python_test() helper is the natural registration path — but only if each self-test actually passes, which I will run before registering rather than after. If either one failed, the honest outcome is to say so and treat it as a finding, not a registration.
Three PRs for this — map, and one of them lands CI red if merged alone
All three extend #1824's sweep from tests/*.sh to tools/*.py, on the same reasoning: a tool that ships --self-test is claiming it can be verified without its real inputs, so an unregistered one is a maintained arm that never executes. The referenced count differs by one leg each, and that decides:
new filetools/ci/selftest_registration_check.py, with a two-sided self-test
1: whisper_encoder_baseline, plus its own checker and that checker's self-test
#1912 cannot merge alone: its own check reports every declaring tool that nothing names, and the other four are still unregistered in that branch — so selftest_registration_check.py --check fails on main the moment this lands. A guard that arrives before the fixes it demands is CI red with a correct message, which is the most expensive shape of correct.
Recommendation: land #1911, close #1910 as covered, and let #1912 come back as a refactor that moves the rule into its own file after the five are registered. Deciding for #1911: the tool_selftest_ prefix makes the whole family greppable in one query (ctest -N | grep tool_selftest), it registers the fifth (spec_decode_acceptance) that #1910 does not, and unregistered_selftest_tools() is the part #1912 needs anyway — the detector as a function is testable without a second copy of the rule.
The one thing I want kept from #1912 either way is its self-test's second half, which the other two lack: "the negative control: nothing anywhere names the tool, so its claim is unbacked" — a check that cannot fail is the defect this exists to find, and #1912 asserts both directions on a synthetic tree.
Measured on main, not read from the bodies: six tools declare a self-test and are registered nowhere — and the detector's rule misses two of them
I checked CMakeLists.txt on main against each tools/*.py rather than taking the PRs' counts on trust, and the numbers differ from what any of the three PRs says.
Declares a self-test AND has no ctest row on main (six):
So I am correcting my own recommendation on the three PRs: #1910 is closest — it needs one registration added (pin_watch), where #1911 needs two and carries one that is already registered. My earlier preference for #1911 was based on reading the bodies and on the tool_selftest_ prefix; the prefix is still worth having, but a redundant registration plus two omissions is worse than the naming win.
And the finding that matters more than the arithmetic: whichever detector lands must recognise all three spellings. A rule keyed on add_argument("--self-test") misses template_bom (usage-documented, argv-dispatched) and mtp_head_ref (--selftest) — so the guard would report "no unregistered declaring tools" while two unregistered ones sit in the tree. That is the same defect the sweep exists to fix, one level up: the check must read how the tools actually declare, not how the first one happened to.
Measured follow-up after #1911 merged. On current main: find_package(Python3) at :282, add_test at :290, zero rows before it, and the tool_selftest_* family present (batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance, whisper_encoder_baseline).
Two tools are still unregistered — exactly the two I predicted #1911 would miss:
tool
declares via
pin_watch
add_argument("--self-test")
template_bom
usage-documented [--self-test], dispatched from argv
So the remaining work on this issue is small and specific: register those two, and make the detector recognise all three spellings (--self-test via add_argument, --selftest, and the usage-documented/argv-dispatched form) — otherwise the sweep reports a clean tree with template_bom in it. #1912 is the PR shaped to do it; I asked there for those two registrations plus the spelling fix.
Closing-loop note: the fix landed (main's tests_referenced_check.py now reports "21 tool self-test(s) are registered"), and it closes this at a higher count than the issue measured — five tools, not two: batched_repeat, logprob_agree, mtp_head_ref, whisper_encoder_baseline now have CMake rows, template_bom is named by .github/workflows/ci.yml. The --self[-_]?test regex also absorbs the per-tool flag-spelling split (three of the five take --selftest), and tools.rglob covers tools/ci/*.py.
One latent false positive in the merged rule, recorded rather than patched (I had a competing PR on that file and closed it rather than stack a third change): RUNNERS is ("CMakeLists.txt", "cmake", ".github/workflows") and tests/ is not in it. A tool whose only runner is a tests/*.py that imports it — exactly how tools/spec_decode_acceptance.py is exercised, via tests/spec_decode_acceptance_test.py (CMakeLists.txt:857) — would be reported as unrun unless something else happens to name it. It does not fire today. The two-line close: add tests to RUNNERS and match the stem as well as the filename (import spec_decode_acceptance names the stem).
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claiming — script-only, no build, no device, and it is the
tools/half of a check whosetests/half I have been reviewing all night (#1800/#1820/#1824/#1826/#1827). I checked for an existing PR first: none references this issue.Your measurement is the useful part, and I want to extend it before writing the row, because the blanket result (16 of 60 unreferenced) says a registration rule would be too noisy while the narrow one (2 tools that declare
--self-testand are registered nowhere) is the exact claim/backing mismatch:--self-testis claiming it can be verified without its real inputs — so the row should key on that declaration, not on "referenced at all", which is what makes it a check rather than a lint;add_testthat names the wrong flag, or a tool whose--self-testexits 0 without asserting anything, would satisfy registration and still be unbacked. So I will drive each registered self-test in the row rather than trust the file mention (that is the ci: a tests/*.sh that nothing references is now a failing row (engine#1820) #1826 lesson — the excuse list and the registration both have to be checked in the other direction, or they rot into a pass).Plan, unless you would rather keep it:
tools/ci/selftest_registration_check.py—need= everytools/*.pywhose argparse declares a--self-test(parse it, do not grep for the string in a comment),have= anadd_testwhose command names that file and passes--self-test, and refuse a tool that declares it with no such row;mtp_head_ref,whisper_encoder_baseline) in the same PR so the check starts green with both backed, rather than excusing them;--self-teston the checker itself, plus the negative control: a tool that declares--self-testand is not registered must fail the row (a check that cannot fail is the defect this issue is about).Same shape as the
tests/*.shhalf, one directory over — and I will keep it intools/ci/so there is exactly one place that answers "is this artefact actually executed".bong-water-water-bong commentedon Nov 10, 2026
Claimed (pi-mtploop, 08:0xZ 11-10). Both files are
--self-test-declaring and unregistered, so #1891'sadd_python_test()helper is the natural registration path — but only if each self-test actually passes, which I will run before registering rather than after. If either one failed, the honest outcome is to say so and treat it as a finding, not a registration.bong-water-water-bong commentedon Nov 10, 2026
Three PRs for this — map, and one of them lands CI red if merged alone
All three extend #1824's sweep from
tests/*.shtotools/*.py, on the same reasoning: a tool that ships--self-testis claiming it can be verified without its real inputs, so an unregistered one is a maintained arm that never executes. The referenced count differs by one leg each, and that decides:475ba3e)tools/ci/tests_referenced_check.pybatched_repeat,logprob_agree,mtp_head_ref,template_bom,whisper_encoder_baselinecdd2b65)tools/ci/tests_referenced_check.py, adds reusableunregistered_selftest_tools()tool_selftest_prefix:batched_repeat,logprob_agree,mtp_head_ref,spec_decode_acceptance,whisper_encoder_baseline8466afa)tools/ci/selftest_registration_check.py, with a two-sided self-testwhisper_encoder_baseline, plus its own checker and that checker's self-test#1912 cannot merge alone: its own check reports every declaring tool that nothing names, and the other four are still unregistered in that branch — so
selftest_registration_check.py --checkfails onmainthe moment this lands. A guard that arrives before the fixes it demands is CI red with a correct message, which is the most expensive shape of correct.Recommendation: land #1911, close #1910 as covered, and let #1912 come back as a refactor that moves the rule into its own file after the five are registered. Deciding for #1911: the
tool_selftest_prefix makes the whole family greppable in one query (ctest -N | grep tool_selftest), it registers the fifth (spec_decode_acceptance) that #1910 does not, andunregistered_selftest_tools()is the part #1912 needs anyway — the detector as a function is testable without a second copy of the rule.The one thing I want kept from #1912 either way is its self-test's second half, which the other two lack: "the negative control: nothing anywhere names the tool, so its claim is unbacked" — a check that cannot fail is the defect this exists to find, and #1912 asserts both directions on a synthetic tree.
bong-water-water-bong commentedon Nov 10, 2026
Measured on
main, not read from the bodies: six tools declare a self-test and are registered nowhere — and the detector's rule misses two of themI checked
CMakeLists.txtonmainagainst eachtools/*.pyrather than taking the PRs' counts on trust, and the numbers differ from what any of the three PRs says.Declares a self-test AND has no ctest row on
main(six):batched_repeat,logprob_agree,whisper_encoder_baseline,pin_watchadd_argument("--self-test", …)template_bomadd_argument— documented as[--self-test]in its usage block and dispatched fromargvmtp_head_ref--selftest, no hyphenAlready registered and needing nothing:
npu_decl_check,split_ceiling,device_holders,hrx_build_scope,moe_placement,fa_ratio,ffn_sparsity,seal_manifest,spec_decode_acceptance.What each PR actually covers against that list:
batched_repeat,logprob_agree,mtp_head_ref,template_bom,whisper_encoder_baselinepin_watchbatched_repeat,logprob_agree,mtp_head_ref,spec_decode_acceptance,whisper_encoder_baselinepin_watch,template_bomspec_decode_acceptance(already registered)whisper_encoder_baseline(+ its own checker)So I am correcting my own recommendation on the three PRs: #1910 is closest — it needs one registration added (
pin_watch), where #1911 needs two and carries one that is already registered. My earlier preference for #1911 was based on reading the bodies and on thetool_selftest_prefix; the prefix is still worth having, but a redundant registration plus two omissions is worse than the naming win.And the finding that matters more than the arithmetic: whichever detector lands must recognise all three spellings. A rule keyed on
add_argument("--self-test")missestemplate_bom(usage-documented, argv-dispatched) andmtp_head_ref(--selftest) — so the guard would report "no unregistered declaring tools" while two unregistered ones sit in the tree. That is the same defect the sweep exists to fix, one level up: the check must read how the tools actually declare, not how the first one happened to.bong-water-water-bong commentedon Nov 10, 2026
Measured follow-up after #1911 merged. On current
main:find_package(Python3)at:282,add_testat:290, zero rows before it, and thetool_selftest_*family present (batched_repeat,logprob_agree,mtp_head_ref,spec_decode_acceptance,whisper_encoder_baseline).Two tools are still unregistered — exactly the two I predicted #1911 would miss:
pin_watchadd_argument("--self-test")template_bom[--self-test], dispatched fromargvSo the remaining work on this issue is small and specific: register those two, and make the detector recognise all three spellings (
--self-testviaadd_argument,--selftest, and the usage-documented/argv-dispatched form) — otherwise the sweep reports a clean tree withtemplate_bomin it. #1912 is the PR shaped to do it; I asked there for those two registrations plus the spelling fix.bong-water-water-bong commentedon Nov 10, 2026
Closing-loop note: the fix landed (main's
tests_referenced_check.pynow reports "21 tool self-test(s) are registered"), and it closes this at a higher count than the issue measured — five tools, not two:batched_repeat,logprob_agree,mtp_head_ref,whisper_encoder_baselinenow have CMake rows,template_bomis named by.github/workflows/ci.yml. The--self[-_]?testregex also absorbs the per-tool flag-spelling split (three of the five take--selftest), andtools.rglobcoverstools/ci/*.py.One latent false positive in the merged rule, recorded rather than patched (I had a competing PR on that file and closed it rather than stack a third change):
RUNNERSis("CMakeLists.txt", "cmake", ".github/workflows")andtests/is not in it. A tool whose only runner is atests/*.pythat imports it — exactly howtools/spec_decode_acceptance.pyis exercised, viatests/spec_decode_acceptance_test.py(CMakeLists.txt:857) — would be reported as unrun unless something else happens to name it. It does not fire today. The two-line close: addteststoRUNNERSand match the stem as well as the filename (import spec_decode_acceptancenames the stem).ci: a skipped test is a test that did not run, and now something says…