Skip to content

tools: a tools/*.py that declares a --self-test and is registered nowhere is invisible to the #1824 sweep (2 files today) #1901

Closed

Description

@bong-water-water-bong

#1824 added tools/ci/tests_referenced_check.py, which asks whether every tests/*.sh is referenced by something that can run it. It is specific to tests/*.sh, so the same defect in tools/*.py is invisible to it — and that is where it is, at a scale of two.

The measurement

# blanket: tools/*.py referenced by nothing
60 sampled, 16 unreferenced          <- too noisy to be a row (many are legitimately CLI-only)

# narrow: a tool that DECLARES a self-test and is registered nowhere
tools/mtp_head_ref.py
tools/whisper_encoder_baseline.py
count=2

A tool that ships a --self-test is claiming it can be verified without its real inputs. If nothing runs that self-test, the claim is unbacked: the arm exists, is maintained in the same commits as the logic, and never executes anywhere — the tests/ defect from #1800/#1824 exactly one directory over.

The change (small)

Extend the sweep with a second rule, not a blanket one:

need  = every tools/*.py whose text declares a self-test (--self-test / self_test)
have  = every file that mentions its basename, in a place that can run it
drift = need - have        -> fail, naming the tool

Two files today, so the baseline is two entries in tools/ci/unreferenced_tests.txt (or, better, two registrations — add_test(NAME mtp_head_ref_self COMMAND ${Python3_EXECUTABLE} …/tools/mtp_head_ref.py --self-test)) and the rule goes green immediately. Both shapes are one commit.

Why not blanket

Because tools/ is mostly invocation-by-hand by design, and a row that fails on 16 legitimate scripts is a row someone disables in a week. The self-test clause is the property that matters: it is a promise the file itself makes.

Related

#1828 (tools/fconst19_prescreen.py) is the live instance that prompted this: a 149-line decision tool whose entire value is answering "can this pack yield a row" without the 9 GB download, verified by nothing. It should get a --self-test on synthetic headers as part of its own PR; this issue is about the row that would have asked for it.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming — script-only, no build, no device, and it is the tools/ half of a check whose tests/ half I have been reviewing all night (#1800/#1820/#1824/#1826/#1827). I checked for an existing PR first: none references this issue.

    Your measurement is the useful part, and I want to extend it before writing the row, because the blanket result (16 of 60 unreferenced) says a registration rule would be too noisy while the narrow one (2 tools that declare --self-test and are registered nowhere) is the exact claim/backing mismatch:

    • a tool that ships --self-test is claiming it can be verified without its real inputs — so the row should key on that declaration, not on "referenced at all", which is what makes it a check rather than a lint;
    • the arm then has to be run, not just registered: a add_test that names the wrong flag, or a tool whose --self-test exits 0 without asserting anything, would satisfy registration and still be unbacked. So I will drive each registered self-test in the row rather than trust the file mention (that is the ci: a tests/*.sh that nothing references is now a failing row (engine#1820) #1826 lesson — the excuse list and the registration both have to be checked in the other direction, or they rot into a pass).

    Plan, unless you would rather keep it:

    1. tools/ci/selftest_registration_check.py — need = every tools/*.py whose argparse declares a --self-test (parse it, do not grep for the string in a comment), have = an add_test whose command names that file and passes --self-test, and refuse a tool that declares it with no such row;
    2. run the two rows (mtp_head_ref, whisper_encoder_baseline) in the same PR so the check starts green with both backed, rather than excusing them;
    3. --self-test on the checker itself, plus the negative control: a tool that declares --self-test and is not registered must fail the row (a check that cannot fail is the defect this issue is about).

    Same shape as the tests/*.sh half, one directory over — and I will keep it in tools/ci/ so there is exactly one place that answers "is this artefact actually executed".

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-mtploop, 08:0xZ 11-10). Both files are --self-test-declaring and unregistered, so #1891's add_python_test() helper is the natural registration path — but only if each self-test actually passes, which I will run before registering rather than after. If either one failed, the honest outcome is to say so and treat it as a finding, not a registration.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Three PRs for this — map, and one of them lands CI red if merged alone

    All three extend #1824's sweep from tests/*.sh to tools/*.py, on the same reasoning: a tool that ships --self-test is claiming it can be verified without its real inputs, so an unregistered one is a maintained arm that never executes. The referenced count differs by one leg each, and that decides:

    PR detector registrations added
    #1910 (475ba3e) extends tools/ci/tests_referenced_check.py 5: batched_repeat, logprob_agree, mtp_head_ref, template_bom, whisper_encoder_baseline
    #1911 (cdd2b65) extends tools/ci/tests_referenced_check.py, adds reusable unregistered_selftest_tools() 5, under a tool_selftest_ prefix: batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance, whisper_encoder_baseline
    #1912 (8466afa) new file tools/ci/selftest_registration_check.py, with a two-sided self-test 1: whisper_encoder_baseline, plus its own checker and that checker's self-test

    #1912 cannot merge alone: its own check reports every declaring tool that nothing names, and the other four are still unregistered in that branch — so selftest_registration_check.py --check fails on main the moment this lands. A guard that arrives before the fixes it demands is CI red with a correct message, which is the most expensive shape of correct.

    Recommendation: land #1911, close #1910 as covered, and let #1912 come back as a refactor that moves the rule into its own file after the five are registered. Deciding for #1911: the tool_selftest_ prefix makes the whole family greppable in one query (ctest -N | grep tool_selftest), it registers the fifth (spec_decode_acceptance) that #1910 does not, and unregistered_selftest_tools() is the part #1912 needs anyway — the detector as a function is testable without a second copy of the rule.

    The one thing I want kept from #1912 either way is its self-test's second half, which the other two lack: "the negative control: nothing anywhere names the tool, so its claim is unbacked" — a check that cannot fail is the defect this exists to find, and #1912 asserts both directions on a synthetic tree.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured on main, not read from the bodies: six tools declare a self-test and are registered nowhere — and the detector's rule misses two of them

    I checked CMakeLists.txt on main against each tools/*.py rather than taking the PRs' counts on trust, and the numbers differ from what any of the three PRs says.

    Declares a self-test AND has no ctest row on main (six):

    tool how it declares
    batched_repeat, logprob_agree, whisper_encoder_baseline, pin_watch add_argument("--self-test", …)
    template_bom not via add_argument — documented as [--self-test] in its usage block and dispatched from argv
    mtp_head_ref --selftest, no hyphen

    Already registered and needing nothing: npu_decl_check, split_ceiling, device_holders, hrx_build_scope, moe_placement, fa_ratio, ffn_sparsity, seal_manifest, spec_decode_acceptance.

    What each PR actually covers against that list:

    PR registers correct missing redundant
    #1910 batched_repeat, logprob_agree, mtp_head_ref, template_bom, whisper_encoder_baseline 5 pin_watch —
    #1911 batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance, whisper_encoder_baseline 4 pin_watch, template_bom spec_decode_acceptance (already registered)
    #1912 whisper_encoder_baseline (+ its own checker) 1 the other five —

    So I am correcting my own recommendation on the three PRs: #1910 is closest — it needs one registration added (pin_watch), where #1911 needs two and carries one that is already registered. My earlier preference for #1911 was based on reading the bodies and on the tool_selftest_ prefix; the prefix is still worth having, but a redundant registration plus two omissions is worse than the naming win.

    And the finding that matters more than the arithmetic: whichever detector lands must recognise all three spellings. A rule keyed on add_argument("--self-test") misses template_bom (usage-documented, argv-dispatched) and mtp_head_ref (--selftest) — so the guard would report "no unregistered declaring tools" while two unregistered ones sit in the tree. That is the same defect the sweep exists to fix, one level up: the check must read how the tools actually declare, not how the first one happened to.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured follow-up after #1911 merged. On current main: find_package(Python3) at :282, add_test at :290, zero rows before it, and the tool_selftest_* family present (batched_repeat, logprob_agree, mtp_head_ref, spec_decode_acceptance, whisper_encoder_baseline).

    Two tools are still unregistered — exactly the two I predicted #1911 would miss:

    tool declares via
    pin_watch add_argument("--self-test")
    template_bom usage-documented [--self-test], dispatched from argv

    So the remaining work on this issue is small and specific: register those two, and make the detector recognise all three spellings (--self-test via add_argument, --selftest, and the usage-documented/argv-dispatched form) — otherwise the sweep reports a clean tree with template_bom in it. #1912 is the PR shaped to do it; I asked there for those two registrations plus the spelling fix.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing-loop note: the fix landed (main's tests_referenced_check.py now reports "21 tool self-test(s) are registered"), and it closes this at a higher count than the issue measured — five tools, not two: batched_repeat, logprob_agree, mtp_head_ref, whisper_encoder_baseline now have CMake rows, template_bom is named by .github/workflows/ci.yml. The --self[-_]?test regex also absorbs the per-tool flag-spelling split (three of the five take --selftest), and tools.rglob covers tools/ci/*.py.

    One latent false positive in the merged rule, recorded rather than patched (I had a competing PR on that file and closed it rather than stack a third change): RUNNERS is ("CMakeLists.txt", "cmake", ".github/workflows") and tests/ is not in it. A tool whose only runner is a tests/*.py that imports it — exactly how tools/spec_decode_acceptance.py is exercised, via tests/spec_decode_acceptance_test.py (CMakeLists.txt:857) — would be reported as unrun unless something else happens to name it. It does not fire today. The two-line close: add tests to RUNNERS and match the stem as well as the filename (import spec_decode_acceptance names the stem).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions