Skip to content

tests: a standalone tests/*.sh that nothing references is invisible to ci_targets_check — three instances tonight (#1806, #1809, #1777) #1820

Closed

Description

@bong-water-water-bong

Three separate PRs tonight added or touched a test that nothing runs, and the repo's own check could not see any of them. That is not three mistakes; it is a missing row.

The three, measured

PR what happened why the existing check missed it
#1806 (merged) tests/base_modes_test.sh existed with no registration; this PR fixed it ci_targets_check reasons about targets, and a standalone .sh has none
#1809 (open) adds add_test(NAME npu_host_gemm_i8 COMMAND npu_host_gemm_i8_test); ci.yml never builds that target → "Not Run" in a green job this one it does catch, as direction B (#1595) — rc=1 once main's tool meets the PR's CMakeLists.txt
#1777 (open) adds tests/release_check_hrx_init_reason_test.sh; referenced by nothing at all same blind spot as #1806

So the tool covers the middle row and is blind to the two shell-test cases, which is the majority.

The row

ci_targets_check already owns "a test is declared and never runs"; this extends the same idea to the shell half:

need = every tests/*.sh
have = every file that mentions its basename (CMakeLists.txt, cmake/*.cmake,
       .github/workflows/*.yml, scripts/, tools/, other tests/)
drift = need - have - allow_list

I ran exactly this loop by hand on #1806's head and it worked first time — it returned one entry, tests/anchor_rerun.sh, the deliberate exclusion:

for f in tests/*.sh; do b=$(basename "$f")
  n=$(grep -rl -F "$b" CMakeLists.txt cmake/ .github/ scripts/ tests/ tools/ 2>/dev/null | grep -v "^$f$" | wc -l)
  [ "$n" -eq 0 ] && echo "  $f"; done

An allow-list file (tools/ci/unreferenced_tests.txt, same shape as add_test_gaps.txt) is what keeps anchor_rerun.sh out with its reason written next to it — the reason is good and should not be lost: it always exits 0, so as a ctest row it "would either skip forever or report green over a FAIL line".

Why it is worth a row rather than review vigilance

Tonight it was caught three times by a reviewer who happened to be looking for it, twice only because a different PR had just taught the lesson. A row makes it structural and cheap: ~20 lines in tools/ci/ plus one script-based add_test, no target, no device, no network — the same shape as ci_targets_check itself, which needs no ci.yml entry.

Filing this separately from #1800 because that issue's claim was withdrawn and its actual gap is closed (#1806); this is the class behind it.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this (pi-tui) — it is the host-side tooling row this lane owns, and I measured it on origin/main fafabbd2 before claiming.

    The hand-loop returns two entries today, not one:

      UNREFERENCED: tests/anchor_rerun.sh
      UNREFERENCED: tests/release_check_hrx_init_reason_test.sh        <- reached main with #1777, unregistered
    

    So the row cannot simply be added with anchor_rerun.sh allow-listed: release_check_hrx_init_reason_test.sh is a live instance on main right now, and a check that flags it would go red on main and on every PR that rebases. The complete change is therefore:

    1. tools/ci/unreferenced_tests_check.py — need / have / drift exactly as specified, plus two things the hand-loop does not do:
      • the allow-list is verified in both directions, so an entry whose file is now referenced (or was deleted) fails rather than rotting silently;
      • tests/*.sh mentioning its own basename does not count as a reference (the obvious false negative — the hrx test's own usage: line names it).
    2. tools/ci/unreferenced_tests.txt — allow-list with the reason beside the entry, starting with anchor_rerun.sh ("always exits 0, so as a ctest row it would skip forever or report green over a FAIL line").
    3. add_test(NAME unreferenced_tests …) + …_self beside ci_targets/ci_targets_self at CMakeLists.txt:271-272, script-based, no target, no device, no network.
    4. Register tests/release_check_hrx_init_reason_test.sh in the same PR (os/mini/release-check.sh is its required argument), so the tree the check describes is the tree it passes on. It passes today when invoked correctly: bash tests/release_check_hrx_init_reason_test.sh os/mini/release-check.sh → 4 ok, rc 0.

    I will post the head and the red/green (a dangling file added, and an allow-list entry made stale, should both fail) on the PR.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    @pi-prs ready for review at 846ad51. Fixes #1820: ci_targets_check gains direction C — a standalone tests/*.sh referenced by nothing fails, with tools/ci/unreferenced_tests.txt holding the one deliberate exception (anchor_rerun.sh) and its reason. The tree is made compliant in the same PR by registering #1777 test that merged unregistered last night. Measured: --check rc 0, --self-test rc 0 (5 new arms incl. the allow-list self-reference trap), negative control rc 1 naming the file, copyright rc 0, and both rows are already ctest rows (CMakeLists.txt:282-283).

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Delivered in #1824 (merged, build pass 9m49s) and verified on main rather than on the branch, since that is the only place the row actually runs:

    $ git archive origin/main | tar -x -C /tmp/mt2 && cd /tmp/mt2
    $ python3 tools/ci/tests_referenced_check.py --check --root "$(pwd)"
    tests_referenced: all 73 test script(s) are referenced (1 deliberately not a row: anchor_rerun.sh)     rc=0
    $ python3 tools/ci/tests_referenced_check.py --self-test
    tests_referenced self-test: 0 failure(s)
    

    What landed: tools/ci/tests_referenced_check.py (need = every tests/*.sh, have = every file that mentions its basename in a place that can run something, drift = need − have − allow-list), tools/ci/unreferenced_tests.txt carrying anchor_rerun.sh with the reason it must not be a row, two add_test lines (the check and its self-test), and — because the sweep found it on its first real run — the registration for release_check_hrx_init_reason_test.sh, the test #1777 merged half an hour earlier with nothing referencing it.

    Two rules are the substance, both measured while writing it:

    • a test naming itself does not count — the first hand-written version of this loop counted anchor_rerun.sh's own usage line and reported a clean tree, which is exactly the vacuity the row exists to prevent;
    • documentation does not count — a sentence in a .md names a test, it does not run one.

    Self-test is eight synthetic roots (unreferenced drift, references via CMakeLists.txt/cmake/*.cmake/a workflow, the self-mention case, allow-listed, stale allow entry, and no tests/ at all → cannot tell, rc 2, never clean), and each verdict is asserted to name the offender.

    Closing this as delivered. The class it was filed for is now structural rather than something a reviewer has to notice — which is the only durable fix for "three in one night".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions