Skip to content

tests_referenced: a row that names a script which is not in the tree is drift - #2346

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/row-path
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/row-path

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

engine#2183 shipped a row whose script a rebase had dropped. Its commit message reads "the lv3 row script my rebase pattern missed — the row was Not Run", and every gate was green on it:

mv tests/whisper_lv3_row.sh tests/whisper_lv3_row.sh.bak
  tests_referenced_check --check      rc 0
  tests_referenced_check HERMETIC=1   rc 0
  ci_targets_check --check            rc 0

referenced() is satisfied by the mention in CMakeLists.txt, and ci_targets_check follows $<TARGET_FILE:…> targets rather than plain paths — so a row can name a file that is not in the tree, report Not Run for ever, and every gate stays green. That is the same defect as an unreferenced test, one step further along.

missing_mentions() asks the inverse question — every tests/*.sh a RUNNER names must exist — and joins the same verdict. rc 0 on main, rc 1 on the plant (renaming tests/amdxdna_restart_test.sh gives one finding that names the row).

The false positive that shaped it: the first regex matched the tail, so ${CMAKE_SOURCE_DIR}/tools/tests/test-bench-preflight.sh read as tests/test-bench-preflight.sh and reported two existing files as missing. tools/tests/x.sh is a different file from tests/x.sh; the pattern now takes the full path after the variable, and an arm pins exactly that.

Three arms added; dropping missing_mentions from the verdict fails one.

tests_referenced .........  Passed  2.19 s
tests_referenced_self ....  Passed  0.08 s
copyright --check rc 0

…ich is not in the tree

engine#2183 shipped a row whose script a rebase had dropped. Its commit message reads *"the lv3 row script my
rebase pattern missed -- the row was Not Run"*, and every gate was green on it:

    mv tests/whisper_lv3_row.sh tests/whisper_lv3_row.sh.bak
      tests_referenced_check --check      rc 0
      tests_referenced_check HERMETIC=1   rc 0
      ci_targets_check --check            rc 0

`referenced()` is satisfied by the *mention* in CMakeLists.txt, and `ci_targets_check` follows `$<TARGET_FILE:...>`
targets rather than plain paths, so a row can name a file that is not there, report **Not Run** for ever, and every
gate stays green. That is the same defect as an unreferenced test one step further along: a row that cannot fail.

`missing_mentions()` asks the inverse question -- every `tests/*.sh` a RUNNER names must exist -- and joins the same
verdict. On the real tree it is **rc 0** (main has no stale mention), and planting one is rc 1:

    renamed tests/amdxdna_restart_test.sh   -> rc 1, one finding naming the row

**The false positive that shaped it:** the first regex matched the tail, so `${CMAKE_SOURCE_DIR}/tools/tests/
test-bench-preflight.sh` read as `tests/test-bench-preflight.sh` and reported **two existing files as missing**.
`tools/tests/x.sh` is a different file from `tests/x.sh`; the pattern now takes the full path after the variable,
and an arm pins exactly that.

Three arms added; dropping `missing_mentions` from the verdict fails one. Self-test ok, ctest `tests_referenced`
0.78 s and `tests_referenced_self` 0.04 s, copyright --check rc 0, ci_targets_check --check rc 0.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect regex for path matching

The regex pattern MENTION may not correctly handle all edge cases of path references, particularly when dealing with nested directories or complex path structures. This could lead to false negatives where valid script paths are not detected.

MENTION = re.compile(r"\$\{CMAKE_SOURCE_DIR\}/([A-Za-z0-9_./+-]+\.sh)|(?<![\w/.-])(tests/[A-Za-z0-9_.+-]+\.sh)")
Potential performance issue with repeated calls

The function missing_mentions is called twice in the check function: once for printing and once for the condition check. This could lead to redundant computation and performance degradation, especially if the function is expensive to execute.

for runner, mentioned in missing_mentions(root):
    print("tests_referenced: %s names %s, which is not in the tree -- the row exists and can only report "
          "Not Run" % (runner, mentioned))
tools_drift = unregistered_selftest_tools(root)
for rel in tools_drift:
    print("tests_referenced: %s declares a self-test and no runner names it -- add a ctest row (add_test in "
          "CMakeLists.txt or cmake/*.cmake) or a workflow step, or the arm is dead code" % rel)
if drift or gone or idle or py_drift or tools_drift or missing_mentions(root):
    return 1

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 4337c2e — a row whose script a rebase dropped, with every gate green, and the diagnosis is that two checks were each satisfied by the mention rather than the file.

The case

mv tests/whisper_lv3_row.sh tests/whisper_lv3_row.sh.bak
  tests_referenced_check --check      rc 0
  tests_referenced_check HERMETIC=1   rc 0
  ci_targets_check --check            rc 0

and the commit message is the confession: "the lv3 row script my rebase pattern missed — the row was Not Run". HERMETIC=1 passing too is the part that should not be possible if the hermetic arm reads the filesystem — worth checking that the arm's subject is the tree and not the registry.

The mechanism, which is the same shape twice

"referenced() is satisfied by the mention in CMakeLists.txt, and ci_targets_check follows $<TARGET_FILE:…> ta[rgets]"

So: one check verified that a name appears in a registration file, the other followed a target expression, and neither asked "does the file this row names exist?". A mv to .bak satisfies all three and runs nothing — and the failure surfaces as a test that is silently Not Run, which is the worst outcome for a test: not red, not green, just absent from the count.

That is #2238 ("a self-test that no runner ran") arriving through a rename rather than a missing registration, and it is the fourth variant of the class in this session (#2238, #2256's --help, #2302's zz_probe_target, this).

Two asks

  1. make "the row's script exists" its own check — one stat per row, in tests_referenced_check, with the row name and the missing path in the message. It is the cheapest possible assertion and it closes the rename case;
  2. state what HERMETIC=1 actually changes — because if the hermetic arm is supposed to read the tree rather than the registry, it should have failed here, and its passing is either a second defect or a name that promises more than it does. One sentence.

And the PR's provenance is the best kind

"engine#2183 shipped a row whose script a rebase had dropped" — a defect found in a real PR by a lane that noticed a test was "Not Run" in a log. Two checkers, three green ticks, and one absent test; that is exactly the class this session has been collecting, and the fix is to make the check read the filesystem rather than the paperwork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant