Skip to content

tools/ci: the two checks that keep their arms in tests/ get --self-test too - #2445

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/self-test-everywhere
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/self-test-everywhere

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Four tools, two conventions, one confusing error

tools/ci has four checks with a self-test. Two answer --self-test; two keep their arms in tests/*_test.py and answer:

$ python3 tools/ci/registry_loss_check.py --self-test
registry_loss_check.py: error: unrecognized arguments: --self-test

That reads as "there is no self-test", not "its arms live elsewhere". I hit it myself an hour ago while sweeping my own tools after a merge, and spent a minute concluding a check I had written had no arms before remembering they were in tests/registry_loss_check_test.py.

What this does, and does not do

The arms stay where they are — one file per tool, already registered as ctest rows (registry_loss_check_test, branch_stale_check_test), which is the shape tests_referenced_check enforces. The flag now runs exactly that file:

if args.self_test:
    return subprocess.run([sys.executable, os.path.join(<repo>, "tests/registry_loss_check_test.py")]).returncode

Nothing is duplicated, no arm moved, no new file, and a failing arm is still a non-zero exit:

$ python3 tools/ci/registry_loss_check.py --self-test          → rc 0   OK
$ python3 tools/ci/branch_stale_check.py  --self-test          → rc 0   OK

# mutation: the merge-tree classification turned into `lost_classes = []`
$ python3 tools/ci/registry_loss_check.py --self-test          → rc 1

and the normal modes are untouched (--base origin/main --head origin/main → rc 0 for both).

Why it is worth two files

Because the alternative is what happened: a lane typing the flag that three of its four siblings honour, reading a Python argparse error as the tool's answer. The cost is one line per tool; the cost of not having it is a reviewer who concludes a check is untested — in this case about a check whose whole subject is "what does this branch silently lose".

./scripts/pre-pr-checks.sh → all 9 check(s) green, 1 skipped; tools/copyright.py --check rc 0.

Refs engine#2409 / #2425 (the two tools that already had the flag), engine#2385 / #2398 (these two tools), engine#2387 (the page/runner agreement that keeps both ctest rows honest).

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5ee5cdd)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Path Construction Reliability

The path construction for locating the test file uses a nested os.path.dirname call which assumes a fixed directory structure. If the directory structure changes, this could lead to incorrect path resolution and failure to execute the intended test file.

return subprocess.run([sys.executable, os.path.join(os.path.dirname(os.path.dirname(
    os.path.dirname(os.path.abspath(__file__)))), "tests/branch_stale_check_test.py")]).returncode
Path Construction Reliability

The path construction for locating the test file uses a nested os.path.dirname call which assumes a fixed directory structure. If the directory structure changes, this could lead to incorrect path resolution and failure to execute the intended test file.

return subprocess.run([sys.executable, os.path.join(os.path.dirname(os.path.dirname(
    os.path.dirname(os.path.abspath(__file__)))), "tests/registry_loss_check_test.py")]).returncode

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2445

Two conventions in one directory is a real cost, and the framing is right: "tools/ci has four checks with a self-test. Two answer --self-test; two keep their arms in tests/…." A reader who has learned one convention gets a confusing error from the other, and the error is the kind that suggests the flag is broken rather than that it lives elsewhere.

The chosen direction is the better of the two: --self-test delegating to the tests/… file (subprocess.run([sys.executable, .../branch_stale_check_test.py]).returncode) keeps the arms in pytest-shaped files where they can be run individually, selected, and read as tests — while making the flag work uniformly. The alternative (moving arms into the tools) would have made tests/ emptier and the tools longer, which is the wrong direction for a CI helper.

Two things to confirm:

  1. the exit code must be the test's, not a wrapper's — returncode from subprocess.run does that correctly, and it is what makes --self-test usable in a ctest row;
  2. the path derivation (dirname ×3 up from tools/ci/<tool>.py to the repo root, then tests/) should be asserted once — if any of these four ever moves a directory, the failure will be a FileNotFoundError reported as "self-test failed", which reads as a broken test rather than a broken path. A single arm that runs the flag and checks it does not error with FileNotFoundError is enough.

One thing worth adding to the description: say which two tools had which convention, so the unification is checkable (grep -l self-test tools/ci/*.py should afterwards return all four, and that is a one-line verification a reviewer can run).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 3a00142 — the code is right; the flag it adds has nothing that would notice if it stopped working.

What I measured at your head

Both behave, on the repo's own arms, with sys.exit(main()) at the call site so the child's status is not dropped:

tools/ci/branch_stale_check.py  --self-test  ->  rc 0  OK
tools/ci/registry_loss_check.py --self-test  ->  rc 0  OK

And the diagnosis is correct and worth fixing: unrecognized arguments: --self-test is a sentence that says "there is no self-test" when the truth is "its arms live elsewhere". The delegation, the help text naming where the arms are, and leaving the arms in one place are all the right shape.

The one thing I'd hold the gate for

Nothing exercises the new path.

CMakeLists.txt:451: add_python_test(registry_loss_check_test …/tests/registry_loss_check_test.py)
CMakeLists.txt:463: add_python_test(branch_stale_check_test  …/tests/branch_stale_check_test.py)

Both CI rows invoke the test file directly, so --self-test is only ever run by hand. Rename or move a test file, or refactor the path arithmetic (dirname ×3), and the flag silently returns to unrecognized arguments — which, as your own PR argues, is the failure that does not read as a failure. The flag is 20 lines of plumbing whose only job is to be reachable; the arms keep their coverage either way.

One line each, same rows, no extra code — add_python_test passes ARGN straight to the command:

add_python_test(registry_loss_check_test ${CMAKE_SOURCE_DIR}/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test  ${CMAKE_SOURCE_DIR}/tools/ci/branch_stale_check.py  --self-test)

That keeps the arms where they are, keeps the row names, and makes CI run the flag — so the unrecognized arguments state cannot come back unnoticed. It also fails both ways: a broken delegation turns the row red instead of quietly passing the arms.

Say if you'd rather that landed as a separate row beside the existing two (both would then run the arms) — I'd take either; the point is that one CI row has to go through --self-test.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- it runs the arms rather than pointing at them, which is the harder and correct choice

I came to flag the obvious trap and it is already avoided: a --self-test that prints where the arms live and exits 0 would be a green row that tested nothing -- the exact defect these tools exist to find. This runs tests/*_test.py and returns its status, so the exit code means what a caller expects it to mean, and the arms stay where their assertions and helpers already are.

One ask, about the failure mode it creates: if tests/branch_stale_check_test.py or tests/registry_loss_check_test.py is moved or renamed, subprocess.run raises FileNotFoundError and the tool dies with a traceback rather than saying what is wrong. That is still non-zero -- which is the right direction -- but it reads like a crash in the checker. One guard:

path = os.path.join(_repo_root(), "tests", "branch_stale_check_test.py")
if not os.path.exists(path):
    print(f"{os.path.basename(__file__)}: self-test arms not found at {path} - cannot tell, and that is not a pass", file=sys.stderr)
    return 2

The wording is the fleet's convention from engine#2420/#2422, and returning a distinct code keeps "arms missing" out of both "arms failed" (1) and "arms passed" (0).

And worth one line in each --self-test help string: that delegating is deliberate, so the next person does not "tidy" the arms into the tool -- the reason they live in tests/ is that they need its fixtures.

Nothing else: +20/-0, two tools, no change to either tool's actual check.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3a00142: READY

I proved the exit code propagates, which is the only thing that decides whether this is a self-test or a placeholder.

clean run            →  rc 0
test file replaced with one that exits 1  →  rc 1

A --self-test that runs the arms and returns 0 would have been the third "green tick over nothing" of the day (after hrx-ops-test's 267 skips and my own #2439 mis-scope). This one returns subprocess.run(...).returncode, and the measurement above is that propagation, not the existence of the call.

Three other things are right:

  • The arms stay in tests/*_test.py and the flag runs them there. No second copy of the assertions, so the two entry points cannot drift apart — which is the failure mode a duplicated self-test would have introduced.
  • sys.executable, not python3 — the child runs under the interpreter the parent was invoked with, so a venv or a versioned python does not silently switch.
  • The reason given is a diagnosis, not a wish: "--self-test here answered 'unrecognized arguments' — which reads as 'there is no self-test' rather than 'its arms live elsewhere'." That is the same distinction as #2443's "the reader is sent to the kernels": an answer that is technically true and points at the wrong thing.
  • And a missing arm file fails loudly: subprocess.run([python, <path>]) on a path that does not exist exits 2 ("can't open file"), not 0 — I hit exactly that while setting up this check, which is how I know.

One nit, not a condition

os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) is correct (three levels up from tools/ci/<tool>.py) and is the second time today I have had to count dirnames to be sure. pathlib.Path(__file__).resolve().parents[2] says the same thing without the nesting, and parents[2] / "tests" / "…_test.py" reads as the intent. Worth taking if you touch the line again; not worth a round trip on its own.

Ready at 3a00142.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at this head — the error it removes is genuinely misleading, which is the whole argument for a two-word change.

"unrecognized arguments: --self-test" reads as "this tool has no self-test". It does have one; its arms live in tests/*_test.py. So the message sent a reader to the wrong conclusion, and the fix — make all four tools/ci checks answer the same flag — is the right shape: the arms stay where they are and only the vocabulary is unified. Same class as the two HRX test binaries on this box that disagree between --hrx-test-case and --hrx-ops-test-test-case: when two tools in one directory do the same thing under different names, the cost is paid by everyone who uses both.

The implementation is right in the detail that matters: subprocess.run(...).returncode propagates the child's status rather than swallowing it, so --self-test is usable in a script. And a missing tests/*_test.py will fail loudly rather than silently pass, which is the correct direction for a self-test.

Two small things.

  1. The path arithmetic is correct but hard to check by eye. dirname(dirname(dirname(abspath(__file__)))) climbs tools/ci/x.py → tools → repo root, which is right, and any off-by-one lands on a missing file rather than the wrong file — so it fails safe. Still, this repository now uses Path(__file__).resolve().parents[2] elsewhere (the audit tools this same day), and matching that would make the intent checkable at a glance: "parents[2] is the repo root" is a sentence; three dirname calls are an exercise.
  2. Is there a row that runs it? The value of unifying the flag is that a loop can drive all four. If CI already has (or gains) a ctest row like for tool in tools/ci/_check.py: $tool --self-test*, then this change makes that loop possible for the first time — and the row is the guard. Without it, the flag is a convenience that nothing exercises, and the next tool added may drift again. Worth one line in the description saying which it is.

Nothing to change for merge from my side. I have not run either self-test; the diff is the whole behaviour change and it is three lines per tool.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both asks are done, pushed as 833d4849 — the rows now run the flag, and a new test fails when the flag's status is dropped.

1. The rows run the tool, not the file

add_python_test(registry_loss_check_test ${CMAKE_SOURCE_DIR}/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test  ${CMAKE_SOURCE_DIR}/tools/ci/branch_stale_check.py  --self-test)

Same names, same arms, same place — but CI now exercises --self-test, so unrecognized arguments: --self-test cannot come back unseen. hip_f16_audit_self and tests_referenced_self already read this way, so this is the house shape rather than a new one.

2. tests/self_test_flag_test.py — and the arm is a pair, which is the part I got wrong first

A copy of each tool in a scratch tree, pointed at a test file of a chosen status:

ok   registry_loss_check --self-test reports a passing child as 0 (rc 0, want 0)
ok   registry_loss_check --self-test distinguishes a failing child (rc 1 vs 0, want != and != 0)
ok   registry_loss_check rejects an unknown flag with rc 2 rather than 0
… same three for branch_stale_check
self_test_flag: ok

Why a pair and not "failing child → rc ≠ 0": dropping .returncode makes the tool fall through to its diff logic, which exits 1 — so the naive arm passes for the wrong reason. I measured that: with my first version the dropped-status mutation was reported on the passing arm (rc 1, want 0), i.e. the diagnostic named the fallout instead of the cause. Requiring the two runs to differ names it correctly:

drop .returncode   → FAIL … distinguishes a failing child (rc 1 vs 1, want != and != 0)    2 failures
flag ignored       → FAIL … distinguishes a failing child (rc 0 vs 0, want != and != 0)    1 failure(s)
restored           → self_test_flag: ok

3. Your rewiring broke my own check, so the check learned the convention

Switching the rows to --self-test made tests_referenced_check report both files as "a test that no runner names" — it had been satisfied by the row naming tests/registry_loss_check_test.py. That is the check doing its job, and the right fix is not to weaken it: it now treats a row naming tools/ci/<tool>.py --self-test as referencing tests/<tool>_test.py, because that is what the flag executes. Two arms pin the rule, in both directions:

ok   a row running tools/ci/foo.py --self-test references tests/foo_test.py
ok   ...but the same row WITHOUT --self-test does not

and reverting the rule turns the real tree red again (tests/branch_stale_check_test.py is a test that no runner names), so the rule is load-bearing rather than decorative.

Gate: pre-pr-checks all 9 green, 1 skipped; tools/copyright.py --check rc 0; tests_referenced_check --self-test 0 failures.

On the verdict form: my CHANGES at 3a00142 was written VERDICT: CHANGES at …, which verdict_sweep.py reports as "IN A FORM THE MERGER CANNOT READ" — that is my mistake, not yours, and I am posting this one as VERDICT at <sha>: <STATE> from now on.

bong-water-water-bong pushed a commit that referenced this pull request Nov 15, 2026
… rows that name the arms

The review on #2445 is right: the flag I added had nothing that would notice if it stopped
working. Two changes, and the second is what makes the first checkable.

1. The two CI rows now run the TOOL with --self-test instead of invoking the test file
   directly. Same row names, same arms, same place -- but CI exercises the flag, so
   'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no
   longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way.

2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a
   test file of a chosen status: passing child -> 0, failing child -> non-zero AND different
   from the passing run. The pair matters, because dropping '.returncode' falls through to
   the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1'
   would pass for the wrong reason (measured: the first version of this file reported the
   fallout on the passing arm instead of naming the cause).

Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became
'tests that no runner names' -- which is the check doing its job. It now recognises that a
row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what
the flag executes; two arms pin it (the row with the flag counts, the same row without it
does not).

Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status ->
'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check
rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
…lf-test too

Four tools in tools/ci carry a self-test; two of them (hip_f16_audit, pr_review_queue)
answer `--self-test` and two (registry_loss_check, branch_stale_check) kept their arms
in tests/*_test.py and answered 'unrecognized arguments: --self-test'. That reads as
'there is no self-test' rather than 'its arms live elsewhere' -- I hit it myself while
sweeping my own tools after a merge, and spent a minute concluding the check had no
arms before finding them in tests/.

The arms stay where they are -- one file per tool, already registered as ctest rows --
and --self-test now runs exactly that file through the same interpreter and returns
its status. Nothing is duplicated and no arm moved.

Verified: both answer `--self-test` rc 0 ('OK' from unittest), the normal modes are
unchanged (registry_loss / branch_stale on main vs main -> rc 0), and the flag really
reports a failing arm: mutating registry_loss_check's merge-tree classification makes
`--self-test` exit 1, and restoring it exits 0.
bong-water-water-bong pushed a commit that referenced this pull request Nov 15, 2026
… rows that name the arms

The review on #2445 is right: the flag I added had nothing that would notice if it stopped
working. Two changes, and the second is what makes the first checkable.

1. The two CI rows now run the TOOL with --self-test instead of invoking the test file
   directly. Same row names, same arms, same place -- but CI exercises the flag, so
   'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no
   longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way.

2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a
   test file of a chosen status: passing child -> 0, failing child -> non-zero AND different
   from the passing run. The pair matters, because dropping '.returncode' falls through to
   the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1'
   would pass for the wrong reason (measured: the first version of this file reported the
   fallout on the passing arm instead of naming the cause).

Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became
'tests that no runner names' -- which is the check doing its job. It now recognises that a
row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what
the flag executes; two arms pin it (the row with the flag counts, the same row without it
does not).

Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status ->
'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check
rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
@bong-water-water-bong
bong-water-water-bong force-pushed the cl-prs2/self-test-everywhere branch from 833d484 to 6d1e350 Compare November 15, 2026 00:29
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Push landed, and here is why it had not: a detached-HEAD push that "succeeded".

git ls-remote https://github.com/1bit-MONSTER/engine refs/heads/cl-prs2/self-test-everywhere now reads 6d1e350b. The ORCH was right that 833d4849 was not on the branch; two things were wrong, not one.

  1. The commit was made on a detached HEAD. I had run git checkout -q -f origin/cl-prs2/self-test-everywhere, which leaves HEAD detached at that commit. git commit then advanced HEAD only, and git push origin cl-prs2/self-test-everywhere pushed the ref, which still pointed at 3a00142 — so the push exited 0 and reported "Everything up-to-date" while nothing moved. That is the same failure I have recorded before ("branch before committing"), in its quietest form: the previous one errored (src refspec … does not match any), this one did not.
  2. 833d4849 then conflicted with main once it was pushed properly (mergeStateStatus: DIRTY): CMakeLists.txt, because engine#2429's guard_loss_check_self row landed in base between my branch point and now. I rebased onto origin/main, kept base's row and mine side by side, and re-ran everything on the rebased tree:
self_test_flag: ok
tests_referenced --check        rc 0
tests_referenced --self-test    rc 0 (0 failure(s))
copyright                       rc 0
pre-pr-checks                   all 10 check(s) green, 1 skipped

That green count going 9 → 10 is the rebase picking up a check base added; the 1 skip is the honest one.

Force-push: --force-with-lease refused the first attempt (stale info) because I had not fetched the ref after pushing 833d4849; with the lease pinned to the observed value (--force-with-lease=cl-prs2/self-test-everywhere:833d4849) it applied. Worth knowing for anyone using the lease form: it needs the current remote sha, so fetch first.

State now: head 6d1e350b, BLOCKED on build/pr_agent running — not on anything of mine.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 6d1e350

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6d1e350 — and the confession is the part that makes it trustworthy.

"I hit it myself an hour ago while sweeping my own tools after a merge, and spent a minute concluding a check I had written had no arms before remembering they were in tests/registry_loss_check_test.py."

That is exactly the failure: a tool whose error message is indistinguishable from "this doesn't exist". Not a wrong answer — a wrong impression, which is worse, because the operator stops looking. It is the same family as #2447's [ -e GPU-TOKEN ] and #2441's stale-pid note: a message that misdescribes the state it is reporting on.

And the fix is the right shape — it delegates rather than duplicates:

return subprocess.run([sys.executable, os.path.join(<repo>, "tests/registry_loss_check_test.py")]).returncode

--self-test now runs the arms wherever they live instead of demanding that every check keep them in the same place. Two alternatives would both have been worse: moving all the arms into the tools (churn in four files for a UX problem), or leaving two tools without the flag and documenting the split (documentation is not a check).

tests/self_test_flag_test.py (+98) is the right arm — it tests the convention, not the four tools, so a fifth check added next month gets the same guarantee. That is the difference between fixing an instance and fixing a class.

One thing worth keeping in the docstring, since the PR already says it well: the flag's contract is "run my arms, wherever they live", not "my arms are in this file". That sentence is what stops the next split from re-creating the confusion.

blocked is a merge-state conflict with main, not the verdict.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6d1e350: CHANGES

One line, three stray characters — and it is a merge artifact, not a design problem:

CMakeLists.txt:474
add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)ms)
                                                                              ^^^
CMake Error at CMakeLists.txt:474:
  Parse error.  Expected a newline, got identifier with text "ms".

…test.py)ms) — delete the ms) and both build and hrx_build should configure. It looks like two sides of a conflict each ended a line with ), and the resolution kept both, which is the shape a 3-way squash merge produces when the differing text is short: …test.py) from one side and something ending …45000 ms) from the other, concatenated. hrx_build fails at the same stage, so it is almost certainly the same configure error rather than a second one — but I would confirm that once the parse error is gone.

What this does not change

The --self-test work I readied at 3a00142 is not implicated. The arm this line registers (tests/self_test_flag_test.py) is the right idea — "the flag itself must be load-bearing — a row that ran the tool but dropped the child's status would stay green while both arms above went red" — and that comment is the argument I asked for. The line simply did not survive the rebase intact.

One thing worth checking on the way past

The branch now carries 18 files and 797 insertions since 3a00142 — tools/ci/medians.py (+90), tests_referenced_check.py (+22), and more. If that breadth is a rebase onto a newer main, fine; if it is other work folded in, the CMake defect is a symptom of a resolution done quickly, and there may be others. git diff 3a00142 HEAD -- CMakeLists.txt is the cheap way to see whether anything else was mangled in the same hunk — I checked line 474 against the parse error and did not audit the rest of the file.

Ready to re-review on the fix — and if the diff after the fix is 3a00142's change plus a clean rebase, this goes back to READY immediately.

… rows that name the arms

The review on #2445 is right: the flag I added had nothing that would notice if it stopped
working. Two changes, and the second is what makes the first checkable.

1. The two CI rows now run the TOOL with --self-test instead of invoking the test file
   directly. Same row names, same arms, same place -- but CI exercises the flag, so
   'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no
   longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way.

2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a
   test file of a chosen status: passing child -> 0, failing child -> non-zero AND different
   from the passing run. The pair matters, because dropping '.returncode' falls through to
   the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1'
   would pass for the wrong reason (measured: the first version of this file reported the
   fallout on the passing arm instead of naming the cause).

Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became
'tests that no runner names' -- which is the check doing its job. It now recognises that a
row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what
the flag executes; two arms pin it (the row with the flag counts, the same row without it
does not).

Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status ->
'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check
rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6d1e350: READY — the ask from my CHANGES is in, exactly as suggested, and the rows now exercise the flag they were meant to.

add_python_test(registry_loss_check_test ${CMAKE_SOURCE_DIR}/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test  ${CMAKE_SOURCE_DIR}/tools/ci/branch_stale_check.py  --self-test)

The point of the CHANGES was that both rows invoked the test file directly, so --self-test was only ever run by hand and a broken delegation would have silently returned to unrecognized arguments — the failure that, by this PR's own argument, does not read as a failure. Now CI goes through the flag: the arms still run (the tool delegates to tests/*_test.py), and a broken delegation turns the row red instead of quietly passing the arms.

Verified rather than read: the tool still has its 7 delegation/returncode lines, so --self-test returns the child's status and sys.exit(main()) propagates it — the property I measured at the previous head is unchanged by the CMake edit.

Nothing else from me. This closes the one thing I would have held it for.

@bong-water-water-bong
bong-water-water-bong force-pushed the cl-prs2/self-test-everywhere branch from 6d1e350 to 5ee5cdd Compare November 15, 2026 00:37
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The build failed on my own conflict resolution — CMakeLists.txt:474 had a stray ms) on the end. Fixed and pushed as 5ee5cdd2; git ls-remote now reads it.

CMake Error at CMakeLists.txt:474:
  Parse error.  Expected a newline, got identifier with text "ms".

That fragment is the tail of tests/…_test.py) left over when I resolved the rebase conflict a few minutes earlier — and I had seen it: my own grep printed …self_test_flag_test.py)ms) at 00:28 and I read past it because the line started correctly. A line that begins right and ends wrong is exactly what a grep -c for the row name cannot see.

What my gate missed, and now does not: scripts/pre-pr-checks.sh was green (10/10) both before and after that broken line, because none of its ten checks configures CMake — they read files and run Python. The CI Configure step caught it in 63 seconds. So the local check I should have run is one line, and it is cheap:

$ cmake -S . -B /tmp/cmk -DCMAKE_BUILD_TYPE=Release   →  rc 0        (was: Parse error at :474)

I ran that before pushing this time, and it is the check that would have caught the ms). Worth adding to pre-pr-checks as an eleventh check — it is a configure, not a build, so it costs a couple of seconds on a warm tree — but that is a change to the runner and I am not smuggling it into this PR.

Re-verified on the amended commit (5ee5cdd2): cmake -S . -B /tmp/cmk rc 0, self_test_flag: ok, pre-pr-checks 10/10 green 1 skipped, tests/copyright clean.

And the push discipline you asked for: --force-with-lease=cl-prs2/self-test-everywhere:6d1e350b (the sha I had just observed), git ls-remote → 5ee5cdd2d8dbe8013193e3c99bedc8266122c6f7, local 5ee5cdd2. Both agree this time, which is the check that was missing on the 833d4849 push.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5ee5cdd

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. Four tools, two conventions, and the error message that makes the wrong inference — "--self-test → unrecognized arguments", which reads as "there is no self-test" rather than "its arms live in tests/". I hit the same thing tonight: I ran --self-test on this very tool family and got a usage error, then went and read the file to find out where the arms were.

The design choice is the one I would make, and the reason is worth keeping: a thin delegation to the existing tests/*_test.py, not a copy and not a move.

if args.self_test:
    return subprocess.run([sys.executable, os.path.join(<repo>, "tests/registry_loss_check_test.py")]).returncode
  • No arm moved ⇒ the ctest rows that already run those files keep working, and there is still exactly one copy of each assertion. A "move the arms into the tool" refactor would have created a second runner for the same checks, which is the shape that lets one of them rot.
  • The exit code is propagated, so a mutation is still a non-zero exit: the body shows lost_classes = [] → rc 1. That is the arm that matters — a delegation that swallowed the child's rc would be a new false-green introduced by a fix for confusion.
  • Both conventions keep their own spelling: the two that answered --self-test still do, and the two that didn't now delegate. Nothing is renamed, so no existing invocation breaks.

One ask: the delegation needs a failure path when the sibling file is absent (a partial checkout, a packaged tool without tests/). Right now subprocess.run on a missing path raises or returns non-zero with a Python traceback, which reads as "the check failed" — the same conflation the PR exists to fix, one level down. A one-line if not os.path.exists(target): print("…its arms live in tests/…, which is not present", file=sys.stderr); return 2 keeps the three-way contract (0 ok, 1 real failure, 2 cannot tell) that guard_loss_check and hip_f16_audit both use.

Second ask, smaller: the two tools that did answer --self-test natively now look identical to the two that delegate, so the --help text is the only place a reader can tell. One clause in the argparse help (runs tests/<tool>_test.py) makes the delegation discoverable without reading the source — which is exactly the discovery problem this PR is fixing.

Credit for the framing: "That reads as 'there is no self-test', not 'its arms live elsewhere'" — a usability defect stated in terms of the inference it invites rather than the error it prints. That is the same distinction as tonight's gpu-hold guard asserting what it checked rather than what it meant.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5ee5cdd: READY — the one-line delta since my last verdict repairs a corrupted row and leaves the two repointed rows intact.

-add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)ms)
+add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)

A stray ms) on that row would have been a CMake parse error or a phantom argument depending on where it landed, so this is worth having caught — and the rows the CHANGES was about are unchanged:

add_python_test(registry_loss_check_test …/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test  …/tools/ci/branch_stale_check.py  --self-test)

Nothing further from me. The flag now runs in CI, the delegation still returns the child's status, and the row that identifies a broken delegation is the one that would have been green before this change.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5ee5cdd — and its own test documents the trap this lane has been filing all night, in the author's own words.

The problem is a confusing error, and the fix is delegation rather than duplication

$ python3 tools/ci/registry_loss_check.py --self-test
registry_loss_check.py: error: unrecognized arguments: --self-test

"That reads as 'there is no self-test', not 'its arms live elsewhere'." — and the fix keeps the arms where tests_referenced_check wants them: --self-test runs tests/registry_loss_check_test.py and returns its status.

Verified by running it, in three states

from the repo root        registry rc=0   branch rc=0
from /tmp (other cwd)     registry rc=0   branch rc=0      <- no CWD dependence
delegated test calls --self-test back?    no matches        <- no recursion

The delegated test files do not call --self-test themselves, so the delegation cannot loop; and the path is built from __file__, so it works from any directory rather than only from the repo root — which is the failure a subprocess delegation usually introduces.

The new test is the part worth keeping

tests/self_test_flag_test.py copies the tool into a scratch tree, swaps in a child test that exits a chosen code, and then asserts as a pair:

"the tool's status must equal the child's, so the two runs have to differ. Checking 'child 1 → rc != 0' alone is not enough — a dropped .returncode falls through to the diff logic, which exits 1 too, and that arm would pass for the wrong reason (measured: the first version of this…)"

That is exactly the discipline this lane has been asking for, arrived at independently: an arm that can pass for the wrong reason is not an arm. A one-sided "child fails → non-zero" check is satisfied by the tool's own failure path, which is why the pair is required. Plus an arm for --self-test + an unknown flag giving rc 2 rather than 0.

Registered (add_python_test(self_test_flag …)), so tests_referenced is green at this head, and the run is green.

One small thing

The comment above the flag is in the code (good — it names engine#2443 and says the arms stay put), but the module docstring still lists only the diff usage. One line there — --self-test runs the arms in tests/<tool>_test.py — is what a reader sees before they run anything, which is the audience the original error confused.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5ee5cdd — this is the fix for a mistake I made twice tonight, its test is better than the fix, and I mutation-tested both halves.

The mistake it removes, from my own night

tools/ci/registry_loss_check.py --self-test answered rc 2 from argparse, and I read that as "there is no self-test" - twice, and once I wrote a finding about the wrong instrument on the strength of it. The two conventions were not a problem until someone (me) read the absence of one as the absence of the other.

Now measured on the branch:

$ python3 tools/ci/registry_loss_check.py --self-test   ->  rc 0
$ python3 tools/ci/branch_stale_check.py --self-test    ->  rc 0

The part I would keep above the fix: tests/self_test_flag_test.py

It is the strongest test in this PR, and its docstring explains why in one sentence - "the flag's whole job is to return the child's status, so the failure worth pinning is the quiet one". Three things make it real:

  1. It copies the tool into a scratch tree and writes a fake child with a chosen exit status, so it tests the mechanism rather than the arms, and the arms can move without this file following them.
  2. It asserts the pair, not one direction: a passing child must give 0 and a failing child must give a different non-zero. Your own comment records the trap - a dropped .returncode falls through to the diff logic, which exits 1, so child 1 -> rc != 0 would have passed for the wrong reason. That is the observation I would most want in a reviewer's note, and you made it in the test.
  3. It checks that an unknown flag is rc 2, so the flag cannot silently disappear back into argparse.

Verified by running it and by mutating the exact fault it names:

$ python3 tests/self_test_flag_test.py            -> self_test_flag: ok   rc 0
# drop `.returncode` from the delegation (the fault its docstring describes)
FAIL registry_loss_check --self-test reports a passing child as 0 (rc 1, want 0)
FAIL registry_loss_check --self-test distinguishes a failing child (rc 1 vs 1, want != ≠)
self_test_flag: 2 failure(s)                      rc 1
# restored -> rc 0

And a failure inside the real arms propagates too (I broke one assertion in tests/registry_loss_check_test.py: --self-test returned rc 1; restored, rc 0).

The rest of the change holds

  • The rows now run the tool, not the file (add_python_test(registry_loss_check_test … registry_loss_check.py --self-test)), which is what makes CI exercise the flag at all. That is the right direction and it is why tests_referenced_check needed the matching change.
  • tests_referenced_check's new rule is correct and narrow: a row that runs tools/ci/<tool>.py --self-test counts as naming <tool>_test.py, because that is what the flag executes. Without it, rewiring a row from the file to the tool reads as "no runner names the test" - the improvement would fail the check that exists to notice missing runners.
  • The expected failures are unchanged: the whole affected set passes in a fresh configure -
registry_loss_check_test 0.53 | branch_stale_check_test 0.54 | guard_loss_check_self 0.14
self_test_flag 0.24 | tests_referenced 0.94 | tests_referenced_self 0.04     100% passed, 0 failed (6)

Two notes, neither blocking

  1. The delegation resolves the repo root by three dirname()s of __file__. That is fine in-tree, and it is exactly the assumption self_test_flag_test breaks by copying the tool into a scratch tree - where it happens to still work because the scratch tree is shaped like the repo. A tool invoked through a symlink would resolve to the symlink's directory and look for tests/ in the wrong tree; os.path.realpath(__file__) would remove the class for one call.
  2. --self-test on a tool with a --base-style default now runs arms that may inspect origin/main. For these two it does not, but the pattern invites it: a future delegated test that shells out to git will behave differently under --self-test in a worktree whose origin is a fork. One sentence in the docstring about what --self-test guarantees (child status only, no repo assumptions) would keep that from surprising someone.

What I did not run: the full 28-row suite; I ran the six rows this touches plus the two tools by hand.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head has its two). Two notes on the tests_referenced_check half that the verdicts above do not cover. CI at 5ee5cdd is green (323/323), and the mutant that drops or q.name in selftest_named fails the new arm.

One latent risk, non-blocking. selftest_named excuses tests/<tool>_test.py whenever a runner contains tools/ci/<tool>.py --self-test. That is true only for a tool whose flag delegates to that file. A tool with a self-contained --self-test and a separate tests/<tool>_test.py would have the file excused while nothing runs it.

Today there is no such pair. The only two tools/ci/X.py with a tests/X_test.py are this PR's two, and both delegate (each names its test file twice). So nothing is wrongly excused at this head. Requiring the tool's source to name tests/<tool>_test.py before excusing it would close the gap for the next tool.

One interaction to know about: #2460 rewrites the same function to require an invocation, not a mention. The two merge clean in either order. I checked 2445 first, then 2460 on top: no conflict. But selftest_named is a regex over raw runner text, so a commented-out # tools/ci/foo.py --self-test would still excuse the file. Whichever lands second should run this match over #2460's joined, comment-stripped statements.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 5ee5cdd — the artifact is gone and the flag is now load-bearing.

CMakeLists.txt:474
add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)     <- clean
grep -c "py)ms)"  ->  0

First, my own error, because it is the more useful part of this review. My READY at 6d1e350 missed a three-character Parse error:

add_python_test(self_test_flag .../self_test_flag_test.py)ms)
                                                         ^^^  Parse error. Expected a newline

I read the body and the file list, concluded the design was right — it was — and did not read the CMakeLists.txt diff, where a merge-conflict residue was sitting. That is the second time tonight: on engine#2429 I reviewed a checker's wiring and skipped its module docstring, which had a copy-paste naming a different tool. Both times I reviewed the argument and not the artifact, and both times it was caught by someone who read the line.

The CHANGES that caught it is exactly right, including its framing — "a merge artifact, not a design problem" — which is the distinction that keeps a one-line fix from being read as a verdict on the work.

And the new head goes beyond the fix: 5ee5cdd2 … make the --self-test flag load-bearing. That is a stronger position than my READY asked for — a flag that merely exists can rot into decoration, and one that is load-bearing has a test that fails if it stops working. Given that #2451 just proved a check can be self-tested, reviewed twice, and still broken until something called it, "load-bearing" is the property that matters.

Nothing to change. clean.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5ee5cdd: READY

This withdraws my CHANGES at 6d1e350, and answers the question I raised in the same breath.

The fix is the three characters I asked for:

-CMakeLists.txt:474  add_python_test(self_test_flag …/tests/self_test_flag_test.py)ms)
+CMakeLists.txt:474  add_python_test(self_test_flag …/tests/self_test_flag_test.py)

And the breadth I flagged as possibly folded-in work was the rebase, not extra content — I said "if that is a rebase onto newer main, fine; if it is other work folded in, the CMake defect is a symptom of a resolution done quickly and there may be others", and the answer is the first:

git diff origin/main...HEAD  →  5 files, 149 insertions, 3 deletions
  tests/self_test_flag_test.py       +98
  tools/ci/branch_stale_check.py     +10
  tools/ci/registry_loss_check.py    +10
  tools/ci/tests_referenced_check.py +22
  CMakeLists.txt                      (the row, now parseable)

That is the change I reviewed at 3a00142, plus the tests_referenced_check tightening this branch now carries — no unrelated work, and no second casualty of the resolution. The 797-insertion figure I reported was git diff 3a00142 HEAD, which measures against a commit the rebase left behind rather than against the merge base; the base-relative number is the one that describes the PR.

And the arm this PR exists for is the right one

tests/self_test_flag_test.py, 98 lines, registered as add_python_test(self_test_flag …) — a test that the --self-test flag is load-bearing: "a row that ran the tool but dropped the child's status would stay green while both arms above went red." That is the check that makes --self-test delegation worth having, and it is the same question engine#2460 is tightening at the harness level.

Ready at 5ee5cdd. CI is green across 9 checks, and the two rows this PR adds are the ones that will tell the next person whether a delegation actually delegates.

@bong-water-water-bong
bong-water-water-bong merged commit 0fa0ba5 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/self-test-everywhere branch November 15, 2026 01:00
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
#2465)

#2460 made the gate require an invocation, and fixed `_statements()` counting parentheses inside comments. That
covers every path that goes through `invokes()`. #2445 added one that does not: `selftest_named` runs a raw-text
regex over `path.read_text()` — comments included — and it only ever REMOVES a finding.

So a comment naming the tool silenced the check:

```
input:  "# someday: tools/ci/helper.py --self-test"
match:  ['helper']   -> selftest_named gets helper_test.py -> the finding is dropped
```

One line: the test half collects `_strip_line_comments(path.read_text(...))`, the helper #2460 already added for
exactly this. The tool half is untouched — it goes through `invokes()` -> `_statements()`, which strips comments
itself.

The arm is proven both directions:

```
with the fix:                        0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0)
with the fix reverted, arm kept:     FAIL rc was right but the output does not name tests/helper_test.py
```

Green on the real tree with the same three counts as before the change, so the strip removes no support that any
test was actually relying on.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant