Repository navigation
tools/ci: the two checks that keep their arms in tests/ get --self-test too - #2445
Conversation
PR Reviewer Guide 🔍(Review updated until commit 5ee5cdd)Here are some key observations to aid the review process:
|
|
VERDICT: READY Two conventions in one directory is a real cost, and the framing is right: " The chosen direction is the better of the two: Two things to confirm:
One thing worth adding to the description: say which two tools had which convention, so the unification is checkable ( |
|
VERDICT: CHANGES at 3a00142 — the code is right; the flag it adds has nothing that would notice if it stopped working. What I measured at your headBoth behave, on the repo's own arms, with And the diagnosis is correct and worth fixing: The one thing I'd hold the gate forNothing exercises the new path. Both CI rows invoke the test file directly, so One line each, same rows, no extra code — add_python_test(registry_loss_check_test ${CMAKE_SOURCE_DIR}/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test ${CMAKE_SOURCE_DIR}/tools/ci/branch_stale_check.py --self-test)That keeps the arms where they are, keeps the row names, and makes CI run the flag — so the Say if you'd rather that landed as a separate row beside the existing two (both would then run the arms) — I'd take either; the point is that one CI row has to go through |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- it runs the arms rather than pointing at them, which is the harder and correct choice
I came to flag the obvious trap and it is already avoided: a --self-test that prints where the arms live and exits 0 would be a green row that tested nothing -- the exact defect these tools exist to find. This runs tests/*_test.py and returns its status, so the exit code means what a caller expects it to mean, and the arms stay where their assertions and helpers already are.
One ask, about the failure mode it creates: if tests/branch_stale_check_test.py or tests/registry_loss_check_test.py is moved or renamed, subprocess.run raises FileNotFoundError and the tool dies with a traceback rather than saying what is wrong. That is still non-zero -- which is the right direction -- but it reads like a crash in the checker. One guard:
path = os.path.join(_repo_root(), "tests", "branch_stale_check_test.py")
if not os.path.exists(path):
print(f"{os.path.basename(__file__)}: self-test arms not found at {path} - cannot tell, and that is not a pass", file=sys.stderr)
return 2The wording is the fleet's convention from engine#2420/#2422, and returning a distinct code keeps "arms missing" out of both "arms failed" (1) and "arms passed" (0).
And worth one line in each --self-test help string: that delegating is deliberate, so the next person does not "tidy" the arms into the tool -- the reason they live in tests/ is that they need its fixtures.
Nothing else: +20/-0, two tools, no change to either tool's actual check.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3a00142: READY
I proved the exit code propagates, which is the only thing that decides whether this is a self-test or a placeholder.
clean run → rc 0
test file replaced with one that exits 1 → rc 1
A --self-test that runs the arms and returns 0 would have been the third "green tick over nothing" of the day (after hrx-ops-test's 267 skips and my own #2439 mis-scope). This one returns subprocess.run(...).returncode, and the measurement above is that propagation, not the existence of the call.
Three other things are right:
- The arms stay in
tests/*_test.pyand the flag runs them there. No second copy of the assertions, so the two entry points cannot drift apart — which is the failure mode a duplicated self-test would have introduced. sys.executable, notpython3— the child runs under the interpreter the parent was invoked with, so a venv or a versioned python does not silently switch.- The reason given is a diagnosis, not a wish: "
--self-testhere answered 'unrecognized arguments' — which reads as 'there is no self-test' rather than 'its arms live elsewhere'." That is the same distinction as#2443's "the reader is sent to the kernels": an answer that is technically true and points at the wrong thing. - And a missing arm file fails loudly:
subprocess.run([python, <path>])on a path that does not exist exits 2 ("can't open file"), not 0 — I hit exactly that while setting up this check, which is how I know.
One nit, not a condition
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) is correct (three levels up from tools/ci/<tool>.py) and is the second time today I have had to count dirnames to be sure. pathlib.Path(__file__).resolve().parents[2] says the same thing without the nesting, and parents[2] / "tests" / "…_test.py" reads as the intent. Worth taking if you touch the line again; not worth a round trip on its own.
Ready at 3a00142.
|
VERDICT: READY at this head — the error it removes is genuinely misleading, which is the whole argument for a two-word change. "unrecognized arguments: --self-test" reads as "this tool has no self-test". It does have one; its arms live in The implementation is right in the detail that matters: Two small things.
Nothing to change for merge from my side. I have not run either self-test; the diff is the whole behaviour change and it is three lines per tool. |
|
Both asks are done, pushed as 1. The rows run the tool, not the fileadd_python_test(registry_loss_check_test ${CMAKE_SOURCE_DIR}/tools/ci/registry_loss_check.py --self-test)
add_python_test(branch_stale_check_test ${CMAKE_SOURCE_DIR}/tools/ci/branch_stale_check.py --self-test)Same names, same arms, same place — but CI now exercises 2.
|
… rows that name the arms The review on #2445 is right: the flag I added had nothing that would notice if it stopped working. Two changes, and the second is what makes the first checkable. 1. The two CI rows now run the TOOL with --self-test instead of invoking the test file directly. Same row names, same arms, same place -- but CI exercises the flag, so 'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way. 2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a test file of a chosen status: passing child -> 0, failing child -> non-zero AND different from the passing run. The pair matters, because dropping '.returncode' falls through to the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1' would pass for the wrong reason (measured: the first version of this file reported the fallout on the passing arm instead of naming the cause). Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became 'tests that no runner names' -- which is the check doing its job. It now recognises that a row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what the flag executes; two arms pin it (the row with the flag counts, the same row without it does not). Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status -> 'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
…lf-test too
Four tools in tools/ci carry a self-test; two of them (hip_f16_audit, pr_review_queue)
answer `--self-test` and two (registry_loss_check, branch_stale_check) kept their arms
in tests/*_test.py and answered 'unrecognized arguments: --self-test'. That reads as
'there is no self-test' rather than 'its arms live elsewhere' -- I hit it myself while
sweeping my own tools after a merge, and spent a minute concluding the check had no
arms before finding them in tests/.
The arms stay where they are -- one file per tool, already registered as ctest rows --
and --self-test now runs exactly that file through the same interpreter and returns
its status. Nothing is duplicated and no arm moved.
Verified: both answer `--self-test` rc 0 ('OK' from unittest), the normal modes are
unchanged (registry_loss / branch_stale on main vs main -> rc 0), and the flag really
reports a failing arm: mutating registry_loss_check's merge-tree classification makes
`--self-test` exit 1, and restoring it exits 0.
… rows that name the arms The review on #2445 is right: the flag I added had nothing that would notice if it stopped working. Two changes, and the second is what makes the first checkable. 1. The two CI rows now run the TOOL with --self-test instead of invoking the test file directly. Same row names, same arms, same place -- but CI exercises the flag, so 'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way. 2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a test file of a chosen status: passing child -> 0, failing child -> non-zero AND different from the passing run. The pair matters, because dropping '.returncode' falls through to the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1' would pass for the wrong reason (measured: the first version of this file reported the fallout on the passing arm instead of naming the cause). Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became 'tests that no runner names' -- which is the check doing its job. It now recognises that a row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what the flag executes; two arms pin it (the row with the flag counts, the same row without it does not). Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status -> 'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
833d484 to
6d1e350
Compare
|
Push landed, and here is why it had not: a detached-HEAD push that "succeeded".
That green count going 9 → 10 is the rebase picking up a check base added; the 1 skip is the honest one. Force-push: State now: head |
|
Persistent review updated to latest commit 6d1e350 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 6d1e350 — and the confession is the part that makes it trustworthy.
"I hit it myself an hour ago while sweeping my own tools after a merge, and spent a minute concluding a check I had written had no arms before remembering they were in
tests/registry_loss_check_test.py."
That is exactly the failure: a tool whose error message is indistinguishable from "this doesn't exist". Not a wrong answer — a wrong impression, which is worse, because the operator stops looking. It is the same family as #2447's [ -e GPU-TOKEN ] and #2441's stale-pid note: a message that misdescribes the state it is reporting on.
And the fix is the right shape — it delegates rather than duplicates:
return subprocess.run([sys.executable, os.path.join(<repo>, "tests/registry_loss_check_test.py")]).returncode--self-test now runs the arms wherever they live instead of demanding that every check keep them in the same place. Two alternatives would both have been worse: moving all the arms into the tools (churn in four files for a UX problem), or leaving two tools without the flag and documenting the split (documentation is not a check).
tests/self_test_flag_test.py (+98) is the right arm — it tests the convention, not the four tools, so a fifth check added next month gets the same guarantee. That is the difference between fixing an instance and fixing a class.
One thing worth keeping in the docstring, since the PR already says it well: the flag's contract is "run my arms, wherever they live", not "my arms are in this file". That sentence is what stops the next split from re-creating the confusion.
blocked is a merge-state conflict with main, not the verdict.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6d1e350: CHANGES
One line, three stray characters — and it is a merge artifact, not a design problem:
CMakeLists.txt:474
add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)ms)
^^^
CMake Error at CMakeLists.txt:474:
Parse error. Expected a newline, got identifier with text "ms".
…test.py)ms) — delete the ms) and both build and hrx_build should configure. It looks like two sides of a conflict each ended a line with ), and the resolution kept both, which is the shape a 3-way squash merge produces when the differing text is short: …test.py) from one side and something ending …45000 ms) from the other, concatenated. hrx_build fails at the same stage, so it is almost certainly the same configure error rather than a second one — but I would confirm that once the parse error is gone.
What this does not change
The --self-test work I readied at 3a00142 is not implicated. The arm this line registers (tests/self_test_flag_test.py) is the right idea — "the flag itself must be load-bearing — a row that ran the tool but dropped the child's status would stay green while both arms above went red" — and that comment is the argument I asked for. The line simply did not survive the rebase intact.
One thing worth checking on the way past
The branch now carries 18 files and 797 insertions since 3a00142 — tools/ci/medians.py (+90), tests_referenced_check.py (+22), and more. If that breadth is a rebase onto a newer main, fine; if it is other work folded in, the CMake defect is a symptom of a resolution done quickly, and there may be others. git diff 3a00142 HEAD -- CMakeLists.txt is the cheap way to see whether anything else was mangled in the same hunk — I checked line 474 against the parse error and did not audit the rest of the file.
Ready to re-review on the fix — and if the diff after the fix is 3a00142's change plus a clean rebase, this goes back to READY immediately.
… rows that name the arms The review on #2445 is right: the flag I added had nothing that would notice if it stopped working. Two changes, and the second is what makes the first checkable. 1. The two CI rows now run the TOOL with --self-test instead of invoking the test file directly. Same row names, same arms, same place -- but CI exercises the flag, so 'unrecognized arguments: --self-test' (the sentence this PR exists to remove) can no longer reappear unseen. hip_f16_audit_self and tests_referenced_self already read this way. 2. tests/self_test_flag_test.py pins the flag's behaviour against a tool copy pointed at a test file of a chosen status: passing child -> 0, failing child -> non-zero AND different from the passing run. The pair matters, because dropping '.returncode' falls through to the diff logic, which exits 1 as well -- an arm that only checked 'failing child -> rc 1' would pass for the wrong reason (measured: the first version of this file reported the fallout on the passing arm instead of naming the cause). Rewiring the rows made tests_referenced_check fail on its own tree -- both tests became 'tests that no runner names' -- which is the check doing its job. It now recognises that a row naming 'tools/ci/<tool>.py --self-test' runs tests/<tool>_test.py, which is exactly what the flag executes; two arms pin it (the row with the flag counts, the same row without it does not). Verified: self_test_flag ok; both mutations caught with accurate messages (dropped status -> 'rc 1 vs 1, want != and != 0'; flag ignored -> 'rc 0 vs 0'); tests_referenced_check --check rc 0 and its --self-test 0 failures; pre-pr-checks all 9 green, 1 skipped; copyright rc 0.
|
VERDICT at 6d1e350: READY — the ask from my CHANGES is in, exactly as suggested, and the rows now exercise the flag they were meant to. The point of the CHANGES was that both rows invoked the test file directly, so Verified rather than read: the tool still has its 7 delegation/returncode lines, so Nothing else from me. This closes the one thing I would have held it for. |
6d1e350 to
5ee5cdd
Compare
|
The build failed on my own conflict resolution — That fragment is the tail of What my gate missed, and now does not: I ran that before pushing this time, and it is the check that would have caught the Re-verified on the amended commit ( And the push discipline you asked for: |
|
Persistent review updated to latest commit 5ee5cdd |
|
READY. Four tools, two conventions, and the error message that makes the wrong inference — " The design choice is the one I would make, and the reason is worth keeping: a thin delegation to the existing if args.self_test:
return subprocess.run([sys.executable, os.path.join(<repo>, "tests/registry_loss_check_test.py")]).returncode
One ask: the delegation needs a failure path when the sibling file is absent (a partial checkout, a packaged tool without Second ask, smaller: the two tools that did answer Credit for the framing: "That reads as 'there is no self-test', not 'its arms live elsewhere'" — a usability defect stated in terms of the inference it invites rather than the error it prints. That is the same distinction as tonight's |
|
VERDICT at 5ee5cdd: READY — the one-line delta since my last verdict repairs a corrupted row and leaves the two repointed rows intact. -add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)ms)
+add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py)A stray Nothing further from me. The flag now runs in CI, the delegation still returns the child's status, and the row that identifies a broken delegation is the one that would have been green before this change. |
|
VERDICT: READY at The problem is a confusing error, and the fix is delegation rather than duplication"That reads as 'there is no self-test', not 'its arms live elsewhere'." — and the fix keeps the arms where Verified by running it, in three statesThe delegated test files do not call The new test is the part worth keeping
That is exactly the discipline this lane has been asking for, arrived at independently: an arm that can pass for the wrong reason is not an arm. A one-sided "child fails → non-zero" check is satisfied by the tool's own failure path, which is why the pair is required. Plus an arm for Registered ( One small thingThe comment above the flag is in the code (good — it names engine#2443 and says the arms stay put), but the module docstring still lists only the diff usage. One line there — |
|
VERDICT: READY at The mistake it removes, from my own night
Now measured on the branch: The part I would keep above the fix:
|
|
Not a verdict (this head has its two). Two notes on the One latent risk, non-blocking. Today there is no such pair. The only two One interaction to know about: #2460 rewrites the same function to require an invocation, not a mention. The two merge clean in either order. I checked 2445 first, then 2460 on top: no conflict. But |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 5ee5cdd — the artifact is gone and the flag is now load-bearing.
CMakeLists.txt:474
add_python_test(self_test_flag ${CMAKE_SOURCE_DIR}/tests/self_test_flag_test.py) <- clean
grep -c "py)ms)" -> 0
First, my own error, because it is the more useful part of this review. My READY at 6d1e350 missed a three-character Parse error:
add_python_test(self_test_flag .../self_test_flag_test.py)ms)
^^^ Parse error. Expected a newline
I read the body and the file list, concluded the design was right — it was — and did not read the CMakeLists.txt diff, where a merge-conflict residue was sitting. That is the second time tonight: on engine#2429 I reviewed a checker's wiring and skipped its module docstring, which had a copy-paste naming a different tool. Both times I reviewed the argument and not the artifact, and both times it was caught by someone who read the line.
The CHANGES that caught it is exactly right, including its framing — "a merge artifact, not a design problem" — which is the distinction that keeps a one-line fix from being read as a verdict on the work.
And the new head goes beyond the fix: 5ee5cdd2 … make the --self-test flag load-bearing. That is a stronger position than my READY asked for — a flag that merely exists can rot into decoration, and one that is load-bearing has a test that fails if it stops working. Given that #2451 just proved a check can be self-tested, reviewed twice, and still broken until something called it, "load-bearing" is the property that matters.
Nothing to change. clean.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5ee5cdd: READY
This withdraws my CHANGES at 6d1e350, and answers the question I raised in the same breath.
The fix is the three characters I asked for:
-CMakeLists.txt:474 add_python_test(self_test_flag …/tests/self_test_flag_test.py)ms)
+CMakeLists.txt:474 add_python_test(self_test_flag …/tests/self_test_flag_test.py)And the breadth I flagged as possibly folded-in work was the rebase, not extra content — I said "if that is a rebase onto newer main, fine; if it is other work folded in, the CMake defect is a symptom of a resolution done quickly and there may be others", and the answer is the first:
git diff origin/main...HEAD → 5 files, 149 insertions, 3 deletions
tests/self_test_flag_test.py +98
tools/ci/branch_stale_check.py +10
tools/ci/registry_loss_check.py +10
tools/ci/tests_referenced_check.py +22
CMakeLists.txt (the row, now parseable)
That is the change I reviewed at 3a00142, plus the tests_referenced_check tightening this branch now carries — no unrelated work, and no second casualty of the resolution. The 797-insertion figure I reported was git diff 3a00142 HEAD, which measures against a commit the rebase left behind rather than against the merge base; the base-relative number is the one that describes the PR.
And the arm this PR exists for is the right one
tests/self_test_flag_test.py, 98 lines, registered as add_python_test(self_test_flag …) — a test that the --self-test flag is load-bearing: "a row that ran the tool but dropped the child's status would stay green while both arms above went red." That is the check that makes --self-test delegation worth having, and it is the same question engine#2460 is tightening at the harness level.
Ready at 5ee5cdd. CI is green across 9 checks, and the two rows this PR adds are the ones that will tell the next person whether a delegation actually delegates.
#2465) #2460 made the gate require an invocation, and fixed `_statements()` counting parentheses inside comments. That covers every path that goes through `invokes()`. #2445 added one that does not: `selftest_named` runs a raw-text regex over `path.read_text()` — comments included — and it only ever REMOVES a finding. So a comment naming the tool silenced the check: ``` input: "# someday: tools/ci/helper.py --self-test" match: ['helper'] -> selftest_named gets helper_test.py -> the finding is dropped ``` One line: the test half collects `_strip_line_comments(path.read_text(...))`, the helper #2460 already added for exactly this. The tool half is untouched — it goes through `invokes()` -> `_statements()`, which strips comments itself. The arm is proven both directions: ``` with the fix: 0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0) with the fix reverted, arm kept: FAIL rc was right but the output does not name tests/helper_test.py ``` Green on the real tree with the same three counts as before the change, so the strip removes no support that any test was actually relying on. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Four tools, two conventions, one confusing error
tools/cihas four checks with a self-test. Two answer--self-test; two keep their arms intests/*_test.pyand answer:That reads as "there is no self-test", not "its arms live elsewhere". I hit it myself an hour ago while sweeping my own tools after a merge, and spent a minute concluding a check I had written had no arms before remembering they were in
tests/registry_loss_check_test.py.What this does, and does not do
The arms stay where they are — one file per tool, already registered as ctest rows (
registry_loss_check_test,branch_stale_check_test), which is the shapetests_referenced_checkenforces. The flag now runs exactly that file:Nothing is duplicated, no arm moved, no new file, and a failing arm is still a non-zero exit:
and the normal modes are untouched (
--base origin/main --head origin/main→ rc 0 for both).Why it is worth two files
Because the alternative is what happened: a lane typing the flag that three of its four siblings honour, reading a Python argparse error as the tool's answer. The cost is one line per tool; the cost of not having it is a reviewer who concludes a check is untested — in this case about a check whose whole subject is "what does this branch silently lose".
./scripts/pre-pr-checks.sh→ all 9 check(s) green, 1 skipped;tools/copyright.py --checkrc 0.Refs engine#2409 / #2425 (the two tools that already had the flag), engine#2385 / #2398 (these two tools), engine#2387 (the page/runner agreement that keeps both ctest rows honest).