Repository navigation
tests: seven helpers treated --help as a path, and one WROTE a file named --help - #2339
Conversation
PR Reviewer Guide 🔍(Review updated until commit 3f9ebd7)Here are some key observations to aid the review process:
✅ Resolved findingstests/fake_spawn_exec.py:26-29Incorrect Argument Parsing The script tests/laya_route_cases.py:187-190File Creation on --help The script tests/fake_lemonade.py:24-27Incorrect Argument Parsing The script |
|
VERDICT: READY The finding is the one that mattered, and the fix is in the right place: the check was never wrong, its root was too narrow.
Two things I checked rather than assumed:
One thing that will drift, and it is the third time tonight I have flagged the same pattern: the covered roots are now an explicit list ( Not verified by me: a run of the two new rows (they need the tree as the PR has it). The criterion and the registration are what I read; both are host-side and one command each. |
|
VERDICT: READY at The finding is a scope, and the list is the evidence
Seven helpers, and one of them writes — which is the #2268 defect ("a tool that answers Two asks
And the writer deserves its own line in the body
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 2c27e9f - the eight fixes are right; the checker's own reach still disagrees with its comment
What is fixed, verified by running it the two ways the PR names:
$ tool_help_check.py --check --root tests rc 0 no tool crashes on --help (54 asked, 6 not asked)
$ tool_help_check.py --check --root onnx rc 0 no tool crashes on --help (7 asked, 0 not asked)
and the worst of the eight is the one the body leads with -- tests/laya_route_cases.py --help takes sys.argv[1] as an output path, so --help is not a crash, it is a file named --help written into the checkout. That is the same defect the tool's own header records (oga_* wrote 172 KB of JSON into the repo root), which makes this PR a fix to a check that was silently not
covering the place the bug lives. The return 2 idiom the eight now share is right: not 0 (help is not success) and not a traceback.
The remaining item, and it is the same class one level up -- the checker's scope is wider than its comment says:
SKIP_DIR_NAMES = {'.git', 'build', 'node_modules', '__pycache__', 'third_party', '.venv'}
# Scanned root is tools/ only. `tests/` fixtures are not asked for help, and `os/licenses/` holds
# vendored third-party sources -- a traceback in someone else's example file is not our defect.
…
root = '.' # <- line 293: the widest possible scope, not tools/So on a working tree with a build directory the default --check walks into it, and on this tree it reports:
tool_help: build1336/_deps/httplib-src/test/dns_test_fixture.py -- `--help` raises: ValueError: invalid literal for int()
tool_help: build1336/_deps/json-src/tests/thirdparty/imapdl/filterbr.py -- `--help` raises: FileNotFoundError: …
tool_help_check: 2 tool(s) crash on --help, of 170 asked rc 1
build is skipped by exact name, so build1336 -- and build-hrx, build_x, any name anyone uses -- is walked, and the tool reports tracebacks in vendored third-party python as our failures. That is the thing the comment says it does not do.
Two small edits close it: skip by prefix (part == 'build' or part.startswith('build'), or a build* glob), and fix the comment, which currently says "Scanned root is tools/ only. tests/ fixtures are not asked for help" while the code scans .. The second matters as much as the first: a reader who trusts that comment will think tests/ is deliberately out of
scope, which is exactly the assumption this PR exists to correct.
Why I am not simply passing it: the eight fixes are what the PR is for and they are good; but the sentence in the file now actively teaches the wrong thing about the tool's reach, in a PR whose whole subject is a checker whose reach was not what anyone thought. A --root tests green run and a stale promise of narrow scope in the same file is the combination worth not shipping.
Not run: --self-test explicitly; the two --root runs above exercise the eight fixes, and the default run is what reports the vendored-tree noise.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2c27e9f: READY
The root cause is the checker's corpus, and the fix is two rows — which is the right level, because the alternative is seven patches and an eighth helper next week.
The body names it exactly: tools/ci/tool_help_check.py --check runs with --root tools, so tests/ and onnx/ were never asked. Adding tool_help_check_tests (--root tests) and tool_help_check_onnx (--root onnx) makes the existing check ask the trees it never asked, and that is what finds the seven, not a new mechanism.
The seven are a useful record, because each failed differently and only the first two classes were known:
tests/laya_route_cases.py --help WROTE a file named `--help` into the checkout
tests/fake_lemonade.py --help ValueError: invalid literal for int(): '--help'
tests/fake_spawn_exec.py --help ValueError: invalid literal for int(): '--help'
tests/batch_check_state_ci.py --help FileNotFoundError: '--help'
tests/embed_forward_ci.py --help FileNotFoundError: '--help'
laya_route_cases.py is the write class from #2268, now in Python — and there is a satisfying compounding here: the checker that #2268 taught to notice a write (before/after tree state, not just an exit code) is the same checker that now covers tests/. The mechanism was already right; it was pointed at one directory.
Why this is the third instance of one lesson tonight, and the reason to note it rather than just merge: #2258 found twelve shell scripts leaking code into usage text, #2268 found a tool that wrote on --help, and this finds seven Python helpers — in each case the check existed and the corpus did not. A ratchet whose corpus never read the directory the defect lived in reports a clean tree honestly and uselessly.
The rc-2 answer for a helper asked for --help is also the right one: a usage line and a distinct exit code, so a caller can tell "you asked wrong" from "the helper ran and failed".
No concerns, and the two new ctest rows are the part that keeps it fixed.
…a file named --help `tools/ci/tool_help_check.py --check` covers `--root tools`, so `tests/` and `onnx/` were never asked. Asking them: ``` tests/laya_route_cases.py --help wrote a file named `--help` into the checkout tests/fake_lemonade.py --help ValueError: invalid literal for int(): '--help' tests/fake_spawn_exec.py --help ValueError: invalid literal for int(): '--help' tests/batch_check_state_ci.py --help FileNotFoundError: '--help' tests/embed_forward_ci.py --help FileNotFoundError: '--help' tests/gptoss_forward_ci.py --help FileNotFoundError: '--help' tests/serve_spec_lookup_test.py --help FileNotFoundError: '--help' onnx/tests/onnx_conv_integer_types_test.py --help FileNotFoundError: '--help' ``` Eight of 61. **The first is the one that matters**: `--help` is not a crash there, it is a *write* -- the generator takes `sys.argv[1]` as its output path and `json.dump`s into it, so asking a tool what it does dirties the checkout with a file called `--help`. The other seven are the same bug read-only: the first positional is taken as a runner path or a port with no check. Each now answers with its own usage line and **rc 2**, touching nothing: ``` $ python3 tests/laya_route_cases.py --help # in an empty directory rc=2 Writes tests/laya_route_cases.json: 200 hand-labelled chat requests ... $ ls -A # nothing created ``` **Measured through the gate, not by eye** -- `tool_help_check --check` per root: ``` --root tools rc 0 (95 asked, 7 not asked) unchanged --root tests rc 0 (54 asked, 6 not asked) was 7 crashes of 54 --root onnx rc 0 ( 7 asked, 0 not asked) was 1 crash of 7 ``` Two ctest rows added (`tool_help_check_tests`, `tool_help_check_onnx`, 36.53 s and 1.90 s) so the next one of these is a red row rather than a stray file in someone's checkout, and the flags are checked without a pipe (`rc=2` read directly; `| head` reports `head`'s status, not the tool's). `copyright --check` rc 0.
2c27e9f to
3f9ebd7
Compare
|
Persistent review updated to latest commit 3f9ebd7 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 3f9ebd7 - withdraws my CHANGES at 2c27e9f, with the two lines it asked for pasted so they cost ten seconds
Withdrawn because the class only bites a developer tree, not CI: build is in SKIP_DIR_NAMES, so a CI checkout (which uses build/) never walks into it. The noise I measured -- 2 tool(s) crash on --help with vendored build1336/_deps/** files named -- needs a build directory that is not called exactly build, which is my box and not the runner's. So the eight fixes should land now, and the two edits are a follow-up rather than a gate.
The two lines, exactly:
-SKIP_DIR_NAMES = {'.git', 'build', 'node_modules', '__pycache__', 'third_party', '.venv'}
-# Scanned root is tools/ only. `tests/` fixtures are not asked for help, and `os/licenses/` holds
-# vendored third-party sources -- a traceback in someone else's example file is not our defect.
+SKIP_DIR_NAMES = {'.git', 'node_modules', '__pycache__', 'third_party', '.venv'}
+# A build tree is skipped by PREFIX: `build`, `build1336`, `build-hrx` are all build trees, and the exact
+# name matched only the first -- so `--check` on a developer tree walked its `_deps/**` and reported
+# tracebacks in vendored python as ours.
+def is_build(part): return part == "build" or part.startswith("build")
…
- if any(part in SKIP_DIR_NAMES for part in f.parts):
+ if any(part in SKIP_DIR_NAMES or is_build(part) for part in f.parts):And the comment is the half worth not losing, because the code contradicts it: root = '.' at :293, so the actual scope is the whole repo including build*/_deps/** and tests/ -- which is what this PR is about. A reader who edits SKIP_DIR_NAMES will believe the comment and not look at root.
Measured, so the follow-up has a number: on my tree the default --check is rc 1 with two vendored crashes (build1336/_deps/httplib-src/test/dns_test_fixture.py, build1336/_deps/json-src/tests/thirdparty/imapdl/filterbr.py); on main before this PR it was 10, of which 3 were the same artifacts and 7 were the eight the PR fixes.
Everything I verified at 2c27e9f still holds: --root tests rc 0 (54 asked), --root onnx rc 0 (7 asked), and the laya_route_cases.py --help write -- the case the tool's own header records.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the scope was the bug, and registering two ctest rows is the durable half of the fix
The root cause is not the seven helpers, it is the row that asked them. tool_help_check.py --check ran with --root tools, so tests/ and onnx/ were never asked -- the same failure as "a check that exists and was not run", one level up: a check that runs and covers a third of the tree. Asking them found seven helpers and one onnx runner path, and the two additions here -- tool_help_check_tests and tool_help_check_onnx as ctest rows -- mean the trees get asked by CI from now on rather than by whoever remembers. That is the part that lasts; the per-file --help handlers are the part that just stops today's noise.
rc 2 for -h/--help is the right choice and consistent with the rest of this family (pre-pr-checks.sh returns 2 for usage, and a usage request is not a failure).
Two things I would add, both about the eighth helper rather than these seven:
-
Nothing yet asserts that
--helpleaves the checkout alone.tests/laya_route_cases.py --help"WROTE a file named--helpinto the checkout" -- a file that a latergit add -Acommits, or that a test comparing directory contents trips over. Adding a usage line stops it happening; it does not stop the next helper doing it. One assertion in the checker closes the class: after running each--help,git status --porcelainmust be empty (and it should fail if a path literally named--helpnow exists). Cheap, and it turns "we fixed seven" into "this cannot come back quietly". -
--root testsand--root onnxare a list, and a list is how this happened. A newscripts/ornpu/tree would again be uncovered until someone remembers. Either default--rootto the repository root and let the trees opt out, or have--checkfail when a top-level directory containing*.pyis not covered by some row -- the second is stricter and matches how the baselines in this repo are already asserted.
Nothing else to raise: 31 lines, tests included, and the destructive case is named in the body rather than left to be discovered.
tools/ci/tool_help_check.py --checkruns with--root tools, sotests/andonnx/were never asked. Asking them:Eight of 61.
The first is the one that matters: there
--helpis not a crash, it is a write — the generator takessys.argv[1]as its output path andjson.dumps into it, so asking a tool what it does leaves a file called--helpin the checkout. Reproduced in an empty directory: it prints200 cases -> --helpand creates that file. The other seven are the same bug read-only: the first positional is taken as a runner path or a port with no check.Each now prints its own usage line and exits rc 2, touching nothing:
Measured through the gate, not by eye —
tool_help_check --checkper root:Two ctest rows added (
tool_help_check_tests30.40 s,tool_help_check_onnx0.98 s) so the next one of these is a red row rather than a stray file in someone's checkout.copyright --checkrc 0. The rc values above are read without a pipe —| headreportshead's status, which is how a crashing tool can look like it answered.