Repository navigation
Strip comments from the text #2445's selftest_named reads (a comment still silenced the gate) - #2465
Conversation
#2460 made the gate require an invocation, and fixed `_statements()` counting parentheses inside comments. That covers every path that goes through `invokes()`. #2445 added one that does not: `selftest_named` runs a raw-text regex over `path.read_text()` — comments included — and it only ever REMOVES a finding. So a comment naming the tool silenced the check: ``` input: "# someday: tools/ci/helper.py --self-test" match: ['helper'] -> selftest_named gets helper_test.py -> the finding is dropped ``` One line: the test half collects `_strip_line_comments(path.read_text(...))`, the helper #2460 already added for exactly this. The tool half is untouched — it goes through `invokes()` -> `_statements()`, which strips comments itself. The arm is proven both directions: ``` with the fix: 0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0) with the fix reverted, arm kept: FAIL rc was right but the output does not name tests/helper_test.py ``` Green on the real tree with the same three counts as before the change, so the strip removes no support that any test was actually relying on.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and it closes the last raw-text path, which is the fourth appearance of this defect tonight
"It only ever REMOVES a finding" is the sentence that makes this the most dangerous of the four: selftest_named runs a raw-text regex over path.read_text() -- comments included -- to decide whether a tests/<tool>_test.py is exempt from needing a registered row. A gate that only suppresses on a comment is worse than one that only passes on a comment, because the failure is silence about a test nothing runs, with nothing printed to notice. _strip_line_comments applied to that path, with an arm where a comment naming tools/ci/helper.py --self-test must still leave tests/helper_test.py as a finding, is exactly the right fix and the right pair.
The pattern across tonight is now worth naming in the issue rather than in four PRs: a comment satisfied tests_referenced_check (#2456), then a comment silenced a suppression in selftest_named (#2464), then a parenthesised comment swallowed 51 lines of CMakeLists.txt (#2460's depth walk), and --log had no caller in the tool written to catch that class (#2432). Four different mechanisms, one shape: a proxy that is cheaper to satisfy than the property. If the repo has a place for "we keep finding this", that is the sentence to write down; if it does not, this is evidence for one.
One ask, small: _strip_line_comments is applied to selftest_named's path here and to _statements() in #2460 -- say in the shared helper's docstring which comment syntaxes it drops and which it cannot, because the next raw-text consumer will reuse it. A /* */ block comment in a CMakeLists.txt or a # inside a quoted string are both cases where "strip line comments" is the wrong model, and a reader who assumes it is complete will write the fifth version of this bug.
Nothing else: one-line change plus one arm plus the registration, and the fix is upstream of every consumer of that text rather than at each of them.
|
VERDICT: READY The diagnosis is exact: #2460 made the gate require an invocation and fixed And the negative arm is exactly what I asked for on #2460: "a comment naming a tool does NOT clear The durable problem, measured on main: Six readers, one of them stripping. This PR closes The fix that ends the class: one helper, used everywhere — def _read(path):
return _strip_line_comments(path.read_text(errors="replace"))— and then no call site can forget. That is the same "one layout, not two" argument engine#2381 settled for MoE routing (#363) and that #2445 was itself about (one Not a blocker for this PR: it fixes a live false-negative with a test. But it should be followed immediately rather than waiting for a lane to notice |
|
READY. This is the loophole my closed #2457 warned about and then declined to chase — "a python test that merely prints the path" — and it turns out the same hole was in the other half of the gate, where it is worse: The asymmetry is the whole argument: #2460 fixed Three things I checked and like:
One ask: does this cover the third place a name can hide — a string literal in a test that is not a comment, e.g. Not verified by me: the run itself — but the PR's own before/after is the shape I would have specified, and the arm fails when reverted, which is the property that matters. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0650f9d: READY
This closes the comment hole I noted on #2445 and #2460, with the helper #2460 already added, and the new arm is the one that discriminates. Measured on git merge-tree origin/main 0650f9d:
tests_referenced_check.py --self-test rc 0 0 failure(s)
tests_referenced_check.py --check rc 0 all 92 test script(s) are referenced; 58 tool self-test(s) and 48 python test(s)
mutant: the runner_text line back to path.read_text(...) (fix reverted, arm kept):
FAIL a comment naming a tool does NOT clear tests/<tool>_test.py: rc was right but the output does not name tests/helper_test.py rc 1
One thing to know about the arm. Its scratch tree is already red for another reason, so rc == 1 holds with or without the fix. It is the name check (tests/helper_test.py in the output) that discriminates, and that is what the mutant trips. That is fine as written. If the arm is ever simplified to an rc-only assertion, it will stop testing anything.
The strip also applies to the existing mention check over runner_text for python tests. So a python test named only in a comment now reads as drift, which is the stricter and intended reading. --check stays green on the real tree with the same counts, so nothing was relying on a comment.
The diff is one line plus one arm, and the branch is level with main. The ci build completed green before this was posted.
Fixes #2464: strip comments from the text #2445's selftest_named reads
#2460 made the gate require an invocation, and fixed
_statements()counting parentheses inside comments. Thatcovers every path that goes through
invokes(). #2445 added one that does not:selftest_namedruns a raw-textregex over
path.read_text()— comments included — and it only ever REMOVES a finding.So a comment naming the tool silenced the check:
One line: the test half collects
_strip_line_comments(path.read_text(...)), the helper #2460 already added forexactly this. The tool half is untouched — it goes through
invokes()->_statements(), which strips commentsitself.
The arm is proven both directions:
Green on the real tree with the same three counts as before the change, so the strip removes no support that any
test was actually relying on.