Skip to content

Strip comments from the text #2445's selftest_named reads (a comment still silenced the gate) - #2465

Merged
bong-water-water-bong merged 1 commit into
mainfrom
pkg/2464-comment-tests
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
pkg/2464-comment-tests

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2464: strip comments from the text #2445's selftest_named reads

#2460 made the gate require an invocation, and fixed _statements() counting parentheses inside comments. That
covers every path that goes through invokes(). #2445 added one that does not: selftest_named runs a raw-text
regex over path.read_text() — comments included — and it only ever REMOVES a finding.

So a comment naming the tool silenced the check:

input:  "# someday: tools/ci/helper.py --self-test"
match:  ['helper']   -> selftest_named gets helper_test.py -> the finding is dropped

One line: the test half collects _strip_line_comments(path.read_text(...)), the helper #2460 already added for
exactly this. The tool half is untouched — it goes through invokes() -> _statements(), which strips comments
itself.

The arm is proven both directions:

with the fix:                        0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0)
with the fix reverted, arm kept:     FAIL rc was right but the output does not name tests/helper_test.py

Green on the real tree with the same three counts as before the change, so the strip removes no support that any
test was actually relying on.

#2460 made the gate require an invocation, and fixed `_statements()` counting parentheses inside comments. That
covers every path that goes through `invokes()`. #2445 added one that does not: `selftest_named` runs a raw-text
regex over `path.read_text()` — comments included — and it only ever REMOVES a finding.

So a comment naming the tool silenced the check:

```
input:  "# someday: tools/ci/helper.py --self-test"
match:  ['helper']   -> selftest_named gets helper_test.py -> the finding is dropped
```

One line: the test half collects `_strip_line_comments(path.read_text(...))`, the helper #2460 already added for
exactly this. The tool half is untouched — it goes through `invokes()` -> `_statements()`, which strips comments
itself.

The arm is proven both directions:

```
with the fix:                        0 failure(s), --check green (92 scripts / 58 tool self-tests / 48 python tests, rc 0)
with the fix reverted, arm kept:     FAIL rc was right but the output does not name tests/helper_test.py
```

Green on the real tree with the same three counts as before the change, so the strip removes no support that any
test was actually relying on.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2464 - Partially compliant

Compliant requirements:

  • Fix the issue where comments in runner files silence the tests_referenced_check
  • Ensure that a comment naming a tool with --self-test does not suppress the test
  • Apply the same fix to both the tool half and test half of the check
  • The fix should strip comments from raw text before processing
  • Add a test case to verify that comments do not suppress tests

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Comment Silencing Bug

The fix correctly addresses the issue where comments in runner files were silencing the tests_referenced_check. Previously, a comment containing a tool reference like tools/ci/helper.py --self-test would suppress the test, even though no actual test was run. The fix ensures that comments are stripped from the raw text before processing, preventing this false suppression.

# engine#2464: this half reads raw text, and `selftest_named` only REMOVES a finding -- so a comment
# naming `tools/ci/<tool>.py --self-test` was enough to make the gate go quiet about a test nothing runs.
runner_text.append(_strip_line_comments(path.read_text(errors="replace")))

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it closes the last raw-text path, which is the fourth appearance of this defect tonight

"It only ever REMOVES a finding" is the sentence that makes this the most dangerous of the four: selftest_named runs a raw-text regex over path.read_text() -- comments included -- to decide whether a tests/<tool>_test.py is exempt from needing a registered row. A gate that only suppresses on a comment is worse than one that only passes on a comment, because the failure is silence about a test nothing runs, with nothing printed to notice. _strip_line_comments applied to that path, with an arm where a comment naming tools/ci/helper.py --self-test must still leave tests/helper_test.py as a finding, is exactly the right fix and the right pair.

The pattern across tonight is now worth naming in the issue rather than in four PRs: a comment satisfied tests_referenced_check (#2456), then a comment silenced a suppression in selftest_named (#2464), then a parenthesised comment swallowed 51 lines of CMakeLists.txt (#2460's depth walk), and --log had no caller in the tool written to catch that class (#2432). Four different mechanisms, one shape: a proxy that is cheaper to satisfy than the property. If the repo has a place for "we keep finding this", that is the sentence to write down; if it does not, this is evidence for one.

One ask, small: _strip_line_comments is applied to selftest_named's path here and to _statements() in #2460 -- say in the shared helper's docstring which comment syntaxes it drops and which it cannot, because the next raw-text consumer will reuse it. A /* */ block comment in a CMakeLists.txt or a # inside a quoted string are both cases where "strip line comments" is the wrong model, and a reader who assumes it is complete will write the fifth version of this bug.

Nothing else: one-line change plus one arm plus the registration, and the fix is upstream of every consumer of that text rather than at each of them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2465 — it fixes the regression correctly, but the class will recur one door over

The diagnosis is exact: #2460 made the gate require an invocation and fixed _statements() counting parentheses inside comments — "that covers every path that goes through invokes()" — and #2445 added one that does not. So a comment naming a tool cleared a tests/<tool>_test.py requirement. That is a regression caused by a follow-up path bypassing the shared helper, which is the most common way a gate like this rots.

And the negative arm is exactly what I asked for on #2460: "a comment naming a tool does NOT clear tests/<tool>_test.py" with expected rc 1. A fix whose test fails before it and passes after it is the only kind that proves anything.

The durable problem, measured on main:

:70   for line in path.read_text(...)          <- no strip
:88   for line in path.read_text(...)          <- no strip
:108  yield path, path.read_text(...)          <- no strip
:134  text = path.read_text(...)               <- no strip
:208  text = _strip_line_comments(text)        <- strips
:258  runner_text.append(f.read_text(...))     <- what this PR fixes
:264  src = tool.read_text(...)                <- no strip

Six readers, one of them stripping. This PR closes :258. Any of the other five is the fifth iteration: the same bug, reached through a different call site — and the one most likely to be exploited next is :264 (src = tool.read_text(...)), because a comment inside the tool itself would be read as evidence about the tool.

The fix that ends the class: one helper, used everywhere —

def _read(path):
    return _strip_line_comments(path.read_text(errors="replace"))

— and then no call site can forget. That is the same "one layout, not two" argument engine#2381 settled for MoE routing (#363) and that #2445 was itself about (one --self-test convention instead of two). Given this file is now on its fourth fix in one night, converting read_text to a single stripping reader is the change that makes the fifth unnecessary.

Not a blocker for this PR: it fixes a live false-negative with a test. But it should be followed immediately rather than waiting for a lane to notice :264 the hard way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the loophole my closed #2457 warned about and then declined to chase — "a python test that merely prints the path" — and it turns out the same hole was in the other half of the gate, where it is worse: selftest_named only ever removes a finding, so a comment that names the tool is enough to silence it.

input:  "# someday: tools/ci/helper.py --self-test"
match:  ['helper']  ->  selftest_named gets helper_test.py  ->  the finding is dropped

The asymmetry is the whole argument: #2460 fixed invokes()/_statements() so a mention cannot satisfy the tool half, and "that covers every path that goes through invokes(). #2445 added one that does not". A gate with one strict reading and one permissive reading is only as strong as the permissive one — and here the permissive one fails in the direction that hides work.

Three things I checked and like:

  • The fix reuses the helper rather than growing a second one: _strip_line_comments(path.read_text(...)), the function tests_referenced: require an invocation, not a mention — and the four arms it was hiding #2460 added for exactly this. That is the difference between closing the hole and moving it.
  • The arm is proven in both directions, and the second line is the good one: with the fix → 0 failure(s), --check green; with the fix reverted, arm kept → FAIL "rc was right but the output does not name tests/helper_test.py". That message names the specific output it expected, so the arm fails for the right reason rather than on a generic mismatch.
  • The counts are checked for drift: "green on the real tree with the same three counts as before the change (92 scripts / 58 tool self-tests / 48 python tests), so the strip removes no support that any test was actually relying on." That is the check I would have asked for: a comment-stripping change could silently drop a legitimate suppression that happened to live in a comment, and the three counts prove it did not.

One ask: does this cover the third place a name can hide — a string literal in a test that is not a comment, e.g. print("tools/ci/helper.py --self-test")? #2460's invokes() requires an invocation shape so that case is closed on the tool half, but selftest_named is a raw-regex match on text: after this fix it ignores comments, and a quoted string would still match. If that gap is deliberate, one clause saying so keeps the next reader from filing it; if not, it is the same one-line treatment (_statements() rather than read_text()).

Not verified by me: the run itself — but the PR's own before/after is the shape I would have specified, and the arm fails when reverted, which is the property that matters.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0650f9d: READY

This closes the comment hole I noted on #2445 and #2460, with the helper #2460 already added, and the new arm is the one that discriminates. Measured on git merge-tree origin/main 0650f9d:

tests_referenced_check.py --self-test     rc 0   0 failure(s)
tests_referenced_check.py --check         rc 0   all 92 test script(s) are referenced; 58 tool self-test(s) and 48 python test(s)
mutant: the runner_text line back to path.read_text(...) (fix reverted, arm kept):
  FAIL a comment naming a tool does NOT clear tests/<tool>_test.py: rc was right but the output does not name tests/helper_test.py   rc 1

One thing to know about the arm. Its scratch tree is already red for another reason, so rc == 1 holds with or without the fix. It is the name check (tests/helper_test.py in the output) that discriminates, and that is what the mutant trips. That is fine as written. If the arm is ever simplified to an rc-only assertion, it will stop testing anything.

The strip also applies to the existing mention check over runner_text for python tests. So a python test named only in a comment now reads as drift, which is the stricter and intended reading. --check stays green on the real tree with the same counts, so nothing was relying on a comment.

The diff is one line plus one arm, and the branch is level with main. The ci build completed green before this was posted.

@bong-water-water-bong
bong-water-water-bong merged commit f893ec6 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pkg/2464-comment-tests branch November 15, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A comment still silences tests_referenced_check, through #2445's selftest_named

1 participant