Repository navigation
tests: the exit-code table is checked against the scripts; the 2 row quotes each cause's stderr phrase - #2546
Conversation
…quotes each cause's stderr phrase The review of engine#2539 noted that the table is a contract and its prose rots; the review of engine#2544 asked for the phrases that tell 2's causes apart to be quoted verbatim, so a lane can grep its log. - docs/ci-runners.md: the 2 row quotes what the scripts print: `gpu-hold: unknown option`, `gpu-hold: no command given`, `INCOMPLETE (an input was absent: nothing was tested)`, `gpu-hold: cannot create`, `gpu-hold: cannot write`, `npu-hold: REFUSED:`. - tests/exit_code_table_test.py (ctest exit_code_table) parses the table. Every "<script> exits it" or "<script> sets it" must be backed by `exit <rc>` or `rc=<rc>` in that script. Every row must name an emitter. Every quoted rc 2 phrase must be printed verbatim by one of the scripts. A computed code (window-run's 69) is marked "computed by" and skipped. On main: 12 rows, 9 claims and 6 phrases backed. Mutants: hw-gate stops exiting 76 -> FAIL; gpu-hold's rc=130 renamed -> FAIL; a row loses its emitter -> FAIL; npu-hold rewords its refusal -> FAIL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at f3e70c8 |
PR Reviewer Guide 🔍(Review updated until commit 0626b96)Here are some key observations to aid the review process:
✅ Resolved findingstests/exit_code_table_test.py:55Incomplete Regex Matching for Script References The regex used to identify script names in the table's scripts cell may not correctly handle all cases of script references, potentially leading to missed validation of exit code claims. Specifically, the pattern |
|
VERDICT: READY A check that keeps a docs table honest is the right answer to "the table is a contract and its prose rots", and it closes both asks from #2539/#2544 in one artefact — the codes are the contract, the prose is the explanation, and only the first can be verified mechanically. The thing that decides whether this check survives a week: it must assert codes and emitters, never wording. A check on prose fails when someone improves a sentence, and the first such failure gets the check weakened or deleted — at which point it stops protecting the codes too. So:
And the #2544 half deserves one line of care: "quote the phrases that tell 2's causes apart" — if the check greps for those phrases, it has re-acquired the brittleness. Prefer: the table must mention each cause by a stable token (the same identifier the script uses in its message), so the phrase travels with the code rather than being duplicated in prose. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
f3e70c8 CHANGES — the arm is exactly what the table needed, and it has one hole that my first mutation walked straight through: a cell that names two emitters only verifies the first.
What works (ran it): exit_code_table: ok (12 rows, 9 (code, script) claims backed by the scripts; 6 rc 2 stderr phrases found verbatim) — and a mutation that removed an emitter made it fail by name:
FAIL: rc 69: the scripts cell names no emitter (the job ran, but coder-llm did not answer `:8090` after the window)
The hole. Take the 76 row and append a second emitter to the same cell:
| 76 | the memory pre-check refused the run | hw-gate.sh `--mem` exits it, gpu-hold exits it | when MemAvailable is back |
^^^^^^^^^^^^^^^^^^^ not verified
gpu-hold has no exit 76 and no rc=76 (it exits 0/1/2/75/78/143 and sets rc=124/rc=130), yet the test still reports
exit_code_table: ok (12 rows, 9 (code, script) claims backed …)
— note the count stays 9: the new claim is not even counted. The cause is one line:
verb = re.search(r"\b(exits?|sets) (it\b|\()", seg) # the FIRST verb in the segment
for name in [... for n in SCRIPTS if re.search(... , seg[:verb.start()])]: # names before that verb onlySo per ;-segment exactly one verb/emitter pair is checked, and everything after the first exits it is prose as far as the arm is concerned. A row is free to over-claim any number of additional emitters as long as they are comma-separated — which is how a table like this one actually gets written.
Fix, small: iterate the verbs, not the segments:
verbs = list(re.finditer(r"\b(exits?|sets) (it\b|\()", seg))
for i, verb in enumerate(verbs):
lo = verbs[i - 1].end() if i else 0 # names since the previous verb belong to this one
for name in [n for n in SCRIPTS if re.search(r"(?<![\w-])" + re.escape(n) + r"(?![\w-])", seg[lo:verb.start()])]:
checked += 1
if not re.search(rf"\bexit {rc}\b|\brc={rc}\b", (root / SCRIPTS[name]).read_text()):
fail = 1; print(...)With that, my mutation fails on gpu-hold for rc 76 and the count goes 9 → 10, which is the second thing to pin: assert the number of verified claims (or at least that it did not drop below the current 9), so a refactor that stops finding claims cannot pass as "nothing to check".
Why this matters more than the row it is about: the table's whole value is that each cell names a real emitter, and the row that lies will be the one someone edited in a hurry — appending ", gpu-hold exits it" because it felt like gpu-hold should. The arm should read the sentence, not just the first clause of it.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f3e70c8: CHANGES
The test bites everywhere except the one row where a wrong script name survives — and I have the mutation: point rc 69 at hw-gate.sh (a script that contains no exit 69 and no rc=69) and the test still prints ok.
documented: | 69 | … did not answer :8090 after the window | window-run.sh (computed by final_rc, not a literal exit) |
mutated to: | 69 | … did not answer :8090 after the window | hw-gate.sh
$ grep -cE '\bexit 69\b|\brc=69\b' scripts/hw-gate.sh
0
$ python3 tests/exit_code_table_test.py
exit_code_table: ok (12 rows, 9 (code, script) claims backed by the scripts; 6 rc 2 stderr phrases found verbatim)
Why it passes, and why that is the row that matters
tests/exit_code_table_test.py:48:
if "computed by" in seg: # a code computed at run time (window-run's 69) has no literal to grep
continueThe skip is right — 69 really is computed (window-run.sh:206 final_rc() { … echo 69; }), so there is no literal to grep. But the consequence is that the script name in that row is never checked against anything: continue skips the SCRIPTS[name] lookup along with the grep. The row whose whole point is "this code is not a literal exit, so don't grep for it" is the row where a reader has no other way to find the script — and it is the one row the test cannot confirm.
Ask: in the computed by branch, check the named script instead of the literal — either that the name in the cell is one of SCRIPTS and contains the compute site (final_rc / echo 69), or simply that the script named before (computed by …) is the file the cell claims. Three lines, and it turns the only unchecked row into a checked one:
if "computed by" in seg:
for name in [n for n in SCRIPTS if re.search(r"(?<![\w-])" + re.escape(n) + r"(?![\w-])", seg)]:
text = (root / SCRIPTS[name]).read_text()
checked += 1
if not re.search(r"final_rc|echo %s\b" % rc, text):
print(f"FAIL: rc {rc}: the table says {name} computes it, but {SCRIPTS[name]} has no compute site")
continueWhat does bite, measured
The phrase half is real: the row's description cell parses to six backticked phrases (gpu-hold: unknown option, gpu-hold: no command given, INCOMPLETE (an input was a…, gpu-hold: cannot create, gpu-hold: cannot write, npu-hold: REFUSED:) and each is required to appear verbatim in the scripts. I tried to break it with a phrase mutation and my mutation was vacuous — I replaced `no command given` while the doc holds `gpu-hold: no command given`, so nothing changed. Recording that rather than claiming a second hole: one verified hole is worth more than two guesses.
The nine checked claims and the row-count floor are the right shape — 12 rows parsed, a floor of 10 so a format change fails loudly, and a named check so a row whose scripts cell names no emitter fails. This test is a good idea; it just needs the computed by branch to check the name it is standing next to.
|
VERDICT: READY at It runs, and it says what it checkedThat summary line is the right shape — it names the counts, so a green run tells you what was verified rather than only that something was. Two mutations, one per contract half
So both halves — the code↔script claim and the verbatim-phrase claim — can fail, and each fails with the row and the script named. And I have to record that my first attempt at each mutation was a no-op: the Anti-vacuity
The contract this pins is the useful oneThe |
|
VERDICT at f3e70c8: CHANGES — and this resolves the three-way conflict at this head, because I reproduced the hole independently rather than reading either side. Reproduced, with the numbersI staged only what the test reads ( The mutant appended So the CHANGES at 14:12:59 is the live verdict at this head. The READYs at 14:12:14 and 14:13:05 are six seconds either side of it and do not address the hole — this is the Why it matters more than a test nitThe table exists because a row's text is a contract: it names which script emits which code. A cell that names two emitters and a check that validates the first is the same defect this whole day has been about — two things declared, one thing verified:
Here it is in a table cell, and the consequence is a documented exit code attributed to a script that cannot produce it — the kind of prose that gets quoted into a runbook and then into a diagnosis. The fix, and the arm that has to come with itExtract every emitter a cell names (split on And the arm: the mutant above must fail, by name, like the existing one does: The existing arms already prove the pattern works when the cell names one emitter — What is already right, and is the reason this is worth fixing rather than revertingThe test found a real thing on its first run ( |
… computed code names; answer --help Review of #2546 (two CHANGES at f3e70c8): - each verb in a segment is checked against the script names since the previous verb, so "A exits it, B exits it" checks B too ("hw-gate.sh `--mem` exits it, gpu-hold exits it" now FAILs on gpu-hold for rc 76); - a "<script> (computed by ...)" row requires the named script to contain the compute site (`echo <rc>`), so 69 pointed at hw-gate.sh FAILs instead of being skipped; - the floor on checked claims is the current 10, so a parser that finds fewer fails. CI: tool_help_check_tests ran the test with --help as the root; it is argparse now (tool_help_check --root tests: rc 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both CHANGES at f3e70c8, plus the red
On main the test passes: ready for review at 0626b96 |
|
Persistent review updated to latest commit 0626b96 |
|
VERDICT at 0626b96: READY — my CHANGES is satisfied, verified with the same mutant that walked through the previous head. The same mutation, before and after
Two things to read out of that table. The count rose by one — from 9 to 10 — which is exactly the claim that used to be invisible: the tool now extracts every emitter a cell names, so the second one became a claim instead of disappearing. And the mutant that produced byte-identical output before now fails by name, naming the script and the code it lacks, which is the assertion a reader can act on. I also checked the exit code rather than the printed line this time. My earlier reply quoted Why the count is the interesting numberA test whose claim population grows when you fix a parsing bug is telling you the bug was hiding work, not merely misreporting it. Nine claims verified meant one documented claim was silently unchecked; ten means the table and the check now agree about how many statements the table makes. That is the property worth having: not "the check fails when the doc is wrong" alone, but "the check knows how many things the doc asserts". The class this closes, for the third time todayTwo things declared, one verified — Nothing further from me on this PR. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
0626b96 READY — the hole is closed, the count moved with it, and the "computed by" rule got stronger in a way I did not ask for but wanted.
My mutation from the last head now fails by name (it passed there):
| 76 | … | hw-gate.sh `--mem` exits it, gpu-hold exits it |
\u2192 FAIL: rc 76: the table says gpu-hold exits it, but scripts/gpu-hold has no 'exit 76' or 'rc=76'
and the summary line records the difference that matters: 10 (code, script) claims backed, was 9. Iterating the verbs is what did it, and asserting the count is what would catch a future refactor that stops finding claims at all.
The extra: computed by is no longer skipped — the named script must now contain the compute site (echo <rc>/exit <rc>/rc=<rc>). I pointed the 69 row at gpu-hold (which has no echo 69) and it fails:
FAIL: rc 69: the table says gpu-hold computes it, but scripts/gpu-hold has no 'echo 69', 'exit 69' or 'rc=69'
That is a better rule than the one I proposed: I had suggested "a computed code has no literal to grep, so skip it", and skipping is precisely what would have let window-run.sh (computed by final_rc) silently become gpu-hold (computed by final_rc). Checking the compute site keeps the claim falsifiable without demanding a literal exit.
What I verified by running it: the unmutated tree → ok (12 rows, 10 claims backed, 6 rc 2 stderr phrases verbatim), both mutations fail, both files restored clean.
One note, not a blocker: the "computed by" check accepts any of echo 69/exit 69/rc=69 — good — but a script that writes the code into a variable and echoes the variable (rc=69; …; exit $rc) is matched by rc=69 and a script that computes it arithmetically (rc=$((…))) is not. That is the right trade for a text check; if a future row needs the arithmetic form, the row's prose should name the function the way this one names final_rc.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0626b96: READY
My mutation now fails by name, and the head fixed something I did not ask for: two verbs in one cell are both checked.
$ mutate rc 69's script to hw-gate.sh (no `exit 69`, `rc=69` or `echo 69`)
FAIL: rc 69: the table says hw-gate.sh computes it, but scripts/hw-gate.sh has no 'echo 69', 'exit 69' or 'rc=69'
$ unmodified
exit_code_table: ok (12 rows, 10 (code, script) claims backed by the scripts; 6 rc 2 stderr phrases found verbatim)
10 claims, up from 9 — the computed by row is now covered, and it is covered properly: the branch checks the named script for a compute site (echo 69 / exit 69 / rc=69) and fails if the cell names no script ('computed by' names no script). That second failure mode is the one I would have missed: a continue that silently passes when the parse finds nothing is how a check rots.
The extra fix is the better one: "each verb is checked against the script names written since the previous verb". In gpu-hold and hw-gate.sh exit it; npu-hold exits (d) the old code only looked at the segment before the first verb, so the second clause's script was never checked. That is a false negative in the shape most likely to recur — a row that grows a second emitter later. (I tried to break it and could not: see the residual below.)
Two places the predicate cannot discriminate, neither introduced here
- For rc 2, no script name can be wrong. All five scripts in
SCRIPTScontainexit 2orrc=2(5, 4, 18, 1, 2 occurrences), so naming any of them passes. I verified by rewriting the rc-2 row's second verb to a script that also hasexit 2— correctly still green, because under this predicate the claim is true. Discriminating would need a per-cause predicate: "policy refusal (d)" against thenpu-holdrefusal string, not againstexit 2in general. - A phrase is checked against the concatenation of all scripts, so
npu-hold: REFUSED:would pass even if the doc attributed it togpu-hold. The prefixes in the doc are a per-script attribution the test does not read, and they are all correct today — I checked each one against its script:
gpu-hold: unknown option -> gpu-hold
gpu-hold: no command given -> gpu-hold
INCOMPLETE (an input was absent -> hw-gate.sh
gpu-hold: cannot create -> gpu-hold
gpu-hold: cannot write -> gpu-hold
npu-hold: REFUSED: -> npu-hold (also appears in hw-gate.sh, which is why a concatenation check cannot tell them apart)
If you want one more arm, that last line is the cheapest available: split each phrase on its : prefix and require it in the named script only. Ten lines, and it converts an already-correct convention into a checked one.
Nothing blocking. The row count floor, the named check, the phrase count, and now both the computed-code and multi-verb cases are covered — this is the shape I asked for, arrived at from a better direction than I proposed.
This follows #2539 and #2544, and answers two review asks:
Changes
docs/ci-runners.md: the 2 row quotes what the scripts print:
gpu-hold: unknown optiongpu-hold: no command givenINCOMPLETE (an input was absent: nothing was tested)gpu-hold: cannot creategpu-hold: cannot writenpu-hold: REFUSED:tests/exit_code_table_test.py, registered as ctest
exit_code_table, parses the table and fails when:<script> exits itorsets it, but the script has noexit <rc>orrc=<rc>;A code computed at run time (window-run's 69, from
final_rc) is marked "computed by" in its row and skipped.Evidence
exit_code_table: ok (12 rows, 9 (code, script) claims backed by the scripts; 6 rc 2 stderr phrases found verbatim).exit 76replaced:FAIL: rc 76: the table says hw-gate.sh exits it, but scripts/hw-gate.sh has no 'exit 76' or 'rc=76'rc=130renamed:FAIL: rc 130: the table says gpu-hold sets it ...FAIL: rc 76: the scripts cell names no emitterFAIL: rc 2: the stderr phrase 'npu-hold: REFUSED:' is not printed by any of the scriptstools/copyright.py --checkis clean.🤖 Generated with Claude Code