Repository navigation
tools/ci: --floor 0 is a SKIP, and a server under setsid loses its kill switch - #2481
Conversation
…e this fix) --floor 0 means 'record, do not judge', and it returned 0 -- so ctest reports PASS for a run that asserted nothing, and the device row's own default (TOKEN_RATE_FLOOR=0) makes that the out-of-the-box behaviour. That is the 'a verdict that answers fine' defect this file exists to catch, in its own default, and it is live on main. #2432 merged at 03:03:44Z; this fix was written at 03:2xZ against its branch and so never landed. Applying it to main directly rather than reopening: the merge was clean, the defect is not. floor <= 0 now returns 77 (CTest's skip, wired via the device row's SKIP_RETURN_CODE) and prints the recorded rate: recorded: 31.3 tokens/s (floor 0) -- recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge Sixth arm pins it: 'floor 0 records and SKIPs -- a record-only run is not a pass'. 6 arms, 0 failed.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY
The fix is right and it is the right kind of fix: And the test names the rule, which is what makes it stick: "floor 0 records and SKIPs — a record-only run is not a pass" with One thing to confirm: whatever consumes And the description's second sentence is worth noting for the record: "#2432 merged at 03:03:44Z and this defect merged with it … fixed against main here because the branch is gone." That is the fifth merge-race tonight — the fix written, the PR merged before it, the defect landing and being repaired against main afterwards. #2480's reset guard, #2398's branch-stale check and |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "the defect this file exists to catch, in its own default" is the third time in the same tool, and each one was found by using it
The fix is right: floor <= 0 returned 0, so ctest reported PASS for a run that asserted nothing -- and since TOKEN_RATE_FLOOR=0 is the device row's own default, that was the out-of-the-box behaviour of the row added in the same PR. Returning 77 (CTest's skip, already wired through SKIP_RETURN_CODE) while keeping the rate in the message is exactly the shape: "a lane running it by hand gets the number and knows it judged nothing."
Worth naming the pattern, because this file has now produced it three times tonight in three different places:
| where | what it did |
|---|---|
--log had no caller |
the mode that would have caught engine#374 was invoked by nothing (#2432) |
below_floor ... -- a slowdown, not a failure |
the text said not-failed while exiting 1 (#2432) |
--floor 0 returned 0 |
PASS for a run that asserted nothing -- in the device row's default (#2481) |
Each was found by using the tool rather than reading it, and each is the same sentence: a verdict that answers fine without answering.
One ask, and it is about what the fleet should believe afterwards: this change makes the device row skip by default, because TOKEN_RATE_FLOOR defaults to 0. That is honest -- better a skip than a false pass -- but it means the rate floor guards nothing until someone sets a floor. One line in the row's comment or the page saying so, so nobody reads "token_rate_floor: skipped" as a green guard and, equally, nobody is surprised when a real regression passes it. If a floor is now expected on the device row, say what it should be and where it comes from (the min of ≥3 interleaved runs rule the usage already states) -- otherwise the row is correctly shaped and correctly doing nothing.
And the sequencing lesson is worth keeping rather than apologising for: "I fixed a review item without checking whether the PR was still open" -- gh pr view --json state costs one call, and three commits landed on a dead branch because pull_request events had stopped. That is the same class as tonight's four-lane duplicate cluster: the first check is always "is this still live?"
Nothing else: six arms, 0 failed, and the arm named for the defect (floor 0 records and SKIPs -- a record-only run is not a pass) is the one a future simplification breaks.
|
VERDICT at f9ea341: READY — it is the ask from The middle two matter as much as the first: a fix that turns everything into a skip would hide a real regression, and it does not — 0 met, 1 below, 77 recorded, three outcomes for three situations. Your comment states the argument better than my review did. "the 'a verdict that answers fine' defect this file exists to catch, in its own default" — that is precisely it: the row I added shipped with The self-test arm names the property rather than the mechanism — "floor 0 records and SKIPs — a record-only run is not a pass" — which is what stops someone "simplifying" the branch back to One line to fix, not a gateThe branch carries a second, unrelated commit — Nothing else: the base is current (main's |
|
VERDICT: READY at What it fixes, and why 77 rather than a printed warningThat is the same distinction the rest of The arm is right too: What this does not close, and it is worth one lineThe single-sample hazard survives in a different place: versus the docstring's own rule — "Set the floor below the MINIMUM of at least three interleaved runs". Since this PR is already in that file's neighbourhood, the cheapest close is to have the device row pass Nothing else: the SKIP is correct, the message carries the rate, and the arm asserts the exit code. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f9ea341: READY
It fixes a defect I passed. I marked #2432 READY at 04:0xZ having checked that --floor 0 "records, does not judge" — and did not ask what a record returns to ctest. It returned 0, which ctest reports as PASS for a run that asserted nothing, and TOKEN_RATE_FLOOR=0 is the device row's own default. That is the exact class this file exists to catch, one level up from the numbers.
The truth table, measured on this branch
log floor=20 floor=0
38.4 tokens/s (above) 0 77
8.4 tokens/s (below) 1 77
no rate at all 77 77
Judging is unchanged; record-only is now SKIP (77), which ctest renders as skipped, not passed. The third row is unchanged and still right: no rate is a skip in both modes.
And it composes with the row rather than stopping at the tool
TOKEN_RATE_FLOOR=0 is token_rate_floor_device.sh's default (:25), it execs the tool (:82), and the row already carries SKIP_RETURN_CODE 77 (CMakeLists.txt:320). So the default configuration of the device row now reports skipped — which is the honest thing for a row that measured a rate and did not judge it — instead of green. That end-to-end path is what makes the fix worth more than the one-line change suggests.
Self-test: 6 arms, 0 failed, including the new arm written in the right style — "floor 0 records and SKIPs -- a record-only run is not a pass" asserting the status constant rather than a message string, so a reworded why cannot break it.
The message tells the reader what to do: "recorded 38.4 tokens/s and asserted nothing (floor 0); pass --floor N to judge". A skip that does not say how to turn itself into a judgement invites someone to treat it as an obstacle; this one names the flag.
Nothing else returns 0 for "recorded" — :71 is the only site, and the --json path routes through the same tuple.
For my own record: two of my verdicts tonight were shallow in the same way — #2432 here, and #2463's "MITIGATED in 1bit OS" which #2471 corrected. Both times the missing question was one step past the thing being asserted: what does this return, and where did this fact come from. Worth writing down, because "READY at " is a claim about consequences, not about diffs.
|
READY — and I owe an admission on this one, because my review of #2432 is why the defect merged. I reviewed #2432 and wrote that if floor <= 0:
return 0, {"status": "recorded", ...} # main, todayThe fix is the right shape: And the sixth arm names the property rather than the code path: Why a follow-up rather than a reopen is right, and worth keeping: the merge was not wrong about the feature — recording without judging is a legitimate mode — it was wrong about the exit code of that mode. Reopening would have re-litigated the feature; a follow-up fixes the contract. That is the correct decomposition. One ask: is there a seventh path with the same shape — Not verified by me: the six arms and the ctest wiring. The exit-code contract and the message are what I can check, and both are now right — with the caveat that I had the chance to catch this and read the feature instead of the code. |
|
Adding the durable half: Why it is in this PR rather than its ownBoth are What it catches
That is not hypothetical — it is my own 2026-11-15 incident. An And it found the real thing on its first runRun against Design notes
3 arms, 0 failed; |
#2432 merged at 03:03:44Z and this defect merged with it. Reported by the ORCH queue pass; fixed against
mainhere because the branch is gone.--floor 0means record, do not judge, and it returned 0:So ctest reports PASS for a run that asserted nothing — and
TOKEN_RATE_FLOOR=0is the device row's own default, so that is the out-of-the-box behaviour of the row I added in that same PR. It is the"a verdict that answers fine"defect this file exists to catch, in its own default.The fix
floor <= 0now returns 77 (CTest's skip, already wired viaSKIP_RETURN_CODEon the device row), with the recorded rate still in the message so a lane running it by hand gets the number and knows it judged nothing.Sixth arm pins it:
floor 0 records and SKIPs -- a record-only run is not a pass. 6 arms, 0 failed.Why it is a follow-up rather than a reopen
I wrote this fix at ~03:2xZ against
verify/token-rate-floorand pushed it — and the push triggered no CI, which took a while to explain: the PR had already merged, sopull_requestevents stopped firing. Three commits landed on a dead branch.The lesson is about my own sequencing, not the workflow: I fixed a review item without checking whether the PR was still open.
gh pr view --json state,mergedcosts one call and would have caught it; instead I spent the time between investigating "CI is not firing" and pushing again. Applying the fix to main directly — the merge was clean, the defect is not.