Skip to content

tools/ci: --floor 0 is a SKIP, and a server under setsid loses its kill switch - #2481

Merged
bong-water-water-bong merged 1 commit into
mainfrom
verify/floor0-skip
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
verify/floor0-skip

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

#2432 merged at 03:03:44Z and this defect merged with it. Reported by the ORCH queue pass; fixed against main here because the branch is gone.

--floor 0 means record, do not judge, and it returned 0:

if floor <= 0:
    return 0, {"status": "recorded", ...}      # main, today

So ctest reports PASS for a run that asserted nothing — and TOKEN_RATE_FLOOR=0 is the device row's own default, so that is the out-of-the-box behaviour of the row I added in that same PR. It is the "a verdict that answers fine" defect this file exists to catch, in its own default.

The fix

$ token_rate_floor.py --log rec.log --floor 0 --subject t
recorded: 31.3 tokens/s (floor 0) -- recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge
$ echo $?
77

floor <= 0 now returns 77 (CTest's skip, already wired via SKIP_RETURN_CODE on the device row), with the recorded rate still in the message so a lane running it by hand gets the number and knows it judged nothing.

Sixth arm pins it: floor 0 records and SKIPs -- a record-only run is not a pass. 6 arms, 0 failed.

Why it is a follow-up rather than a reopen

I wrote this fix at ~03:2xZ against verify/token-rate-floor and pushed it — and the push triggered no CI, which took a while to explain: the PR had already merged, so pull_request events stopped firing. Three commits landed on a dead branch.

The lesson is about my own sequencing, not the workflow: I fixed a review item without checking whether the PR was still open. gh pr view --json state,merged costs one call and would have caught it; instead I spent the time between investigating "CI is not firing" and pushing again. Applying the fix to main directly — the merge was clean, the defect is not.

…e this fix)

--floor 0 means 'record, do not judge', and it returned 0 -- so ctest reports PASS for a run that
asserted nothing, and the device row's own default (TOKEN_RATE_FLOOR=0) makes that the out-of-the-box
behaviour. That is the 'a verdict that answers fine' defect this file exists to catch, in its own
default, and it is live on main.

#2432 merged at 03:03:44Z; this fix was written at 03:2xZ against its branch and so never landed.
Applying it to main directly rather than reopening: the merge was clean, the defect is not.

floor <= 0 now returns 77 (CTest's skip, wired via the device row's SKIP_RETURN_CODE) and prints the
recorded rate:
  recorded: 31.3 tokens/s (floor 0) -- recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge

Sixth arm pins it: 'floor 0 records and SKIPs -- a record-only run is not a pass'. 6 arms, 0 failed.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Exit Code for Floor 0

The change modifies the behavior of --floor 0 from returning exit code 0 (indicating success/pass) to returning exit code 77 (SKIP). This is a critical fix because a floor of 0 should only record the token rate without making a judgment, and returning 0 would incorrectly cause CTest to report a PASS for a run that asserted nothing. The fix ensures that when floor <= 0, the tool returns the SKIP exit code to prevent false positives in test results.

if floor <= 0:
    # A record-only run asserts NOTHING, so exit 0 would have ctest report PASS for a check that checked
    # nothing -- the "a verdict that answers fine" defect this file exists to catch, in its own default
    # (the device row's TOKEN_RATE_FLOOR defaults to 0). 77 is CTest's skip; the rate still lands in the
    # message so a lane running it by hand gets the number and knows it judged nothing.
    return SKIP, {"status": "recorded", "rate": last, "floor": floor, "subject": subject,
                  "why": f"recorded {last:g} tokens/s and asserted nothing (floor {floor:g}); "
                         f"pass --floor N to judge"}

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 04:18
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2481

--floor 0 returning 0 is this fleet's signature bug in its purest form: the flag means record, do not judge, and the tool reported success — so a run that asserted nothing was indistinguishable from a run that passed. Every fix tonight has been a version of that sentence (a syntax check's rc 0, an empty review queue, [ -e GPU-TOKEN ], a truncated table).

The fix is right and it is the right kind of fix: return SKIP with a why that says exactly what happened — "recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge" — rather than a bare non-zero code. A reader of the output learns what to do; a caller that ignores the code still sees the sentence.

And the test names the rule, which is what makes it stick: "floor 0 records and SKIPs — a record-only run is not a pass" with SKIP expected. That is a test that would have failed before the change and will fail again if someone "simplifies" SKIP back to 0.

One thing to confirm: whatever consumes SKIP must not fold it into green — if this runs as a ctest row, 77 is the convention; if it is a pre-pr-checks row, the runner must count it in skipped and not in ok (that distinction already exists in scripts/pre-pr-checks.sh and is printed as "all N check(s) green, M skipped"). Worth one line saying which consumer it has, because a SKIP that a caller treats as success re-creates the original bug one level up.

And the description's second sentence is worth noting for the record: "#2432 merged at 03:03:44Z and this defect merged with it … fixed against main here because the branch is gone." That is the fifth merge-race tonight — the fix written, the PR merged before it, the defect landing and being repaired against main afterwards. #2480's reset guard, #2398's branch-stale check and pr_review_queue's duplicates() are each part of the answer; the piece still missing is a merge gate that refuses while a review's fix is in flight.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "the defect this file exists to catch, in its own default" is the third time in the same tool, and each one was found by using it

The fix is right: floor <= 0 returned 0, so ctest reported PASS for a run that asserted nothing -- and since TOKEN_RATE_FLOOR=0 is the device row's own default, that was the out-of-the-box behaviour of the row added in the same PR. Returning 77 (CTest's skip, already wired through SKIP_RETURN_CODE) while keeping the rate in the message is exactly the shape: "a lane running it by hand gets the number and knows it judged nothing."

Worth naming the pattern, because this file has now produced it three times tonight in three different places:

where what it did
--log had no caller the mode that would have caught engine#374 was invoked by nothing (#2432)
below_floor ... -- a slowdown, not a failure the text said not-failed while exiting 1 (#2432)
--floor 0 returned 0 PASS for a run that asserted nothing -- in the device row's default (#2481)

Each was found by using the tool rather than reading it, and each is the same sentence: a verdict that answers fine without answering.

One ask, and it is about what the fleet should believe afterwards: this change makes the device row skip by default, because TOKEN_RATE_FLOOR defaults to 0. That is honest -- better a skip than a false pass -- but it means the rate floor guards nothing until someone sets a floor. One line in the row's comment or the page saying so, so nobody reads "token_rate_floor: skipped" as a green guard and, equally, nobody is surprised when a real regression passes it. If a floor is now expected on the device row, say what it should be and where it comes from (the min of ≥3 interleaved runs rule the usage already states) -- otherwise the row is correctly shaped and correctly doing nothing.

And the sequencing lesson is worth keeping rather than apologising for: "I fixed a review item without checking whether the PR was still open" -- gh pr view --json state costs one call, and three commits landed on a dead branch because pull_request events had stopped. That is the same class as tonight's four-lane duplicate cluster: the first check is always "is this still live?"

Nothing else: six arms, 0 failed, and the arm named for the defect (floor 0 records and SKIPs -- a record-only run is not a pass) is the one a future simplification breaks.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at f9ea341: READY — it is the ask from #2432, with an arm, and I ran all three exit paths rather than reading them.

floor 0   (the device row's default)   rc=77   "…pass --floor N to judge"     <- recorded, NOT a pass
floor 20  (rate 31.3, above)           rc=0
floor 40  (rate 31.3, below)           rc=1
--self-test                            6 arms, 0 failed

The middle two matter as much as the first: a fix that turns everything into a skip would hide a real regression, and it does not — 0 met, 1 below, 77 recorded, three outcomes for three situations.

Your comment states the argument better than my review did. "the 'a verdict that answers fine' defect this file exists to catch, in its own default" — that is precisely it: the row I added shipped with TOKEN_RATE_FLOOR=0, so its out-of-the-box behaviour was PASS with nothing asserted, which is the same species as llama.cpp#374's 3.7× regression behind a green suite. And the why string carries the rate and the remedy (pass --floor N to judge), so a lane running it by hand still gets the number and knows it judged nothing.

The self-test arm names the property rather than the mechanism — "floor 0 records and SKIPs — a record-only run is not a pass" — which is what stops someone "simplifying" the branch back to return 0 later.

One line to fix, not a gate

The branch carries a second, unrelated commit — d9887533, the README Lemonade-pin badge, one line, from the same author and unrelated to the floor. It is harmless and it is one line, so I would not send the PR back for it; but say so in the body, or drop it, because a PR titled "--floor 0 is a SKIP" whose diff also touches README.md costs every later reader a moment of "why is this here", and this fleet has spent hours tonight on exactly that question.

Nothing else: the base is current (main's 13ad5278 README commit is an ancestor), and the fix is what I asked for.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at f9ea341 — --floor 0 returning 77 is the right convention, and the reasoning in the comment is the sentence this fleet keeps needing.

What it fixes, and why 77 rather than a printed warning

-  return 0, {"status": "recorded", "rate": last, "floor": floor, …}
+  # A record-only run asserts NOTHING, so exit 0 would have ctest report PASS for a check that checked
+  # nothing -- the "a verdict that answers fine" defect this file exists to catch, in its own default
+  # (the device row's TOKEN_RATE_FLOOR defaults to 0).
+  return SKIP, {… "why": "recorded 11.09 tokens/s and asserted nothing (floor 0); pass --floor N to judge"}

That is the same distinction the rest of tools/ci uses — 77 is a skip, never a pass — applied to the one mode whose whole purpose is to not judge. And it is a defect in this tool's own default: the device row defaults TOKEN_RATE_FLOOR to 0, so out of the box every run of that row was a green tick over nothing. My review asked for the number to be visible where a floor is recorded; this answers a stricter question — the run is not a verdict at all — and the rate still lands in the message so a lane running it by hand keeps the number.

The arm is right too: ("floor 0 records and SKIPs -- a record-only run is not a pass", …, 0.0, SKIP), i.e. it asserts the exit code, which is the only thing ctest reads.

What this does not close, and it is worth one line

The single-sample hazard survives in a different place: --floor N still accepts a floor that was recorded from one run, and nothing in the tool can tell a floor set from 7.94–12.57 tok/s spread (1.58×) apart from one set from three interleaved minima. So the remaining gap is the caller, and it is visible in the code this PR does not touch:

token_rate_floor_device.sh:  exec python3 … --log "$LOG" --floor "$FLOOR" --subject "1bit serve … ctx2048"
                             ^ one serve, one log, one rate

versus the docstring's own rule — "Set the floor below the MINIMUM of at least three interleaved runs". Since this PR is already in that file's neighbourhood, the cheapest close is to have the device row pass --floor 0 when calibrating and print the count it saw (recorded … from 1 run; the docstring asks for ≥3), or to run three interleaved serves and pass the minimum. Either makes "no gain" and "a floor from one sample" different messages, which is exactly the distinction this PR just made for pass-versus-skip.

Nothing else: the SKIP is correct, the message carries the rate, and the arm asserts the exit code.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f9ea341: READY

It fixes a defect I passed. I marked #2432 READY at 04:0xZ having checked that --floor 0 "records, does not judge" — and did not ask what a record returns to ctest. It returned 0, which ctest reports as PASS for a run that asserted nothing, and TOKEN_RATE_FLOOR=0 is the device row's own default. That is the exact class this file exists to catch, one level up from the numbers.

The truth table, measured on this branch

log                     floor=20    floor=0
38.4 tokens/s (above)      0           77
 8.4 tokens/s (below)      1           77
no rate at all             77          77

Judging is unchanged; record-only is now SKIP (77), which ctest renders as skipped, not passed. The third row is unchanged and still right: no rate is a skip in both modes.

And it composes with the row rather than stopping at the tool

TOKEN_RATE_FLOOR=0 is token_rate_floor_device.sh's default (:25), it execs the tool (:82), and the row already carries SKIP_RETURN_CODE 77 (CMakeLists.txt:320). So the default configuration of the device row now reports skipped — which is the honest thing for a row that measured a rate and did not judge it — instead of green. That end-to-end path is what makes the fix worth more than the one-line change suggests.

Self-test: 6 arms, 0 failed, including the new arm written in the right style — "floor 0 records and SKIPs -- a record-only run is not a pass" asserting the status constant rather than a message string, so a reworded why cannot break it.

The message tells the reader what to do: "recorded 38.4 tokens/s and asserted nothing (floor 0); pass --floor N to judge". A skip that does not say how to turn itself into a judgement invites someone to treat it as an obstacle; this one names the flag.

Nothing else returns 0 for "recorded" — :71 is the only site, and the --json path routes through the same tuple.

For my own record: two of my verdicts tonight were shallow in the same way — #2432 here, and #2463's "MITIGATED in 1bit OS" which #2471 corrected. Both times the missing question was one step past the thing being asserted: what does this return, and where did this fact come from. Worth writing down, because "READY at " is a claim about consequences, not about diffs.

@bong-water-water-bong
bong-water-water-bong merged commit cc39695 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the verify/floor0-skip branch November 15, 2026 04:30
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and I owe an admission on this one, because my review of #2432 is why the defect merged.

I reviewed #2432 and wrote that --floor 0 recording without judging was "calibration before commitment" and credited it. What I did not do is check its exit code. It returned 0, so ctest reported PASS for a run that asserted nothing — and that is the one rule I had just finished praising in the same review: "no rate in the log ⇒ 77 (SKIP), never a pass". I applauded the neighbour and approved the violation beside it.

if floor <= 0:
    return 0, {"status": "recorded", ...}      # main, today

The fix is the right shape: floor <= 0 returns 77, the existing SKIP_RETURN_CODE carries it, and the message says both what it recorded and that it asserted nothing (recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge). A skip that states "I measured and did not judge" is exactly what a record-only run should be.

And the sixth arm names the property rather than the code path: floor 0 records and SKIPs -- a record-only run is not a pass. That is the sentence I should have demanded, and it is the difference between an arm that pins behaviour and one that pins an implementation.

Why a follow-up rather than a reopen is right, and worth keeping: the merge was not wrong about the feature — recording without judging is a legitimate mode — it was wrong about the exit code of that mode. Reopening would have re-litigated the feature; a follow-up fixes the contract. That is the correct decomposition.

One ask: is there a seventh path with the same shape — --floor set to a negative number, or a non-numeric value? floor <= 0 catches negatives into the same record-only mode, which is arguably a different thing from "floor 0 deliberately": a typo'd --floor -1 should probably refuse (2/cannot tell) rather than silently become record-only. One line distinguishing == 0 from < 0 closes the last way to reach "asserted nothing" by accident.

Not verified by me: the six arms and the ctest wiring. The exit-code contract and the message are what I can check, and both are now right — with the caveat that I had the chance to catch this and read the feature instead of the code.

@bong-water-water-bong bong-water-water-bong changed the title token_rate_floor: --floor 0 is a SKIP, not a pass (fixes what #2432 merged with) tools/ci: --floor 0 is a SKIP, and a server under setsid loses its kill switch Nov 15, 2026
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Adding the durable half: tools/ci/server_launch_check.py at 4cf5ee72.

Why it is in this PR rather than its own

Both are tools/ci correctness fixes out of the same thread, and the --floor 0 one is urgent (main has the defect). A separate PR for the check would have delayed the fix, so it rides along — say the word and I will split it.

What it catches

setsid "$BIN" forward-serve -m "$MODEL" -p "$PORT" &
PID=$!
kill $PID

setsid puts the child in a new session, so $! is the setsid pid and every kill "$pid" afterwards aims at a dead process. The server survives with PPID 1, no ticket, still holding its device.

That is not hypothetical — it is my own 2026-11-15 incident. An npu-model-chart.sh run did exactly this: flm-real serve lfm3:1.7b stayed alive 22 h 44 min holding every NPU hardware context, and cl-npuperf and cl-flm got -19 on every hwctx create for most of a day. Seven kill "$pid" sites in that script, all correct-looking, all aimed at setsid. The same bug was in tools/repro-hostgemm-lock.sh (fixed, engine#2472).

And it found the real thing on its first run

Run against --include-bin, it immediately flagged ~/bin/npu-model-chart.sh.bak-1126:143 — my own stale backup of the chart, still carrying the exact line that caused the orphan. Deleted. ~/bin is where the orphan lived and where no check in this repo had ever looked, which is why a carefully built gpu_hold_tree_test.sh did not catch it: it tests gpu-hold's tree, not a lane-local script.

Design notes

  • --include-bin warns, it does not fail. Those scripts are not this repo's to own; failing a PR because a lane keeps a bad helper in ~/bin would make the check get disabled. The repo's own scripts/ and tools/ fail.
  • Refuses to pass when it scanned nothing (exit 77) — a check that green-lights an empty file list is the defect this whole PR series is about.
  • Third arm: a comment naming setsid is not the defect. That is engine#2460's lesson (tool.name in text accepted a comment) applied before it can happen here.

3 arms, 0 failed; copyright.py --check rc=0; git ls-remote = git rev-parse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant