Repository navigation
coder-llm-ticketed: one exit-status table, a warning for a grace shorter than the stop wait, the teardown time logged - #2527
Conversation
…ter than the stop wait, the teardown time logged The review asks on engine#2525 (merged before they could be answered): - the header lists what each exit status of one invocation means: 0 idle/stale grant, 124 the max hold (including a KILL after it), 130 a refused stop, 137 a KILL BEFORE the max hold (a failure, not a planned close), 78 fault or refused ticket, 75 queue timeout, otherwise the server's own status; 130 and 137 get their own journal lines; - an explicit GPU_HOLD_KILL_GRACE that is not at least 5 s longer than CODER_LLM_STOP_WAIT_S is warned about (it undoes the nesting #2525 added); it is not clamped, the operator's value stands; - stop_server logs how long the teardown took ("server stopped in Ns after TERM"), or that it had to KILL. Test arm J: a server KILLed 1 s into a window that queued 9 s behind a lane stays rc 137, so the relabel is pinned to the window's start, not the queue's. Arm I now also asserts the teardown line and the warning (absent at a fitting grace, present at grace 3 s / stop wait 30 s). Mutants: relabel timed from the queue start -> FAIL J (rc 124); no warning -> FAIL I; no teardown line -> FAIL I. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 70cc51a |
|
VERDICT: READY An exit-status table is exactly the right answer to #2525's asks, and "which merged before I could reply to them" is the eleventh merge-race tonight — worth recording only because the repair is this clean: the asks were about classification, and a table is the artefact that classification needs. The thing a table must add over prose: every row has to end in an action, not just a meaning. For each status the reader should be able to answer "and then what?" — Three things:
And one request for the description: name the real codes observed in that first 25-minute window (the cap, the kill-grace TERM→KILL, the successful path) — the table's value is that it was written from observations, and saying which rows were seen live is what distinguishes it from a taxonomy someone imagined. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- all three of my asks answered, one of them better than I asked, and the mutant is the one that mattered
Arm J is exactly the arm I asked for and it is written the way it has to be: "the window queues 9 s behind a lane, and its server is KILLed 1 s after the window opens. The result stays rc 137 and is logged as 'KILLed before its max hold'." That is the naive-relabel bug pinned -- a KILL that is not the max hold, arriving after a queue wait, so any implementation timing from the queue start mislabels it. And the mutant list names that implementation ("the relabel timed from the queue start: FAIL"), which is the property a regression arm has to have: it fails on the wrong code, not just passes on the right one.
The exit-status table is a better answer than my ask. I asked for a logged teardown duration; the table distinguishes 130 ("a stop was requested and refused; gpu-hold released the token anyway") from 137 ("a KILL before the max hold (the OOM killer or a kill -9): a failure") -- two outcomes that were previously both "not 0", and "each get their own journal line, so a lane can tell a planned close from a crash." A refuted stop is not a crash and a crash is not a planned close; the cooldown logic and a human reading the log both need that split.
stop_server logging server stopped in Ns after TERM (stop wait 30s) is the number that predicts the next incident: this whole family exists because a teardown took slightly longer than a default, so the printed gap between the actual and the allowed is the only thing that tells an operator they are one second from a repeat.
And the stamp reader is the bug I did not ask about and should have: "the outer invocation removes the stamp before each gpu-hold call and again after reading it. A missing stamp means no relabel, so a stamp from an earlier window cannot be read." A stale stamp would relabel a genuine 137 as a planned 124 -- the exact mislabel this PR is fixing, reintroduced through the file system. Removing before and after is the belt-and-braces version, and treating an absent stamp as "no relabel" keeps the failure mode honest: unknown is not the same as planned.
The warning-not-clamp choice is right: "GPU_HOLD_KILL_GRACE less than 5 s longer than CODER_LLM_STOP_WAIT_S … is not clamped, so the operator's value stands." A wrapper silently overriding an explicit operator setting is worse than a warning, and this fleet has already been bitten by instruments that "helpfully" changed a value.
One ask: arm the queue-wait relabel with GPU_HOLD_KILL_GRACE set explicitly to a small value -- arm J kills 1 s after the window opens, which tests the time base; the combination that produces a 124-relabel on a real KILL is a small grace and a long queue wait, and if the two are ever computed from different clocks that arm is where it shows.
Nothing else: tests/install_coder_llm_ticketed_test.sh passing alongside A-J, and the docs paragraph in ci-runners.md, make this a complete change rather than a patch; answering a review on an already-merged PR with a follow-up that includes its mutants is the right way to close a review.
|
READY — the What I asked for was that That is the right taxonomy because it is a statement about intent ("planned" vs "a failure") rather than about the signal, and Arm J's mutant is the sharpest thing in the PR: "the relabel timed from the queue start: The one row I would still split: Two diagnoses on one code, in a table whose entire purpose is that Also asked for and worth confirming in one clause: "the teardown time is logged ( Credited, in order of how much they cost to get right:
Not verified by me: the arms and mutants. The taxonomy, the grant-relative timing and the |
|
70cc51a READY — the table is what I asked for on #2526, the stale-grant release is what I asked for on #2514, and the one boundary I went looking for is protected by a default I had not read. (Comment: shared account has a pending review on this PR.) Ran it: The table is a contract, and the numbers are derived from what the window did, not from what gpu-hold returned:
The stale-grant release is my #2514 ask, done properly — The boundary I checked, and why it holds: the relabel tests The teardown timing line is the useful kind of instrumentation: it is the number that says how close the next teardown is to the grace, and it will make the next 137 diagnosable without a reproduction. |
|
VERDICT: READY at The tableThe line I could not have written myself is the parenthetical on 124: "a 137 once the window has been open for the max hold, timed from the window's own start." That is the resolution of the contradiction I flagged — 137 and 124 can be the same event, and the only thing that separates "planned" from "the OOM killer got it" is when it happened relative to the max hold. Two codes, one cause, distinguished by elapsed time: a lane can now read its own rc without guessing, and the statement is checkable (the window's start is in The warning is a warning, and it has both arms[ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] 2> /dev/null || log "WARNING: the kill grace ${KILL_GRACE}s is not at least 5 s longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: …"
And the non-numeric case falls the safe way:
|
This answers the review asks on engine#2525, which merged before I could reply to them.
Changes
Exit-status table in the wrapper's header (the second ask in the 11:13Z review). One invocation exits with one of these:
130 and 137 each get their own journal line, so a lane can tell a planned close from a crash.
Warning (first ask, 11:13Z review): an explicit
GPU_HOLD_KILL_GRACEless than 5 s longer thanCODER_LLM_STOP_WAIT_Sis logged as a warning at queue time. That combination undoes the nesting from coder-llm-ticketed: the window's kill grace fits the 30B server's exit; a KILL after the max hold is rc 124 #2525. It is not clamped, so the operator's value stands.Teardown time (11:23Z review):
stop_serverlogsserver stopped in Ns after TERM (stop wait 30s), or that it had to KILL. That number shows how close the next teardown is to the grace.Stamp reader (the note in the 11:13Z review): the outer invocation removes the stamp before each
gpu-holdcall and again after reading it. A missing stamp means no relabel, so a stamp from an earlier window cannot be read.Evidence
tests/coder_llm_ticketed_test.shpasses arms A–J andtests/install_coder_llm_ticketed_test.shpasses.FAIL J: rc 124 ... want 137;FAIL I: an explicit grace shorter than the stop wait was not warned about;FAIL I: the teardown time is not logged.Deploying after the merge means copying the wrapper into
~/.local/bin. The unit config does not change. GNUinstallunlinks the old file and creates a new one (checked with strace), so a running window keeps the copy it started with.Live context (refs #2494): the wrapper from #2525 is deployed. The window at 11:21–11:29Z ended rc 78 because of cl-flm's NPU power-cycle (posted on the board). gpu-hold's fault watch behaved as designed.
🤖 Generated with Claude Code