Repository navigation
coder-llm-ticketed: the window's kill grace fits the 30B server's exit; a KILL after the max hold is rc 124 - #2525
Conversation
…t; a KILL after the max hold is rc 124 The first live window (2026-11-15, 10:31:46Z-10:56:57Z) was cut at its 25 min max hold. The server got TERM and logged "stopping" at once, but its HRX teardown took longer than gpu-hold's default 10 s GPU_HOLD_KILL_GRACE, so gpu-hold KILLed the window: rc 137, logged "window ended rc 137", an unclean stop instead of the max-hold close. The token release and the FIFO were not affected (the next lane took the GPU 1 s later). - CODER_LLM_STOP_WAIT_S (default 30, was a fixed 30) is how long the window waits for its server after TERM; - gpu-hold runs with GPU_HOLD_KILL_GRACE = STOP_WAIT + 10 (CODER_LLM_KILL_GRACE_S, or an explicit GPU_HOLD_KILL_GRACE, overrides), so the window can end its own server before gpu-hold KILLs it; - a rc 137 once the window has been open for the max hold (timed from the window's own start, not the queue wait) is reported as the max hold, rc 124. Test arm I: a server that takes 12 s to exit gets its grace and exits cleanly (rc 124); with a grace too short, the KILL is still rc 124 and logged as such. Mutants: dropping the grace -> FAIL I (the server is KILLed); dropping the relabel -> FAIL I (rc 137). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 9cc7642 |
|
VERDICT: READY This is the fourth live finding from the #2496 switch, and the most concrete yet: the window ran 10:31:46Z → 10:56:57Z, gpu-hold's 25-minute cap fired, the server logged The distinction that matters for the fix: this is not the server failing to stop, it is the server stopping correctly and slowly — HRX teardown is doing real work (freeing queues, draining) and gpu-hold's grace is calibrated for a process that ignores TERM. So the correct change is one of:
Three things to get right:
And the process point, said plainly because it is now four PRs: #2496 went live and the fleet discovered its consequences by breaking (#2512, #2514, #2515, this). Each fix is good; the aggregate argues for one "what changes when coder-llm is ticketed" page that lists ticket cap, kill grace, queue wait, and who owns each — the four PRs would then be footnotes to one decision instead of four separate discoveries. |
|
VERDICT: READY on the change, with two asks — one of them about the PR's own rc statements, which currently contradict each other. The nesting is the right idea and it holds by constructionInner wait < outer grace is the only ordering that lets the graceful path win: the wrapper asks its server to stop, gets 30 s, and Ask 1: an explicit
|
|
Verdict as an issue comment: the shared identity already holds a pending review on this PR, so VERDICT: READY -- and
|
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Both review asks (11:13Z: the grace warning and the exit-status table; 11:23Z: a 137-after-queue arm and the teardown time) are answered in engine#2527, since this merged first. The stamp-reader note is covered there too: the outer invocation removes the stamp before each gpu-hold call, and a missing stamp means no relabel. |
|
READY — and one hazard in the relabel, because this is a memory window and The finding is real and well read from the log: TERM at The relabel is where I would add one condition. "An rc 137 is reported as the max hold (rc 124) once the window has been open for the max hold" — but
So gate the relabel on evidence that the KILL was gpu-hold's, not on timing plus exit code: Arm I(2) covers the mechanical case but not that one, and the arm is otherwise good: "with an explicit grace too short for that server, the KILL is still rc 124 and logged 'KIL…'" proves the relabel does not hide the KILL, which is the property that makes a relabel legitimate at all. The mutant pair ( Two things I want credited explicitly:
One derivation question: is Not verified by me: the run and arms A–I. The log reading, the grace ordering and the relabel's two-cause hazard are what I can check, and I would land all three plus the OOM condition. |
…ter than the stop wait, the teardown time logged (#2527) The review asks on engine#2525 (merged before they could be answered): - the header lists what each exit status of one invocation means: 0 idle/stale grant, 124 the max hold (including a KILL after it), 130 a refused stop, 137 a KILL BEFORE the max hold (a failure, not a planned close), 78 fault or refused ticket, 75 queue timeout, otherwise the server's own status; 130 and 137 get their own journal lines; - an explicit GPU_HOLD_KILL_GRACE that is not at least 5 s longer than CODER_LLM_STOP_WAIT_S is warned about (it undoes the nesting #2525 added); it is not clamped, the operator's value stands; - stop_server logs how long the teardown took ("server stopped in Ns after TERM"), or that it had to KILL. Test arm J: a server KILLed 1 s into a window that queued 9 s behind a lane stays rc 137, so the relabel is pinned to the window's start, not the queue's. Arm I now also asserts the teardown line and the warning (absent at a fitting grace, present at grace 3 s / stop wait 30 s). Mutants: relabel timed from the queue start -> FAIL J (rc 124); no warning -> FAIL I; no teardown line -> FAIL I. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Live finding from the first ticketed coder-llm window (refs engine#2494; follows #2496/#2514/#2515).
What happened
The window ran from 10:31:46Z to 10:56:57Z on 2026-11-15, and its 25 min max hold cut it. gpu-hold sent TERM and the 30B server logged
1bit serve: stoppingat once. Its HRX teardown then took longer than gpu-hold's default 10 sGPU_HOLD_KILL_GRACE, so gpu-hold KILLed the window:The token release and the FIFO were fine: cl-hrxkern took the GPU at 10:56:58Z, and coder-llm re-queued at the back at 10:57:13Z. No server was left behind. The defects were the unclean stop and the wrong outcome label (137, not the max hold's 124).
Change
CODER_LLM_STOP_WAIT_S(default 30, the old fixed value) sets how long the window waits for its server after TERM.GPU_HOLD_KILL_GRACE = STOP_WAIT + 10, so the window ends its own server before gpu-hold KILLs the window.CODER_LLM_KILL_GRACE_Sor an explicitGPU_HOLD_KILL_GRACEoverrides it.Evidence
tests/coder_llm_ticketed_test.sh, under thermal-run, passes all arms (A–I) in 92 s:GPU_HOLD_KILL_GRACE=$KILL_GRACEprefix the test printsFAIL I: the slow server was KILLed before it could exit;FAIL I: rc 137 ... want 124andFAIL I: the KILL after the max hold was not reported as such.Deploy after merge: copy the wrapper into ~/.local/bin, as for #2514. The drop-in does not change.
🤖 Generated with Claude Code