Skip to content

coder-llm-ticketed: the window's kill grace fits the 30B server's exit; a KILL after the max hold is rc 124 - #2525

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-kill-grace
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-kill-grace

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Live finding from the first ticketed coder-llm window (refs engine#2494; follows #2496/#2514/#2515).

What happened

The window ran from 10:31:46Z to 10:56:57Z on 2026-11-15, and its 25 min max hold cut it. gpu-hold sent TERM and the 30B server logged 1bit serve: stopping at once. Its HRX teardown then took longer than gpu-hold's default 10 s GPU_HOLD_KILL_GRACE, so gpu-hold KILLed the window:

06:56:47.478 gpu-hold[3661968] max hold of 25 min reached
06:56:47.601 1bit serve: stopping
06:56:57.707 ... 3931587 Killed  env GPU_HOLD_TICKET=... (the window)
06:56:57.726 gpu-hold[3661968] command exited 137 after 1511s
06:56:57.738 coder-llm-ticketed[3661911] window ended rc 137

The token release and the FIFO were fine: cl-hrxkern took the GPU at 10:56:58Z, and coder-llm re-queued at the back at 10:57:13Z. No server was left behind. The defects were the unclean stop and the wrong outcome label (137, not the max hold's 124).

Change

  • CODER_LLM_STOP_WAIT_S (default 30, the old fixed value) sets how long the window waits for its server after TERM.
  • gpu-hold now runs with GPU_HOLD_KILL_GRACE = STOP_WAIT + 10, so the window ends its own server before gpu-hold KILLs the window. CODER_LLM_KILL_GRACE_S or an explicit GPU_HOLD_KILL_GRACE overrides it.
  • An rc 137 is reported as the max hold (rc 124) once the window has been open for the max hold. That time is counted from the window's own start, not from when it began queueing, so another KILL after a long queue wait is not relabelled.
  • docs/ci-runners.md: one paragraph.

Evidence

  • tests/coder_llm_ticketed_test.sh, under thermal-run, passes all arms (A–I) in 92 s:
    • new arm I(1): a server that takes 12 s to exit after TERM gets its grace and exits cleanly, rc 124, logged as the max hold;
    • new arm I(2): with an explicit grace too short for that server, the KILL is still rc 124 and logged "KILLed after the max hold".
  • Mutants:
    • without the GPU_HOLD_KILL_GRACE=$KILL_GRACE prefix the test prints FAIL I: the slow server was KILLed before it could exit;
    • without the relabel it prints FAIL I: rc 137 ... want 124 and FAIL I: the KILL after the max hold was not reported as such.

Deploy after merge: copy the wrapper into ~/.local/bin, as for #2514. The drop-in does not change.

🤖 Generated with Claude Code

…t; a KILL after the max hold is rc 124

The first live window (2026-11-15, 10:31:46Z-10:56:57Z) was cut at its 25 min max hold. The server got TERM and
logged "stopping" at once, but its HRX teardown took longer than gpu-hold's default 10 s GPU_HOLD_KILL_GRACE, so
gpu-hold KILLed the window: rc 137, logged "window ended rc 137", an unclean stop instead of the max-hold close.
The token release and the FIFO were not affected (the next lane took the GPU 1 s later).

- CODER_LLM_STOP_WAIT_S (default 30, was a fixed 30) is how long the window waits for its server after TERM;
- gpu-hold runs with GPU_HOLD_KILL_GRACE = STOP_WAIT + 10 (CODER_LLM_KILL_GRACE_S, or an explicit
  GPU_HOLD_KILL_GRACE, overrides), so the window can end its own server before gpu-hold KILLs it;
- a rc 137 once the window has been open for the max hold (timed from the window's own start, not the queue
  wait) is reported as the max hold, rc 124.

Test arm I: a server that takes 12 s to exit gets its grace and exits cleanly (rc 124); with a grace too short,
the KILL is still rc 124 and logged as such. Mutants: dropping the grace -> FAIL I (the server is KILLed);
dropping the relabel -> FAIL I (rc 137).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 9cc7642

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2525

This is the fourth live finding from the #2496 switch, and the most concrete yet: the window ran 10:31:46Z → 10:56:57Z, gpu-hold's 25-minute cap fired, the server logged 1bit serve: stopping immediately — and then HRX teardown took longer than gpu-hold's 10 s GPU_HOLD_KILL_GRACE, so the window was kill -9'd 10 s later.

The distinction that matters for the fix: this is not the server failing to stop, it is the server stopping correctly and slowly — HRX teardown is doing real work (freeing queues, draining) and gpu-hold's grace is calibrated for a process that ignores TERM. So the correct change is one of:

  1. per-ticket grace — a ticket that knows its teardown exceeds 10 s declares it (GPU_HOLD_KILL_GRACE already exists as an env override, so the mechanism is there); or
  2. watch for teardown progress — if HRX logs teardown steps, extend while the log advances, kill only when it stalls. That is the stronger version, because it needs no calibration per process.

Three things to get right:

  • kill -9 during teardown is the dangerous case, not the slow one — an HRX teardown interrupted at an arbitrary point leaves the queue/firmware state the next window inherits. So the failure this PR is fixing can produce the following window's GPU fault, which is precisely the kind of inheritance that makes a fault look unreproducible. Worth one sentence, because it changes the priority of this fix;
  • log the teardown duration on every window, so the grace value is data rather than a guess (the 25-minute cap is already logged; make the teardown time the same kind of fact);
  • the docs line should say which value is the default and who may raise it — docs/ci-runners.md is now the third file in this series to be edited by a live finding, and the person reading it will be someone whose window was just killed.

And the process point, said plainly because it is now four PRs: #2496 went live and the fleet discovered its consequences by breaking (#2512, #2514, #2515, this). Each fix is good; the aggregate argues for one "what changes when coder-llm is ticketed" page that lists ticket cap, kill grace, queue wait, and who owns each — the four PRs would then be footnotes to one decision instead of four separate discoveries.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the change, with two asks — one of them about the PR's own rc statements, which currently contradict each other.

The nesting is the right idea and it holds by construction

STOP_WAIT=${CODER_LLM_STOP_WAIT_S:-30}
KILL_GRACE=${GPU_HOLD_KILL_GRACE:-${CODER_LLM_KILL_GRACE_S:-$(( STOP_WAIT + 10 ))}}

Inner wait < outer grace is the only ordering that lets the graceful path win: the wrapper asks its server to stop, gets 30 s, and gpu-hold only KILLs the window at 40 s — so a slow-but-healthy 30B exit is not preempted by the outer backstop. Deriving the outer from the inner (STOP_WAIT + 10) rather than hard-coding both means the two cannot drift when the server gets slower. That is the same lesson this lane has applied twice today, correctly applied in the other direction: the grace has to be sized for the operation it contains.

Ask 1: an explicit GPU_HOLD_KILL_GRACE can silently undo it

"an explicit GPU_HOLD_KILL_GRACE in the environment wins." So GPU_HOLD_KILL_GRACE=10 CODER_LLM_STOP_WAIT_S=30 restores exactly the bug this PR fixes — the outer KILL arrives 20 s before the inner wait gives up — and nothing says so. One guard, in the wrapper, before it starts the window:

[ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] || log "::warning::GPU_HOLD_KILL_GRACE=${KILL_GRACE}s is not longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: gpu-hold will KILL the window before the server can exit"

A clamp would be worse (it would override an operator's explicit choice); a warning names the mistake at the moment it is made.

Ask 2: the PR states two different exit codes for the same event

"The 30B server needed more than gpu-hold's default 10 s at the first live max hold (2026-11-15 10:56Z), which ended rc 137."
"A KILL after the max hold is still reported as rc 124."

Those cannot both describe the max-hold path, and gpu-hold's own comment draws the distinction that matters (:512-514): "a deadline is a timeout (124, as timeout(1) says); a stop is not — the job was asked to end and refused to die, which is 130 … a caller that tells the two apart must not read a stop as a job that hit its own limit."

So there are at least three codes in play — 124 deadline, 130 refused stop, 137 killed by SIGKILL — and this PR's subject is which one a lane sees. Please put the mapping in the doc block, one line each:

  • max hold reached and the tree KILLed → 124 (deadline);
  • a stop requested and refused → 130;
  • the wrapper's own KILL of its server surfacing as the command's status → 137, and when a lane should read that as "the window ended on purpose" rather than "my job crashed".

Right now a lane that sees 137 cannot tell whether its job faulted or its window hit the max hold — which is the exact confusion :512-514 was written to prevent, and the exact confusion this PR's own two sentences create.

Not the bug, worth knowing

The CODER_LLM_WINDOW_STAMP line (date +%s > "$CODER_LLM_WINDOW_STAMP") is guarded by -n, so an unset variable is a no-op — fine, but if the stamp is meant to be read by anything, that reader needs the same guard or it will read a file from a previous day.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 11:15
@bong-water-water-bong
bong-water-water-bong merged commit 07685c2 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/coder-llm-kill-grace branch November 15, 2026 11:20
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verdict as an issue comment: the shared identity already holds a pending review on this PR, so POST /reviews returns 422 ("User can only have one pending review per pull request"). Content unchanged.

VERDICT: READY -- and STOP_WAIT + 10 is the right arithmetic because the two deadlines belong to different owners

The finding is operational and exact: "its HRX teardown then took longer than gpu-hold's default 10 s GPU_HOLD_KILL_GRACE, so gpu-hold KILLed the window" -- 1bit serve: stopping at :47.601, the window Killed at :57.707, rc 137 after 1511 s where the outcome should have been the max hold's 124. Two defects, both real: an unclean stop (HRX teardown cut mid-flight) and a wrong outcome label that the cooldown logic and any reader would both believe.

GPU_HOLD_KILL_GRACE = STOP_WAIT + 10 is the correct shape because the window must finish its own server before the outer deadline fires -- the window is the party that knows its server exists, and giving gpu-hold a grace strictly larger than the window's own wait makes the stop orderly instead of a race whose winner is a scheduler. The CODER_LLM_KILL_GRACE_S / explicit-env override keeps the escape hatch for a server that needs longer.

And the relabel is where the real care is: "that time is counted from the window's own start, not from when it began queueing, so another KILL after a long queue wait is not relabelled." A KILL 30 s after a 3-hour queue wait is not the max hold, and a naive rc 137 -> 124 map would have silently relabelled it and hidden a genuine failure. That distinction is the whole reason this is a good change rather than a sed.

Arm I(1) is the arm that matters: a server that takes 12 s to exit after TERM gets its grace and exits cleanly with rc 124. That is precisely the failing case in the log, pinned as a test, and 12 s sits between the old 10 s and the new 40 s -- so the arm fails on the old code and passes on the new, which is the only property that makes a regression arm worth its runtime.

One ask: assert the label and the touch separately in the 137-after-long-queue arm. The body describes the start-time rule in prose and arm I(1) covers the max-hold path; if a second arm fires the KILL after a queue wait and asserts the result stays a genuine rc 137, the relabel rule is pinned rather than explained. Without it, the next simplification (rc 137 && window_open > max computed from the wrong timestamp) passes every current arm.

Second, smaller: log the teardown duration (stopping -> exited in 11.3 s) in the window's end line. This PR exists because a teardown took slightly longer than a default; the number that would tell the next reader they are one second away from a repeat is exactly that duration, and the window already computes it.

Nothing else: "the token release and the FIFO were fine … no server was left behind" is the check that this did not become the 11-08 orphan class, and verifying it before describing the fix is the order that makes the rest trustworthy. CODER_LLM_STOP_WAIT_S defaulting to the old fixed value keeps every existing window's behaviour unchanged.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Kill Grace Calculation

The calculation of KILL_GRACE may lead to a race condition where the window's own server termination grace is not sufficient to prevent gpu-hold from sending a KILL signal before the server has fully exited. This could result in an unclean shutdown if the server requires more than STOP_WAIT + 10 seconds to terminate.

KILL_GRACE=${GPU_HOLD_KILL_GRACE:-${CODER_LLM_KILL_GRACE_S:-$(( STOP_WAIT + 10 ))}}
Max Hold Grace Handling

The logic for detecting and relabeling a KILL signal (rc 137) as a max hold (rc 124) relies on the CODER_LLM_WINDOW_STAMP file being created and read correctly. If this file is not properly managed or if the timing is off, the window might not be correctly labeled as having hit the max hold, leading to incorrect exit codes.

if [ "$rc" = 137 ] && [ -n "$opened" ] && awk -v e=$(( $(date +%s) - opened )) -v m="$MAX_HOLD" 'BEGIN { exit !(e >= m * 60) }'; then
    log "the window was KILLed after the max hold (rc 137; raise CODER_LLM_STOP_WAIT_S if the server needs longer to exit)"
    rc=124

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both review asks (11:13Z: the grace warning and the exit-status table; 11:23Z: a 137-after-queue arm and the teardown time) are answered in engine#2527, since this merged first. The stamp-reader note is covered there too: the outer invocation removes the stamp before each gpu-hold call, and a missing stamp means no relabel.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and one hazard in the relabel, because this is a memory window and rc 137 has two causes.

The finding is real and well read from the log: TERM at 06:56:47.478, Killed at 06:56:57.707, 10 s of grace for a 30B to unload, exit 137 after 1511s. A SIGKILL during teardown is exactly the thing that could leave a queue pair unmapped while the device is still writing — i.e. the mechanism #2494 is chasing — so giving the window enough time to end its own server before gpu-hold reaches for KILL is the right end of the problem, and STOP_WAIT + 10 puts the window's grace strictly inside the hold's.

The relabel is where I would add one condition. "An rc 137 is reported as the max hold (rc 124) once the window has been open for the max hold" — but 137 means "SIGKILL", and gpu-hold's grace is one of at least two things that sends it. The other one is the kernel OOM killer, and this is a memory window: a run that stopped coder-llm to free memory, then loaded something large, is precisely where an OOM kill is most likely. In that run:

  • elapsed is past the max hold (the window was long, as memory windows are),
  • the child died with rc 137,
  • and the relabel would report it as max hold — misattributing an OOM to the ticket's timeout, in the one report a person reads to decide whether the box is stable.

So gate the relabel on evidence that the KILL was gpu-hold's, not on timing plus exit code: gpu-hold prints its own Killed … line (it is in your log excerpt), and the wrapper already parses that stream. gunzip-free version: relabel only when the log contains the hold's KILL line and the elapsed exceeds the max hold; if the child was killed by something else, leave 137 alone and say so, because "the kernel killed it" and "the ticket ran out" are different diagnoses with different fixes.

Arm I(2) covers the mechanical case but not that one, and the arm is otherwise good: "with an explicit grace too short for that server, the KILL is still rc 124 and logged 'KIL…'" proves the relabel does not hide the KILL, which is the property that makes a relabel legitimate at all. The mutant pair (FAIL I: the slow server was KILLed without the grace prefix; FAIL I: rc 137 … want 124 without the relabel) makes both directions real — one without the fix, one without the reporting — and 92 s for arms A–I is a cheap suite.

Two things I want credited explicitly:

  • The FIFO and token were never at risk, and the evidence says so: "cl-hrxkern took the GPU at 10:56:58Z, and coder-llm re-queued" — one second after the window ended. That is the check that separates "the window overran" from "the window broke the queue", and it is reported without being asked for.
  • CODER_LLM_STOP_WAIT_S defaulting to 30 keeps the old fixed value, so the change is behaviour-preserving where the old value sufficed and only extends where it did not.

One derivation question: is GPU_HOLD_KILL_GRACE = STOP_WAIT + 10 computed from the same variable in the same process, or are these two defaults in two files that have to be kept in sync by hand? With +10 the margin is invisible; if a future change raises STOP_WAIT_S to 120 and the grace is a separate literal, the window goes back to being KILLed mid-teardown and nothing fails — the arm would still pass because the test sets both. If it is derived, say so in one line; that is what makes the margin safe to change.

Not verified by me: the run and arms A–I. The log reading, the grace ordering and the relabel's two-cause hazard are what I can check, and I would land all three plus the OOM condition.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…ter than the stop wait, the teardown time logged (#2527)

The review asks on engine#2525 (merged before they could be answered):

- the header lists what each exit status of one invocation means: 0 idle/stale grant, 124 the max hold (including
  a KILL after it), 130 a refused stop, 137 a KILL BEFORE the max hold (a failure, not a planned close), 78 fault or
  refused ticket, 75 queue timeout, otherwise the server's own status; 130 and 137 get their own journal lines;
- an explicit GPU_HOLD_KILL_GRACE that is not at least 5 s longer than CODER_LLM_STOP_WAIT_S is warned about
  (it undoes the nesting #2525 added); it is not clamped, the operator's value stands;
- stop_server logs how long the teardown took ("server stopped in Ns after TERM"), or that it had to KILL.

Test arm J: a server KILLed 1 s into a window that queued 9 s behind a lane stays rc 137, so the relabel is pinned to
the window's start, not the queue's. Arm I now also asserts the teardown line and the warning (absent at a fitting
grace, present at grace 3 s / stop wait 30 s). Mutants: relabel timed from the queue start -> FAIL J (rc 124); no
warning -> FAIL I; no teardown line -> FAIL I.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant