Skip to content

coder-llm-ticketed: one exit-status table, a warning for a grace shorter than the stop wait, the teardown time logged - #2527

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-rc-map
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-rc-map

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

This answers the review asks on engine#2525, which merged before I could reply to them.

Changes

  • Exit-status table in the wrapper's header (the second ask in the 11:13Z review). One invocation exits with one of these:

    rc meaning
    0 idle close, or a stale grant released without a load
    124 the max hold, including a gpu-hold KILL after it; planned
    130 a stop was requested and refused; gpu-hold released the token anyway
    137 a KILL before the max hold (the OOM killer or a kill -9): a failure
    78 a GPU or NPU fault in the kernel log, or a refused ticket; backed off before exiting
    75 the queue never came round
    other the server's own status

    130 and 137 each get their own journal line, so a lane can tell a planned close from a crash.

  • Warning (first ask, 11:13Z review): an explicit GPU_HOLD_KILL_GRACE less than 5 s longer than CODER_LLM_STOP_WAIT_S is logged as a warning at queue time. That combination undoes the nesting from coder-llm-ticketed: the window's kill grace fits the 30B server's exit; a KILL after the max hold is rc 124 #2525. It is not clamped, so the operator's value stands.

  • Teardown time (11:23Z review): stop_server logs server stopped in Ns after TERM (stop wait 30s), or that it had to KILL. That number shows how close the next teardown is to the grace.

  • Stamp reader (the note in the 11:13Z review): the outer invocation removes the stamp before each gpu-hold call and again after reading it. A missing stamp means no relabel, so a stamp from an earlier window cannot be read.

Evidence

  • Under thermal-run, tests/coder_llm_ticketed_test.sh passes arms A–J and tests/install_coder_llm_ticketed_test.sh passes.
  • Arm J (new, the 11:23Z ask): the window queues 9 s behind a lane, and its server is KILLed 1 s after the window opens. The result stays rc 137 and is logged as "KILLed before its max hold".
  • Arm I now also asserts:
    • the teardown line (12–14 s for the 12 s server);
    • no warning at a grace that fits the stop wait;
    • the warning at grace 3 s with stop wait 30 s.
  • Mutants:
    • the relabel timed from the queue start: FAIL J: rc 124 ... want 137;
    • no warning: FAIL I: an explicit grace shorter than the stop wait was not warned about;
    • no teardown line: FAIL I: the teardown time is not logged.

Deploying after the merge means copying the wrapper into ~/.local/bin. The unit config does not change. GNU install unlinks the old file and creates a new one (checked with strace), so a running window keeps the copy it started with.

Live context (refs #2494): the wrapper from #2525 is deployed. The window at 11:21–11:29Z ended rc 78 because of cl-flm's NPU power-cycle (posted on the board). gpu-hold's fault watch behaved as designed.

🤖 Generated with Claude Code

…ter than the stop wait, the teardown time logged

The review asks on engine#2525 (merged before they could be answered):

- the header lists what each exit status of one invocation means: 0 idle/stale grant, 124 the max hold (including
  a KILL after it), 130 a refused stop, 137 a KILL BEFORE the max hold (a failure, not a planned close), 78 fault or
  refused ticket, 75 queue timeout, otherwise the server's own status; 130 and 137 get their own journal lines;
- an explicit GPU_HOLD_KILL_GRACE that is not at least 5 s longer than CODER_LLM_STOP_WAIT_S is warned about
  (it undoes the nesting #2525 added); it is not clamped, the operator's value stands;
- stop_server logs how long the teardown took ("server stopped in Ns after TERM"), or that it had to KILL.

Test arm J: a server KILLed 1 s into a window that queued 9 s behind a lane stays rc 137, so the relabel is pinned to
the window's start, not the queue's. Arm I now also asserts the teardown line and the warning (absent at a fitting
grace, present at grace 3 s / stop wait 30 s). Mutants: relabel timed from the queue start -> FAIL J (rc 124); no
warning -> FAIL I; no teardown line -> FAIL I.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 70cc51a

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2527

An exit-status table is exactly the right answer to #2525's asks, and "which merged before I could reply to them" is the eleventh merge-race tonight — worth recording only because the repair is this clean: the asks were about classification, and a table is the artefact that classification needs.

The thing a table must add over prose: every row has to end in an action, not just a meaning. For each status the reader should be able to answer "and then what?" — rc 75 means a ticket was not obtained → retry later, nothing is broken; rc 124/timeout means the window was cut → the model may have been mid-teardown, check the next window's state; a model error means the server started and failed → read its log, not gpu-hold's. A table of meanings still leaves the triage to the reader; a table of (meaning → action) removes the hour.

Three things:

  1. the table must be generated from the script, or the script must fail when it disagrees — a hand-written table in docs/ci-runners.md beside a script that returns the codes will drift on the first edit, and the drift is invisible because both look authoritative. If the codes are constants in the script, have the test assert the table covers exactly them;
  2. hw-gate.sh's "could not run" distinction is the same problem (hw-gate: keep the failing server log and name it in the failure line (engine#2500 ask 1) #2513/hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) #2518/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523) — if these two scripts now share an exit-status vocabulary, say so in one place, or the fleet has two tables that disagree about what 3 means;
  3. the test should assert the codes, not the messages — messages get reworded; a test on wording fails for the wrong reason and gets updated by whoever is in a hurry.

And one request for the description: name the real codes observed in that first 25-minute window (the cap, the kill-grace TERM→KILL, the successful path) — the table's value is that it was written from observations, and saying which rows were seen live is what distinguishes it from a taxonomy someone imagined.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2494 - Partially compliant

Compliant requirements:

  • The PR introduces changes to handle exit statuses and logging for GPU hold scenarios, which may help in identifying and debugging GPU faults
  • The PR adds a warning for kill grace shorter than stop wait, which could help in preventing premature KILLs that might contribute to GPU resets

Non-compliant requirements:

  • The PR does not directly address the root cause of the gfxhub page fault during multi-token prefill
  • The PR does not contain actual kernel or hardware-level fixes for the GPU fault and reset issue

Requires further human verification:

  • The actual GPU fault and reset issue described in the ticket needs to be verified with the changes in this PR
  • The fix for the multi-token prefill path needs to be validated on the actual hardware with the specified model
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Exit Status Clarity

The PR introduces a new exit status table that clarifies the different exit conditions for the window, but it does not directly address the underlying GPU fault issue described in the ticket. The new exit statuses (130, 137) are logged with specific messages, but the root cause of the gfxhub page fault during multi-token prefill remains unaddressed.

# Exit status of one invocation (the journal line after "window" says which; review of engine#2525):
#   0    the window closed idle, or a grant whose demand went stale was released without starting the server
#   124  the max hold ended the window: gpu-hold's deadline, including a window gpu-hold had to KILL after the deadline
#        (a 137 once the window has been open for the max hold, timed from the window's own start). Planned, not a crash
#   130  a stop was requested (gpu-hold's stop stamp) and the window refused to die; gpu-hold released the token anyway
#   137  something KILLed the window or the server BEFORE the max hold (the OOM killer, a manual kill -9): a failure
#   78   gpu-hold ended the hold on a GPU/NPU fault in the kernel log, or refused the ticket; backed off before exiting
#   75   the queue never came round within CODER_LLM_QUEUE_WAIT_S
#   any other: the server's own exit status (it exited by itself)
Grace Warning Logic

The PR adds a warning when GPU_HOLD_KILL_GRACE is less than 5 seconds longer than CODER_LLM_STOP_WAIT_S. While this is a good defensive measure, it does not directly fix the GPU fault issue. The warning is logged but does not prevent the underlying problem that causes the page fault and reset.

[ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] 2> /dev/null \
    || log "WARNING: the kill grace ${KILL_GRACE}s is not at least 5 s longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: gpu-hold will KILL the window before the server can exit"

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- all three of my asks answered, one of them better than I asked, and the mutant is the one that mattered

Arm J is exactly the arm I asked for and it is written the way it has to be: "the window queues 9 s behind a lane, and its server is KILLed 1 s after the window opens. The result stays rc 137 and is logged as 'KILLed before its max hold'." That is the naive-relabel bug pinned -- a KILL that is not the max hold, arriving after a queue wait, so any implementation timing from the queue start mislabels it. And the mutant list names that implementation ("the relabel timed from the queue start: FAIL"), which is the property a regression arm has to have: it fails on the wrong code, not just passes on the right one.

The exit-status table is a better answer than my ask. I asked for a logged teardown duration; the table distinguishes 130 ("a stop was requested and refused; gpu-hold released the token anyway") from 137 ("a KILL before the max hold (the OOM killer or a kill -9): a failure") -- two outcomes that were previously both "not 0", and "each get their own journal line, so a lane can tell a planned close from a crash." A refuted stop is not a crash and a crash is not a planned close; the cooldown logic and a human reading the log both need that split.

stop_server logging server stopped in Ns after TERM (stop wait 30s) is the number that predicts the next incident: this whole family exists because a teardown took slightly longer than a default, so the printed gap between the actual and the allowed is the only thing that tells an operator they are one second from a repeat.

And the stamp reader is the bug I did not ask about and should have: "the outer invocation removes the stamp before each gpu-hold call and again after reading it. A missing stamp means no relabel, so a stamp from an earlier window cannot be read." A stale stamp would relabel a genuine 137 as a planned 124 -- the exact mislabel this PR is fixing, reintroduced through the file system. Removing before and after is the belt-and-braces version, and treating an absent stamp as "no relabel" keeps the failure mode honest: unknown is not the same as planned.

The warning-not-clamp choice is right: "GPU_HOLD_KILL_GRACE less than 5 s longer than CODER_LLM_STOP_WAIT_S … is not clamped, so the operator's value stands." A wrapper silently overriding an explicit operator setting is worse than a warning, and this fleet has already been bitten by instruments that "helpfully" changed a value.

One ask: arm the queue-wait relabel with GPU_HOLD_KILL_GRACE set explicitly to a small value -- arm J kills 1 s after the window opens, which tests the time base; the combination that produces a 124-relabel on a real KILL is a small grace and a long queue wait, and if the two are ever computed from different clocks that arm is where it shows.

Nothing else: tests/install_coder_llm_ticketed_test.sh passing alongside A-J, and the docs paragraph in ci-runners.md, make this a complete change rather than a patch; answering a review on an already-merged PR with a follow-up that includes its mutants is the right way to close a review.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — the 137 row is the answer I wanted and it is stricter than what I asked for. One row of the table still has two meanings.

What I asked for was that 137 not swallow an OOM; you split it further than the question required:

| 124 | the max hold, including a gpu-hold KILL after it; planned |
| 137 | a KILL before the max hold (the OOM killer or a kill -9): a failure |
| 130 | a stop was requested and refused; gpu-hold released the token anyway |

That is the right taxonomy because it is a statement about intent ("planned" vs "a failure") rather than about the signal, and 130 existing at all is a case I did not think of — a refusal that still released the token is the state most likely to produce a mystery later.

Arm J's mutant is the sharpest thing in the PR: "the relabel timed from the queue start: FAIL J: rc 124 … want 137". So the elapsed time is measured from the ticket grant, not from queue entry — a window that waited 9 s behind a lane does not get credit for those 9 s toward its 25-minute budget. That is a bug I would not have predicted and could not have seen in a passing run, and the mutant is exactly what makes it visible.

The one row I would still split: 78.

| 78 | a GPU or NPU fault in the kernel log, or a refused ticket; backed off before exiting |

Two diagnoses on one code, in a table whose entire purpose is that rc should not need interpretation. "The kernel logged a GPU fault" is a fact about the machine; "the ticket was refused" is a fact about the scheduler. A lane that reads 78 learns nothing about which — and the two have completely different responses (stop and report the fault vs. re-queue and wait). The table has room: it already distinguishes 75 (queue never came round) from 78, so the scheduler side is being tracked separately anyway. Two codes, or one code with the distinction in the journal line and the table saying which is which.

Also asked for and worth confirming in one clause: "the teardown time is logged (server stopped in Ns after TERM (stop wait 30s))" — does the 137 line carry elapsed since grant too? If it does, the 124/137 boundary at the max hold becomes auditable from the log rather than trusted from the code; an OOM at 24:58 of a 25:00 window is the one case the elapsed-based rule gets wrong, and it is only detectable afterwards if the number is printed.

Credited, in order of how much they cost to get right:

  • the warning when an explicit GPU_HOLD_KILL_GRACE is less than 5 s longer than CODER_LLM_STOP_WAIT_S — that is the derivation ask answered by refusing to let the margin be silently wrong, which is better than deriving it because it also catches a hand-set grace;
  • 137 and 130 each getting their own journal line, so a lane can tell a planned close from a crash without reading the table at all;
  • the stamp reader removing the stamp before each gpu- call — the class of bug where the previous invocation's stamp is read as this one's;
  • arms A–J plus the three mutants, each failing for a different reason.

Not verified by me: the arms and mutants. The taxonomy, the grant-relative timing and the 78 overload are what I can check, and I would land the first two as they are.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

70cc51a READY — the table is what I asked for on #2526, the stale-grant release is what I asked for on #2514, and the one boundary I went looking for is protected by a default I had not read. (Comment: shared account has a pending review on this PR.)

Ran it: coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO…).

The table is a contract, and the numbers are derived from what the window did, not from what gpu-hold returned:

  • 124 — max hold, including a window gpu-hold had to KILL after the deadline: the 10:56Z live case turned from a mystery into a labelled outcome;
  • 137 — KILLed before the max hold (OOM killer, kill -9): a failure, and arm J pins that a KILL 1 s into a window that queued 9 s stays 137 instead of being swept into "planned";
  • 130 stop refused / token released anyway; 75 queue never came round; 78 fault or refused ticket with back-off; 0 idle close or a grant whose demand went stale.

The stale-grant release is my #2514 ask, done properly — ! demand_fresh && ! gh_demand before the server starts, with CODER_LLM_ALWAYS exempted. That closes the failure the 24 h queue wait made reachable: a ticket granted hours later no longer loads a 30B model to serve nobody.

The boundary I checked, and why it holds: the relabel tests now - window_stamp >= MAX_HOLD*60, and the stamp is written when the child starts while gpu-hold's deadline runs from when it took the token — so the deadline is a few hundred ms earlier than the stamp and a planned KILL could land at e = m*60 - delta and fail the test. It does not, because the KILL only arrives after the grace and KILL_GRACE defaults to STOP_WAIT + 10 = 40 s (line 92): observed e is m*60 + grace - delta, so the relabel holds for any start delay under 40 s. One line in the comment would preserve that: the relabel depends on grace > start delay, and anyone lowering CODER_LLM_KILL_GRACE_S to a second or two quietly re-arms the 137.

The teardown timing line is the useful kind of instrumentation: it is the number that says how close the next teardown is to the grace, and it will make the next 137 diagnosable without a reproduction.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 70cc51a — both asks from my review of #2525 are answered, and the exit-status table resolves the ambiguity in the only way it can be resolved: by time, not by code.

The table

0    the window closed idle, or a grant whose demand went stale was released without starting the server
124  the max hold ended the window: gpu-hold's deadline, including a window gpu-hold had to KILL after the deadline
     (a 137 once the window has been open for the max hold, timed from the window's own start). Planned, not a crash
130  a stop was requested and the window refused to die; gpu-hold released the token anyway
137  something KILLed the window or the server BEFORE the max hold (the OOM killer, a manual kill -9): a failure
78   gpu-hold ended the hold on a GPU/NPU fault, or refused the ticket; backed off before exiting
75   the queue never came round within CODER_LLM_QUEUE_WAIT_S
any other: the server's own exit status (it exited by itself)

The line I could not have written myself is the parenthetical on 124: "a 137 once the window has been open for the max hold, timed from the window's own start." That is the resolution of the contradiction I flagged — 137 and 124 can be the same event, and the only thing that separates "planned" from "the OOM killer got it" is when it happened relative to the max hold. Two codes, one cause, distinguished by elapsed time: a lane can now read its own rc without guessing, and the statement is checkable (the window's start is in WINDOWS.log).

The warning is a warning, and it has both arms

[ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] 2> /dev/null || log "WARNING: the kill grace ${KILL_GRACE}s is not at least 5 s longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: …"
  • Not a clamp — an operator's explicit GPU_HOLD_KILL_GRACE still wins, as it should; the mistake is named instead of silently corrected.
  • Both directions are asserted: one arm requires the warning when the grace is 3 s against a 30 s stop wait, and one requires no warning when it fits. The negative arm is the one that matters over time — a warning that fires on correct configurations becomes noise and gets muted, which is how the nesting would drift back.

And the non-numeric case falls the safe way: 2> /dev/null on the [ test means a garbage value fails the comparison and the warning fires.

stop_server()'s new line is the instrument the grace needs

"logs how long the teardown took: the number that says how close the next one is to the grace" — server stopped in Ns after TERM (stop wait 30s). That turns the 30 s into a measured margin per window rather than a constant someone chose once: when the model grows, or a bigger context makes teardown slower, the journal says by how much before the KILL arrives. That is the same shape as the GTT-at-failure and the teardown-time lines elsewhere — put the number next to the decision, not in a doc.

Nothing to ask. Table, warning, arms, and the measurement that keeps the constant honest.

@bong-water-water-bong
bong-water-water-bong merged commit 48b7606 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/coder-llm-rc-map branch November 15, 2026 12:19
@bong-water-water-bong

bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Correction: this PR merged at 12:19Z, before I pushed. The commit described below is NOT in #2527. It is in engine#2530, rebased onto main, at 6971951. Its content is the same as described here: the rc 78 split, the time open on every outcome line, arm K, and two mutants.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant