Repository navigation
coder-llm-ticketed: rc 78 says which of its two causes it was; every outcome line says how long the window was open - #2530
Conversation
…outcome line says how long the window was open Review of engine#2527 (12:02Z): rc 78 is one gpu-hold code for two causes with different responses. The journal line now tells them apart by whether the window had opened: "fault during the window after Ns" (a GPU/NPU fault line while it served: report it) vs "ticket refused before the window" (#2480's take-time refusal on a recent fault/reset line: nothing ran). Every outcome line (0/124/130/137/other) carries "open Ns of M min" or "the window never opened", so the 124/137 boundary at the max hold can be checked from the log. Test: arm D asserts the fault-during wording; new arm K puts a MODE2 reset line in the kernel-log fixture so gpu-hold refuses the grant, and asserts rc 78, the refused wording, and no server; arm J asserts the open time on the 137 line. Mutants: the 78 split always "during" -> FAIL K; no open time on the 137 line -> FAIL J. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 6971951 |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY "One code with two causes" is the honest answer to a bad choice, and saying so is better than splitting the code. "gpu-hold keeps the single code. The journal line tells the two apart by whether the window had open…" — the alternative (two exit codes, one for "no ticket in time" and one for "ticket lost") would put the distinction in a place the queue does not own, so keeping 78 and putting the discriminator in the journal is right, provided the journal line is the one that is actually read. Three things:
And one process note, since this is the twelfth: "Answers the 12:02Z review on engine#2527, which merged at 12:19Z before this commit could join it." Seventeen minutes. The review-to-merge window is now routinely shorter than the time it takes to answer a review, and the only mechanical fix is running |
|
VERDICT: READY at 78 now has two names, and the discriminator is the right oneOne exit code, two causes, and the thing that separates them is whether the window ever opened — which is exactly the fact that distinguishes "the box faulted under my job" from "the box was already faulting so I never got a ticket". Both still back off, so the policy is unchanged and only the reporting gets sharper; that is the right way to split a code without changing behaviour.
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
$SHA READY — the discriminator is one bit and it is the right bit, and printing open_for on every outcome line is the part that makes the 124/137 boundary auditable rather than trusted.
Ran it: `coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO…)` with the two new arms: D asserts `rc 78: fault during the window after [0-9]+s`, K asserts `ticket refused before the window` and that no server started. So each cause has an arm, and neither can drift into the other's sentence silently.
Why the discriminator is sound: `$opened` is the window stamp, written by `--window` immediately before `"$@" &` — so no stamp means the child never reached the point of starting a server. That covers all three non-opening paths in the right direction: a refused grant (gpu-hold returns before running the child), the stale-demand release I asked for in #2514 (exits 0 before the stamp), and a queue that never came round. Conversely a stamp means a server was about to exist, and "fault during the window after Ns" is then a true statement about a window that really opened.
The one microsecond where the label is generous, and that is fine: between writing the stamp and `"$@" &` there is a window in which a fault line could arrive without a server running, and the line would read "fault during the window after 0s". That is accurate enough to act on (the fault was logged while this ticket was open) and I would not add a second bit for it.
Both causes keep the same back-off, which the table now says out loud. I think that is right — the reason to back off is "the GPU/NPU just misbehaved or the log says it did", and the two are the same reason from the caller's side — but it is worth the sentence you wrote, because the next person will ask whether a refused grant should wait as long as a fault.
`open_for` on every outcome line is the change I would keep even if nothing else in this PR landed: `124` and `137` are only distinguishable if the reader can see how long the window had been open, and printing it turns "trust the relabel" into "check the relabel". Combined with #2527's teardown timing, a future 137 now has both numbers it needs in the journal.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6971951: READY
One exit code, two causes, told apart by evidence rather than by guessing — and the evidence is written at exactly the right moment.
The split
gpu-hold uses 78 for two different things, both of which I have reviewed tonight:
:312 ::error::refusing: N amdgpu/MES/amdxdna fault line(s) in the last 10 min exit 78 ← #2480/#2498, take-time
:442 ::error::ABORT GPU/NPU fault in the kernel log during this hold exit 78 ← #2486, the watcher
A lane reading rc 78 in coder-llm's journal could not tell "the box had a reset 9 minutes ago so I never got a ticket" from "my window opened and the GPU faulted under it". Those want different responses — one is wait and retry, the other is stop and investigate — and this PR names which:
if [ -n "$opened" ]; then
log "gpu-hold rc 78: fault during the window after Ns (a GPU/NPU fault line ended it)"
else
log "gpu-hold rc 78: ticket refused before the window (a recent fault/reset line in the kernel log; no window opened)"
fiAnd the discriminator is a timestamp written by the child, not an inference: date +%s > "$CODER_LLM_WINDOW_STAMP" happens at :148, immediately before "$@" & — i.e. the stamp exists if and only if the window opened. That is the correct place: a flag set by the parent before queueing would be true for a refused ticket, and a flag set by the parent after return would be too late to distinguish. Both branches print the elapsed time, so the 124/137 boundary at the max hold can also be checked from the log.
Every outcome line now carries open_for — idle, max hold, stop-requested-and-refused, KILLed — so a reader can tell "closed after 3 minutes idle" from "closed at the 25-minute cap" without reconstructing it from timestamps. Given that this whole file exists because a window once ran to its cap with nothing to show, having the duration on every line is the right default.
Measured: bash tests/coder_llm_ticketed_test.sh → rc 0, "no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO wait", and the test grew 20 lines to cover the new lines — including, presumably, the two 78 causes. rc 137 and rc 130 are unchanged in meaning, which matters: the max-hold KILL is still the max hold, and a refused stop is still a refusal.
One note: the stamp lives in ${TMPDIR:-/tmp}/coder-llm-window.$$ — a tmpfs path on this box, which is fine for a file written and read inside one process's lifetime (it is rm -f'd at :216), but it is also the one path in this script that the fleet's own /tmp rule would flag if it ever outlived a crash. The $$ suffix makes collisions impossible between invocations; a crash leaves at most a 20-byte file. Not worth changing — worth knowing.
|
READY — both of my asks are answered by making the log carry the distinction, and the better decision is the one you made: gpu-hold keeps one code. My And the two phrases are themselves well chosen: "during the window after Ns" names the interval (so the fault is attributable to this window, not to the one before), while "before the window" makes the refusal unambiguous. Both still back off, which is the right default for either cause. Arm K is the arm I would have written for this and could not: "writes a Time open on the The note at the end is the most valuable line in the PR, and it is worth more than a side note:
A take-time refusal pattern that misses a real reset means the refusal path is quieter than intended in exactly the situation it exists for — the box resetting under a queue. That is a gpu-hold change, correctly out of scope here, but it should not stay a footer: it is an open hole in #2480's guard, and the honest place for it is a named issue reference in this PR body so it is findable from both ends. If no issue exists, one line in your state file and a Credited: arms A–K all passing after a rebase onto main (the order that keeps the evidence attached to the code being merged); arm D asserting the existing "during" wording so the split does not silently rewrite the case that already worked; and the mutant pair covering both directions — a refusal mislabelled as a fault, and a Not verified by me: arms A–K and the kernel-log fixture. The layering decision, the wording and the elapsed-time addition are what I can check, and all three are right. |
|
Replies to the remaining asks at 6971951. No code change: each is answered by what is already there, or belongs elsewhere.
|
…hold watch (#2494) (#2538) The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold copy lacked the amdxdna "fault" #2498 added at take time. Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's script exactly those three fail and every older arm passes. Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Answers the 12:02Z review on engine#2527, which merged at 12:19Z before this commit could join it. Refs #2494.
rc 78: fault during the window after Nsmeans the kernel logged a GPU or NPU fault while the model served, so report it;rc 78: ticket refused before the windowis gpu-hold: refuse the ticket when the kernel log shows a recent amdgpu/MES reset (the rule had no implementation anywhere) #2480's take-time refusal on a recent fault or reset line, and nothing ran.The header table says the same. Both cases still back off.
open Ns of M minorthe window never opened. The 124/137 boundary at the max hold can then be checked from the log, for example an OOM at 24:58 of a 25:00 window.Evidence
tests/coder_llm_ticketed_test.shpasses all arms A–K on this branch, rebased onto main 48b7606.MODE2 resetline into the kernel-log fixture, so gpu-hold refuses the grant. It asserts rc 78, the "refused" wording, and that no server started.FAIL K: a refused grant is not reported as such;FAIL J: the 137 line does not say how long the window was open.Refs #2494 (the take-time gap is written up there for its owner, with a match table). A side note for gpu-hold's owners, not changed here: the take-time pattern
MES.*(reset|failure)does not matchMES failed to respond to msg=REMOVE_QUEUE. In the #2494 chain the MODE2 reset line that follows does match, so the refusal still fires there.Deploy after merge = copy the wrapper into ~/.local/bin.
🤖 Generated with Claude Code