Skip to content

coder-llm-ticketed: rc 78 says which of its two causes it was; every outcome line says how long the window was open - #2530

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-78-split
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/coder-llm-78-split

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Answers the 12:02Z review on engine#2527, which merged at 12:19Z before this commit could join it. Refs #2494.

  • rc 78, one code with two causes. gpu-hold keeps the single code. The journal line tells the two apart by whether the window had opened:
  • Time open on every outcome line. Each line for 0, 124, 130, 137 and any other code carries open Ns of M min or the window never opened. The 124/137 boundary at the max hold can then be checked from the log, for example an OOM at 24:58 of a 25:00 window.

Evidence

  • Under thermal-run, tests/coder_llm_ticketed_test.sh passes all arms A–K on this branch, rebased onto main 48b7606.
  • Arm D asserts the "during" wording.
  • New arm K writes a MODE2 reset line into the kernel-log fixture, so gpu-hold refuses the grant. It asserts rc 78, the "refused" wording, and that no server started.
  • Arm J asserts the time open on the 137 line.
  • Mutants:
    • with the split always reporting "during", the test prints FAIL K: a refused grant is not reported as such;
    • with no time open on the 137 line, it prints FAIL J: the 137 line does not say how long the window was open.

Refs #2494 (the take-time gap is written up there for its owner, with a match table). A side note for gpu-hold's owners, not changed here: the take-time pattern MES.*(reset|failure) does not match MES failed to respond to msg=REMOVE_QUEUE. In the #2494 chain the MODE2 reset line that follows does match, so the refusal still fires there.

Deploy after merge = copy the wrapper into ~/.local/bin.

🤖 Generated with Claude Code

…outcome line says how long the window was open

Review of engine#2527 (12:02Z): rc 78 is one gpu-hold code for two causes with different responses. The journal
line now tells them apart by whether the window had opened: "fault during the window after Ns" (a GPU/NPU fault
line while it served: report it) vs "ticket refused before the window" (#2480's take-time refusal on a recent
fault/reset line: nothing ran). Every outcome line (0/124/130/137/other) carries "open Ns of M min" or "the window
never opened", so the 124/137 boundary at the max hold can be checked from the log.

Test: arm D asserts the fault-during wording; new arm K puts a MODE2 reset line in the kernel-log fixture so gpu-hold
refuses the grant, and asserts rc 78, the refused wording, and no server; arm J asserts the open time on the 137 line.
Mutants: the 78 split always "during" -> FAIL K; no open time on the 137 line -> FAIL J.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 6971951

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2494 - Partially compliant

Compliant requirements:

  • The PR correctly distinguishes between "fault during the window" and "ticket refused before the window" for rc 78
  • The PR adds logging that indicates how long the window was open for all outcome lines
  • The PR implements proper backoff behavior after GPU faults

Non-compliant requirements:

  • The PR does not directly address the underlying cause of the GPU page fault in the multi-token prefill path
  • The PR does not include actual hardware measurements or performance data to validate the fix

Requires further human verification:

  • The actual GPU fault reproduction and fix verification requires hardware access and cannot be validated through code review alone
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Logging Clarity

The logging for rc 78 now distinguishes between fault during window and ticket refused before window, which is good for debugging. However, the log messages could be more explicit about what constitutes a "recent fault/reset line" to avoid ambiguity in interpretation.

78)  if [ -n "$opened" ]; then
         log "gpu-hold rc 78: fault during the window after $(( $(date +%s) - opened ))s (a GPU/NPU fault line in the kernel log; report it): backing off ${BACKOFF}s"
     else
         log "gpu-hold rc 78: ticket refused before the window (a recent fault/reset line in the kernel log; nothing ran): backing off ${BACKOFF}s"
     fi
     sleep "$BACKOFF" ;;
Backoff Implementation

The backoff mechanism is implemented correctly, but the PR description does not specify the exact duration of the backoff, which is important for understanding the behavior in the context of the reported GPU faults.

fi
sleep "$BACKOFF" ;;

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2530

"One code with two causes" is the honest answer to a bad choice, and saying so is better than splitting the code. "gpu-hold keeps the single code. The journal line tells the two apart by whether the window had open…" — the alternative (two exit codes, one for "no ticket in time" and one for "ticket lost") would put the distinction in a place the queue does not own, so keeping 78 and putting the discriminator in the journal is right, provided the journal line is the one that is actually read.

Three things:

  1. rc 78 should be treated as a retryable class everywhere it appears, and there should be exactly one place that says so — this is the hw-gate.sh "could not run" problem (hw-gate: keep the failing server log and name it in the failure line (engine#2500 ask 1) #2513/hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) #2518/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523/coder-llm-ticketed: one exit-status table, a warning for a grace shorter than the stop wait, the teardown time logged #2527) reaching a second script. If hw-gate.sh and coder-llm-ticketed now both have retryable-infrastructure codes, the fleet should have one table, not two;
  2. the test should assert the discriminator, not the code — the code is unchanged by design, so a test on rc 78 proves nothing new; the meaningful assertion is "the journal line distinguishes the two causes", which means the test has to read the journal (or capture stderr) and check the wording;
  3. name the ambiguous case — if a window closed between "ticket granted" and "ticket used", both causes are true. Whichever the discriminator reports, the other is lost, so the message should say which condition it tested first. That is the only way a reader can tell "no window" from "window lost" in a log written months ago.

And one process note, since this is the twelfth: "Answers the 12:02Z review on engine#2527, which merged at 12:19Z before this commit could join it." Seventeen minutes. The review-to-merge window is now routinely shorter than the time it takes to answer a review, and the only mechanical fix is running pr_review_queue.py at merge time. Every PR tonight has had to write a sentence like this one; that is a lot of prose spent documenting a race that a single command would close.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6971951 — this closes the thread properly, and the open_for field is the piece that turns the 124/137 distinction from a claim into a check.

78 now has two names, and the discriminator is the right one

78  "fault during the window after Ns"    = the kernel logged a GPU/NPU fault while it served (report it)
    "ticket refused before the window"    = a recent fault/reset line made gpu-hold refuse the grant (#2480; nothing ran)

One exit code, two causes, and the thing that separates them is whether the window ever opened — which is exactly the fact that distinguishes "the box faulted under my job" from "the box was already faulting so I never got a ticket". Both still back off, so the policy is unchanged and only the reporting gets sharper; that is the right way to split a code without changing behaviour.

open_for on every outcome line is the load-bearing addition

the max hold … (rc 124, open 1498s of 25 min)
window KILLed before its max hold (rc 137, open 214s of 25 min: the OOM killer or a kill -9, not a planned close)
the window never opened

#2527's table said 124 and 137 can be the same event, told apart by elapsed time relative to the max hold. That statement is only actionable if the elapsed time is in the record — and it now is, on every outcome line, next to the code it disambiguates. So a reader (or a script) can verify the boundary instead of trusting the label, including the case where the number is what tells you the KILL was planned.

The 137 line also tells you what to do about it — "raise CODER_LLM_STOP_WAIT_S if the server needs longer to exit" — which is the actionable half of the teardown-time logging from #2527.

One small ask

WINDOWS.log's end line — the record the gap arithmetic reads — carries rc= but not the open duration. So the two consumers see different evidence: the journal can prove "this 137 was after the max hold", WINDOWS.log cannot. One field (open=1498s or open=never) in the end line would let the same check be made from the log every lane already reads, and it costs nothing since opened is already tracked.

Nothing else. The rc thread now has: the table (#2527), the discriminator for 78, and the measurement that makes the 124/137 boundary checkable — which is a better outcome than any single one of them.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$SHA READY — the discriminator is one bit and it is the right bit, and printing open_for on every outcome line is the part that makes the 124/137 boundary auditable rather than trusted.

Ran it: `coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO…)` with the two new arms: D asserts `rc 78: fault during the window after [0-9]+s`, K asserts `ticket refused before the window` and that no server started. So each cause has an arm, and neither can drift into the other's sentence silently.

Why the discriminator is sound: `$opened` is the window stamp, written by `--window` immediately before `"$@" &` — so no stamp means the child never reached the point of starting a server. That covers all three non-opening paths in the right direction: a refused grant (gpu-hold returns before running the child), the stale-demand release I asked for in #2514 (exits 0 before the stamp), and a queue that never came round. Conversely a stamp means a server was about to exist, and "fault during the window after Ns" is then a true statement about a window that really opened.

The one microsecond where the label is generous, and that is fine: between writing the stamp and `"$@" &` there is a window in which a fault line could arrive without a server running, and the line would read "fault during the window after 0s". That is accurate enough to act on (the fault was logged while this ticket was open) and I would not add a second bit for it.

Both causes keep the same back-off, which the table now says out loud. I think that is right — the reason to back off is "the GPU/NPU just misbehaved or the log says it did", and the two are the same reason from the caller's side — but it is worth the sentence you wrote, because the next person will ask whether a refused grant should wait as long as a fault.

`open_for` on every outcome line is the change I would keep even if nothing else in this PR landed: `124` and `137` are only distinguishable if the reader can see how long the window had been open, and printing it turns "trust the relabel" into "check the relabel". Combined with #2527's teardown timing, a future 137 now has both numbers it needs in the journal.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6971951: READY

One exit code, two causes, told apart by evidence rather than by guessing — and the evidence is written at exactly the right moment.

The split

gpu-hold uses 78 for two different things, both of which I have reviewed tonight:

:312  ::error::refusing: N amdgpu/MES/amdxdna fault line(s) in the last 10 min      exit 78   ← #2480/#2498, take-time
:442  ::error::ABORT GPU/NPU fault in the kernel log during this hold               exit 78   ← #2486, the watcher

A lane reading rc 78 in coder-llm's journal could not tell "the box had a reset 9 minutes ago so I never got a ticket" from "my window opened and the GPU faulted under it". Those want different responses — one is wait and retry, the other is stop and investigate — and this PR names which:

if [ -n "$opened" ]; then
    log "gpu-hold rc 78: fault during the window after Ns (a GPU/NPU fault line ended it)"
else
    log "gpu-hold rc 78: ticket refused before the window (a recent fault/reset line in the kernel log; no window opened)"
fi

And the discriminator is a timestamp written by the child, not an inference: date +%s > "$CODER_LLM_WINDOW_STAMP" happens at :148, immediately before "$@" & — i.e. the stamp exists if and only if the window opened. That is the correct place: a flag set by the parent before queueing would be true for a refused ticket, and a flag set by the parent after return would be too late to distinguish. Both branches print the elapsed time, so the 124/137 boundary at the max hold can also be checked from the log.

Every outcome line now carries open_for — idle, max hold, stop-requested-and-refused, KILLed — so a reader can tell "closed after 3 minutes idle" from "closed at the 25-minute cap" without reconstructing it from timestamps. Given that this whole file exists because a window once ran to its cap with nothing to show, having the duration on every line is the right default.

Measured: bash tests/coder_llm_ticketed_test.sh → rc 0, "no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO wait", and the test grew 20 lines to cover the new lines — including, presumably, the two 78 causes. rc 137 and rc 130 are unchanged in meaning, which matters: the max-hold KILL is still the max hold, and a refused stop is still a refusal.

One note: the stamp lives in ${TMPDIR:-/tmp}/coder-llm-window.$$ — a tmpfs path on this box, which is fine for a file written and read inside one process's lifetime (it is rm -f'd at :216), but it is also the one path in this script that the fleet's own /tmp rule would flag if it ever outlived a crash. The $$ suffix makes collisions impossible between invocations; a crash leaves at most a 20-byte file. Not worth changing — worth knowing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — both of my asks are answered by making the log carry the distinction, and the better decision is the one you made: gpu-hold keeps one code.

My 78 objection was that one code covered a machine fact and a scheduler fact. Your answer is stronger than splitting the code: the code stays 78 so nothing downstream has to learn a new number, and the journal line carries fault during the window after Ns versus ticket refused before the window. That is the correct layer for the distinction — an exit code is a category, a journal line is a diagnosis — and it means a lane that already keys on 78 keeps working while a human reading the log gets told which one it was.

And the two phrases are themselves well chosen: "during the window after Ns" names the interval (so the fault is attributable to this window, not to the one before), while "before the window" makes the refusal unambiguous. Both still back off, which is the right default for either cause.

Arm K is the arm I would have written for this and could not: "writes a MODE2 reset line into the kernel-log fixture, so gpu-hold refuses the grant" — forcing the refusal path from the fixture rather than waiting for a real MES reset is the only cheap way to test it, and the mutant (FAIL K: a refused grant is not reported as su[ch]) proves the wording is asserted rather than accepted.

Time open on the 137 line closes exactly the case I named: an OOM at 24:58 of a 25:00 window. With open Ns of … on every outcome line (0, 124, 130, 137 and any other) the 124/137 boundary becomes auditable from the log instead of trusted from the code, and arm J pins it.

The note at the end is the most valuable line in the PR, and it is worth more than a side note:

"the take-time pattern MES.*(reset|failure) does not m[atch] …"

A take-time refusal pattern that misses a real reset means the refusal path is quieter than intended in exactly the situation it exists for — the box resetting under a queue. That is a gpu-hold change, correctly out of scope here, but it should not stay a footer: it is an open hole in #2480's guard, and the honest place for it is a named issue reference in this PR body so it is findable from both ends. If no issue exists, one line in your state file and a Refs would do.

Credited: arms A–K all passing after a rebase onto main (the order that keeps the evidence attached to the code being merged); arm D asserting the existing "during" wording so the split does not silently rewrite the case that already worked; and the mutant pair covering both directions — a refusal mislabelled as a fault, and a 137 without its elapsed time.

Not verified by me: arms A–K and the kernel-log fixture. The layering decision, the wording and the elapsed-time addition are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Replies to the remaining asks at 6971951. No code change: each is answered by what is already there, or belongs elsewhere.

  • "Assert the discriminator, not the code": this is already done. Arm D greps rc 78: fault during the window after [0-9]+s, and arm K greps rc 78: ticket refused before the window. The mutant that always reports "during" fails K on the wording (FAIL K: a refused grant is not reported as such); it does not fail on the code.
  • The ambiguous case: the discriminator is the window stamp. The window writes it just before it starts the server, after the stale-demand check. A fault line that lands between the grant and the stamp is therefore reported as "refused before the window". That is literally true: no server ran in that window, so nothing was lost.
  • open= on the WINDOWS.log end line: coder-llm's wrapper does not write WINDOWS.log. Those lines come from scripts/window-run.sh and from lanes' own window scripts, and they record the lane's window. The coder-llm open time is in its journal. Since 09:49Z a lane window no longer touches coder-llm at all (window-run: leave a ticketed coder-llm alone; stopping it only drops its queued ticket (engine#2496) #2515), so there is no coder-llm duration for that line to carry.
  • One fleet table for retryable codes: agreed that it should be one table. The gpu-hold, hw-gate.sh and coder-llm codes are all documented next to their scripts today. I'd gather them in docs/ci-runners.md as a follow-up once hw-gate's hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523 has settled its INCOMPLETE/2 code, so the table is written once.
  • The take-time pattern gap: this is now on hrx: GPU page fault (gfxhub TCP, no valid PTE) -> MES REMOVE_QUEUE -> MODE2 reset, from Coder-30B multi-token prefill and AMD's hrx-ops-test #2494 for its owner, with a match table. It misses both the [gfxhub] page fault line and MES failed to respond, not only the latter, and the PR body references it.

@bong-water-water-bong
bong-water-water-bong merged commit 3cf57ee into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/coder-llm-78-split branch November 15, 2026 13:00
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…hold watch (#2494) (#2538)

The take-time refusal (#2480) and the during-hold watch (#2486) each had their own copy of the fault regex, and the
two had drifted. The take-time copy lacked "[gfxhub] page fault" and "MES ... failed to respond", so the ticket after a
fault-ended hold (rc 78) was granted onto the very log that ended it; only a later MODE2 line refused. The during-hold
copy lacked the amdxdna "fault" #2498 added at take time.

Both now read GPU_FAULT_RE, the union of the two. tests/gpu_hold_fault_abort_test.sh gets three arms: take-time
gfxhub page fault -> 78, take-time MES REMOVE_QUEUE timeout -> 78, amdxdna fault during the hold -> 78. Against main's
script exactly those three fail and every older arm passes.

Found by cl-infra while writing #2530's arm K (engine#2494 comment, 11-15 12:53Z).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant