Repository navigation
window-run: a restart deferred to the next lane's window is not a failure - #2547
Conversation
…lure
Measured on this box, twice, in the two windows that failed tonight (11:30Z and
12:54Z): the window's restore started coder-llm while the NEXT lane's gpu-hold
already held the token. coder-llm cannot load without the GTT that window is
using, so the 2-minute poll expired, a restart was attempted anyway, and the end
line recorded ':8090=000 restart=failed'. Two costs, both real:
- a false 69 (or 69-shaped end line) for a window whose job succeeded;
- the END line delayed by minutes, and the absence of an end line is what
every other lane reads as 'this lane is still holding'.
Now: if a GPU token exists at restore time, the unit is still started (so it is
not left down) and the end line says restart=deferred(gpu held by <pid>) with
the job's own rc.
And the poll itself now tells LOADING from WEDGED: a child loading a 30B answers
503 while its CPU time advances, so restarting it there throws the load away.
Only a child whose CPU time has not moved for ~30 s is dead. The LARGEST
llama-server is sampled -- the embed service's idle child is ~15 MB and would
read as frozen while the 30B loaded normally.
4 arms: the deferral writes its line, still starts the unit, and is not
reported as a failed restart; plus an artifact arm pinning the CPU reader (the
loading path needs a live child, and faking ps+curl+time would test the fake).
Mutation-checked: disabling the deferral branch turns the arm red.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Measured twice, in both windows that ended badly (11:30Z and 12:54Z), with the interleaving quoted — that is the standard of evidence this fleet should require for a race, and it is what makes this obviously not a flake: Three things:
And one note on the shape of the whole series: #2496 went live and produced #2512, #2514, #2515, #2525, #2527, #2530, #2539, #2544, #2545, #2546, #2547 — eleven PRs from one switch. They are all correct, and the aggregate is the argument for the "what changes when coder-llm is ticketed" page I asked for at #2515: the four owner/lifecycle rules (ticket cap, kill grace, queue wait, who restores) belong in one place, or the twelfth will be discovered the same way. |
|
VERDICT at b3a468b: CHANGES — the principle is right and the deferral tests existence instead of liveness, which turns a false failure into a silent down-service. The rule that fixes it already lives in this repo, in the tool that owns the token. The hole_tok=${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}
if [ -e "$_tok" ]; then
_held=$(awk '{print $NF}' "$_tok" | head -1)
printf '… :8090=deferred restart=deferred(gpu held by %s) rc=%s\n' … "${_held:-?}" …
The rule exists, nineteen lines into
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b3a468b: READY
The deferral is safe because two things I checked hold: gpu-hold deletes its token on release (so an existing token means another lane), and coder-llm is Restart=always / RestartSec=15 (so the deferral cannot become a silent outage).
The states, traced
scripts/gpu-hold:177,235,318 rm -f "$TOKEN" ← our own release removes the file
scripts/window-run.sh:375 if [ -e "$_tok" ] ← so an existing token is someone else's
scripts/window-run.sh:377 :8090=deferred restart=deferred(gpu held by <pid>) rc=<window rc>
coder-llm unit Restart=always / RestartSec=15
The middle line is what makes the check honest: the deferral fires on the file existing, not on comparing pids, so it would be wrong if our own token lingered. It does not linger — gpu-hold removes it before exiting, and restore() runs after wait $JOB. And the third line is what keeps it from being a way to hide a broken restart: the unit retries every 15 s by itself, so "deferred" means "loading will be retried by systemd, not by this window" rather than "nobody is watching".
The problem it solves is real and cost minutes today: polling :8090 for a 30B that cannot get GTT delays this window's END line — and every other lane reads that END line as "still holding the GPU", so a stuck restart was lengthening the very queue that was blocking it. That is a deadlock with a 25-minute cap, and the fix is to say so in one line rather than to burn the cap.
Measured
ctest -R '^window_run_self$' 100% tests passed
mutant (deferral block deleted): FAIL a GPU token held by the next lane defers the restart (got 0, want 1)
The other two arms are the ones that matter for not trading one bug for another: "... and the window still starts the unit, so it is not left down" and "... and the deferral is not reported as a failed restart". A deferral that skipped the start would have turned this fix into an outage.
One residual worth a line in the header
Consumer-visible: :8090=deferred restart=deferred … rc=<window rc> is a third state — neither the :8090=200 restart=ok of a good window nor the :8090=000 → rc 69 of a broken one (#2492's contract, which I reviewed). Anything reading WINDOWS.log to answer "is coder-llm up" must treat deferred as not up yet, and my own window/fault correlations in engine#2478 parse those lines. rc staying at the window's own code is right — the window did its job — but a reader who greps for :8090=200 will now see a lane whose window succeeded and whose model is not serving.
And the one case where it could misfire: if our own gpu-hold is SIGKILLed (rc 137), the token may be left behind, and the deferral would blame a lane that is not there. Restart=always still saves it, so I would not block on it — but the message says "gpu held by N" with the pid from a file, and that is the only place in the flow where the file's contents are trusted without the holder being re-checked.
…ueued) Review ask on this PR: the deferred and queued exemptions were each tested alone, so nothing pinned their precedence -- and with a token in another lane's hand AND a live coder-llm ticket queued, both are true at once (which is exactly the interleaving #2547 measured). The deferred branch is checked before the poll and the token is the more specific fact: the LLM is behind a token someone holds, not waiting for a ticket of its own. The arm asserts restart=deferred exactly once and restart=queued zero times. Mutation-checked: defeating the token check turns it red; restoring gives rc 0.
…rt (engine#2496 follow-up) (#2566) * window-run: a ticketed coder-llm that is QUEUED is not a failed restart engine#2496 merged, so `coder-llm` now runs inside `coder-llm-ticketed` and `systemctl start` means "queue for a GPU ticket". With lanes ahead of it the unit is `active` and `:8090` stays empty for minutes -- so the restore's poll expires, restarts it twice, and records the window as a failed restart. Measured on this box an hour after #2496 landed (window 17:07:40-17:08:59Z): the unit ran coder-llm-ticketed, `coder-llm` was 4th in ~/GPU-QUEUE, and the end line read ':8090=000 restart=failed' for a window whose job had succeeded. That is a false signal on EVERY window, which is exactly how a lane learns to ignore the one that matters. The restore now has the third state it was missing: token held by another lane -> restart=deferred (already, #2547) unit active + in the queue -> restart=queued <- this commit unit active + not queued -> the existing asleep/loading/wedged logic 2 arms: the queued case writes restart=queued(waiting for a GPU ticket) and does not write restart=FAILED. Mutation-checked: disabling the branch turns 2 arms red; restoring gives rc 0. * window-run: the queued test asks for LIVENESS, not existence (review of #2566) Two review findings, both mine: 1. coder_llm_queued() matched the label alone, so a queue entry left behind by a killed process licensed 'queued' for every later window -- a permanent false calm, worse than the false alarm this PR removes. gpu-hold itself calls such an entry stale and drops it (stale(): no pid, or whose pid is gone), so the reader was using a weaker rule than the writer. Now: read pid= and require kill -0. The arm that used to write pid=999999 (dead) and assert 'queued' is the bug in fixture form -- it is now the STALE arm, asserting the opposite, and the live arm uses this test's own live shell pid. 2. The queued arm passed only while :8090 happened to be down; with the fleet LLM serving, its poll returned 200 and the arm could not fire. The readiness URL is now WR_HEALTH_URL (default unchanged) and the arms force it to a refusing port, so the arm tests the branch and not the box. Mutation-checked: dropping kill -0 turns the stale arm red; restoring gives rc 0. Self-test green with the live box serving. --------- Co-authored-by: t <t@e.com>
The failure, measured twice tonight
In both windows that ended badly today (11:30Z and 12:54Z),
restore()startedcoder-llmwhile the next lane'sgpu-holdalready held the token:coder-llmcannot load without the GTT that window is using, so the 2-minute poll expired, a restart was attempted anyway, and the end line recorded a failure for a window whose job had succeeded. Two costs, both real:69is the code that is supposed to mean "the fleet LLM did not come back", so a deferral is exactly the case that teaches lanes to ignore it;The change (
scripts/window-run.sh)1. A deferral is named as a deferral. If a GPU token exists at restore time, the unit is still started (so it is not left down) and the end line says:
with the job's own rc — not 69, because nothing failed; the LLM comes back when that window ends. The token's last field is the holder pid, so the line names it.
2. The poll tells loading from wedged. A child loading a 30B answers 503 while its CPU time advances, so restarting it there throws the load away and competes for the GTT the other window holds (this is what actually happened: the child sat at 41 s of CPU, growing, while my restore called it dead). The poll now samples the largest
llama-server's CPU time and breaks to failure only after ~30 s with no progress. Sampling the largest matters: the1bit-embed-c22service also runs allama-server, idle at ~15 MB, and reading that one would report "frozen" while the 30B loaded normally.Arms — 4 new
Mutation-checked: disabling the deferral branch turns its arm red (
FAILs after mutation: 1), restoring gives rc 0.The fourth arm is an artifact arm and I will say why: the loading path needs a live child whose CPU grows, and faking
ps+curl+ elapsed time would mostly test the fake. So it pins the reader (sort -k2 -n | tail -1— taken with the[l]trick so the arm's own text cannot satisfy it) the same way the probe arm pins its URL. The behavioural half of that path is thefroze >= 10break, which the mutation above does not exercise.Not changed
coder_llm_asleep(#2503) still decides whether a silent port on an active unit means "asleep on purpose" — that is a different question ("did the unit start a child at all?") from this one ("is the child growing?"), and both are needed: #2503 covers a ticketed unit waiting for demand, this covers a unit whose child is loading behind another lane's window.