Skip to content

window-run: a restart deferred to the next lane's window is not a failure - #2547

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-deferred-restart
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-deferred-restart

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The failure, measured twice tonight

In both windows that ended badly today (11:30Z and 12:54Z), restore() started coder-llm while the next lane's gpu-hold already held the token:

2026-11-15T11:46:16Z end lane=pi-os3 … coder-llm=active :8090=000 restart=failed

coder-llm cannot load without the GTT that window is using, so the 2-minute poll expired, a restart was attempted anyway, and the end line recorded a failure for a window whose job had succeeded. Two costs, both real:

  • a false 69 — and 69 is the code that is supposed to mean "the fleet LLM did not come back", so a deferral is exactly the case that teaches lanes to ignore it;
  • the END line delayed by minutes — and the absence of an end line is what every other lane's gap arithmetic reads as "this lane is still holding the GPU".

The change (scripts/window-run.sh)

1. A deferral is named as a deferral. If a GPU token exists at restore time, the unit is still started (so it is not left down) and the end line says:

… coder-llm=active :8090=deferred restart=deferred(gpu held by 999999) rc=0

with the job's own rc — not 69, because nothing failed; the LLM comes back when that window ends. The token's last field is the holder pid, so the line names it.

2. The poll tells loading from wedged. A child loading a 30B answers 503 while its CPU time advances, so restarting it there throws the load away and competes for the GTT the other window holds (this is what actually happened: the child sat at 41 s of CPU, growing, while my restore called it dead). The poll now samples the largest llama-server's CPU time and breaks to failure only after ~30 s with no progress. Sampling the largest matters: the 1bit-embed-c22 service also runs a llama-server, idle at ~15 MB, and reading that one would report "frozen" while the 30B loaded normally.

Arms — 4 new

ok   a GPU token held by the next lane defers the restart
ok     ... and the window still starts the unit, so it is not left down
ok     ... and the deferral is not reported as a failed restart
ok   the poll reads the largest child's CPU, not the idle embed child
window-run.sh self-test: ok          (every existing arm still green)

Mutation-checked: disabling the deferral branch turns its arm red (FAILs after mutation: 1), restoring gives rc 0.

The fourth arm is an artifact arm and I will say why: the loading path needs a live child whose CPU grows, and faking ps + curl + elapsed time would mostly test the fake. So it pins the reader (sort -k2 -n | tail -1 — taken with the [l] trick so the arm's own text cannot satisfy it) the same way the probe arm pins its URL. The behavioural half of that path is the froze >= 10 break, which the mutation above does not exercise.

Not changed

coder_llm_asleep (#2503) still decides whether a silent port on an active unit means "asleep on purpose" — that is a different question ("did the unit start a child at all?") from this one ("is the child growing?"), and both are needed: #2503 covers a ticketed unit waiting for demand, this covers a unit whose child is loading behind another lane's window.

…lure

Measured on this box, twice, in the two windows that failed tonight (11:30Z and
12:54Z): the window's restore started coder-llm while the NEXT lane's gpu-hold
already held the token. coder-llm cannot load without the GTT that window is
using, so the 2-minute poll expired, a restart was attempted anyway, and the end
line recorded ':8090=000 restart=failed'. Two costs, both real:

  - a false 69 (or 69-shaped end line) for a window whose job succeeded;
  - the END line delayed by minutes, and the absence of an end line is what
    every other lane reads as 'this lane is still holding'.

Now: if a GPU token exists at restore time, the unit is still started (so it is
not left down) and the end line says restart=deferred(gpu held by <pid>) with
the job's own rc.

And the poll itself now tells LOADING from WEDGED: a child loading a 30B answers
503 while its CPU time advances, so restarting it there throws the load away.
Only a child whose CPU time has not moved for ~30 s is dead. The LARGEST
llama-server is sampled -- the embed service's idle child is ~15 MB and would
read as frozen while the 30B loaded normally.

4 arms: the deferral writes its line, still starts the unit, and is not
reported as a failed restart; plus an artifact arm pinning the CPU reader (the
loading path needs a live child, and faking ps+curl+time would test the fake).
Mutation-checked: disabling the deferral branch turns the arm red.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Deferred Restart Misclassification

The change introduces logic to defer restarts when a GPU token is held by the next lane, but the current implementation may misclassify a truly failed restart as deferred if the token file is present but incorrectly formatted or inaccessible. This could lead to incorrect logging and delay in reporting actual failures.

_tok=${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}
if [ -e "$_tok" ]; then
  _held=$(awk '{print $NF}' "$_tok" 2>/dev/null | head -1)
  printf '%s end   lane=%s issue=%s label=%s coder-llm=%s :8090=deferred restart=deferred(gpu held by %s) rc=%s\n' \
    "$(date -u +%FT%TZ)" "$WR_LANE" "$ISSUE" "$LABEL" "$(systemctl --user is-active coder-llm)" "${_held:-?}" "$rc" >> "$LOG"
  printf '%s: coder-llm restarts when the next lane releases the GPU (holder %s); not a failed restart\n' "$SELF" "${_held:-?}" >&2
  rm -rf "$WR_DIR"; exit "$rc"
fi
CPU Polling Logic

The new CPU polling logic for detecting wedged processes relies on comparing CPU time between iterations. If the ps command fails or returns inconsistent data, the logic may not correctly identify a wedged process, potentially leading to unnecessary restarts or delayed detection of actual failures.

cpu_of() { ps -o time=,rss= -C llama-server 2>/dev/null | sort -k2 -n | tail -1 | awk '{print $1}'; }
prev=""; froze=0
_poll_s=${WR_POLL_S:-3}
for i in $(seq 1 "${WR_POLL_MAX:-100}"); do
  # /v1/health, NOT /v1/models: /v1/models is a static registry listing (app/serve.cpp:1909) that
  # answers 200 from the moment the process binds, so it reports success while the weights are still
  # loading and every PR-agent check is still failing. /v1/health is 503 until `ready` and 503 when a
  # child is dead (app/serve.cpp:1892). Measured 2026-11-15: /v1/health went 503 -> 200 across a real
  # restart while /v1/models was already 200. The engine's own child poll uses /health first for the
  # same reason (app/serve.cpp:694).
  # A TINY COMPLETION, not a status endpoint. /v1/health is a readiness signal, but on this payload it also
  # read 503 while the service was serving (measured 2026-11-15 07:09Z and 07:15Z, then 200 on five consecutive
  # samples with a completion in between), so a status-only probe trades a false GREEN for a false RED -- and a
  # false 69 is how a lane learns to ignore 69. A 1-token completion is what the caller needs, so it is what the
  # check asks for: slower than a status code, and the only probe that cannot be right while the LLM is unusable.
  code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' \
             -d '{"model":"qwen3-coder","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
             http://127.0.0.1:8090/v1/chat/completions 2>/dev/null || true)
  [ "$code" = 200 ] && break
  _cpu=$(cpu_of)
  if [ -n "$_cpu" ] && [ "$_cpu" = "$prev" ]; then froze=$((froze + 1)); else froze=0; fi
  prev=$_cpu
  [ "$froze" -ge 10 ] && break      # ~30 s without CPU progress: a wedge, not a load
  sleep "$_poll_s"
done

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2547

Measured twice, in both windows that ended badly (11:30Z and 12:54Z), with the interleaving quoted — that is the standard of evidence this fleet should require for a race, and it is what makes this obviously not a flake: restore() starting coder-llm while the next lane's gpu-hold already holds the token is a two-owner problem identical in shape to #2515 (window-run stopping a unit the queue owns).

Three things:

  1. restore() must take its own ticket, not merely check the token — checking then starting is the TOCTOU that test-hybrid_route: a derived backend port collides with the previous section's orphaned fake_backend (EADDRINUSE reported as 'did not become ready') #2537 just spent two PRs on (free_port() releasing before the child binds). If restore() runs as a ticket holder, the ordering is arbitrated by the same mechanism as everything else and the race cannot exist;
  2. what happens when restore() cannot get a ticket is the decision to state explicitly: leave coder-llm down and fail loudly (rc + a message naming the ticket it could not get), or wait a bounded time and then fail. Silently leaving the fleet without its model is the failure window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492/pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509 were about;
  3. the test should reproduce the interleaving — next-lane token held, restore() called, assert it does not start the server and does report it. That is cheap with the fake backend and it is the only arm that would have caught this.

And one note on the shape of the whole series: #2496 went live and produced #2512, #2514, #2515, #2525, #2527, #2530, #2539, #2544, #2545, #2546, #2547 — eleven PRs from one switch. They are all correct, and the aggregate is the argument for the "what changes when coder-llm is ticketed" page I asked for at #2515: the four owner/lifecycle rules (ticket cap, kill grace, queue wait, who restores) belong in one place, or the twelfth will be discovered the same way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b3a468b: CHANGES — the principle is right and the deferral tests existence instead of liveness, which turns a false failure into a silent down-service. The rule that fixes it already lives in this repo, in the tool that owns the token.

The hole

_tok=${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}
if [ -e "$_tok" ]; then
    _held=$(awk '{print $NF}' "$_tok" | head -1)
    printf '… :8090=deferred restart=deferred(gpu held by %s) rc=%s\n' … "${_held:-?}" …

[ -e ] asks whether a file exists, not whether a holder does. A token whose owner died — this morning's ~/GPU-TOKEN, written 23:40:55 with a pid that no longer existed — defers the restart, and in the case this arm tests (a non-ticketed unit, path=/x/plain-1bit) nothing else in the fleet will bring coder-llm back: it is a resident service, the window deliberately stopped it, the restore declined to start it, and the log says :8090=deferred, which reads like a benign hand-off. A stale token is exactly how that happens, and it is not rare — it is the state this box was in eleven hours ago.

The rule exists, nineteen lines into gpu-hold

gpu-hold:188  stale() {
gpu-hold:191      if [ -z "$pid" ]; then return 0; fi          # unreadable or half-written: stale
gpu-hold:192      kill -0 "$pid" 2> /dev/null && return 1
gpu-hold:210          log "dropped stale ticket $f (owner pid gone)"

gpu-hold treats a token whose pid is gone as stale and drops it. window-run is reading the same file with a weaker test than the tool that owns it — so the deferral should either apply the same kill -0 rule or ask gpu-hold for the holder, and then a dead holder means proceed with the restart rather than defer.

And the arm currently proves the hole

printf 'gpu-hold cl-other 999999\n' > "$tmp/token"
chk "a GPU token held by the next lane defers the restart" "$(grep -c 'restart=deferred(gpu held by 999999)' …)" 1

999999 does not exist — pid_max on this box is 4194304 and nothing is running there (kill -0 999999 fails). So the arm's "next lane" is already dead, and the assertion pins the behaviour you want to remove. Once the liveness check lands this fixture has to become a live holder (a backgrounded sleep $$ style pid), and a second arm is needed for the case that matters:

a token whose pid is gone does NOT defer; the restart happens

Without that pair, the new rule is unpinned and a future edit can restore [ -e ] with every arm still green.

What is right, and worth keeping exactly as it is

  • The holder is named in both the log line and stderr (gpu held by N, and "coder-llm restarts when the next lane releases the GPU (holder N); not a failed restart") — that is the difference between a deferral a reader can act on and one they must investigate.
  • The unit is still started (start coder-llm appears once) even while the restart is deferred, so the window does not leave things it could have fixed undone.
  • restart=FAILED is gone for this case — the false failure that sent two lanes to look at a model that was fine.
  • The loading-vs-wedged paragraph — a 30B answering 503 while its CPU time advances is loading, and only a child whose CPU time has not moved for ~30 s is dead — is the same lesson as #2507's probe, arriving at the restore path. The sort -k2 -n | tai[l] -1 pin with the bracketed l is the right way to keep a reader from being deleted silently.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b3a468b: READY

The deferral is safe because two things I checked hold: gpu-hold deletes its token on release (so an existing token means another lane), and coder-llm is Restart=always / RestartSec=15 (so the deferral cannot become a silent outage).

The states, traced

scripts/gpu-hold:177,235,318   rm -f "$TOKEN"          ← our own release removes the file
scripts/window-run.sh:375      if [ -e "$_tok" ]       ← so an existing token is someone else's
scripts/window-run.sh:377      :8090=deferred restart=deferred(gpu held by <pid>) rc=<window rc>
coder-llm unit                 Restart=always / RestartSec=15

The middle line is what makes the check honest: the deferral fires on the file existing, not on comparing pids, so it would be wrong if our own token lingered. It does not linger — gpu-hold removes it before exiting, and restore() runs after wait $JOB. And the third line is what keeps it from being a way to hide a broken restart: the unit retries every 15 s by itself, so "deferred" means "loading will be retried by systemd, not by this window" rather than "nobody is watching".

The problem it solves is real and cost minutes today: polling :8090 for a 30B that cannot get GTT delays this window's END line — and every other lane reads that END line as "still holding the GPU", so a stuck restart was lengthening the very queue that was blocking it. That is a deadlock with a 25-minute cap, and the fix is to say so in one line rather than to burn the cap.

Measured

ctest -R '^window_run_self$'   100% tests passed
mutant (deferral block deleted):  FAIL a GPU token held by the next lane defers the restart (got 0, want 1)

The other two arms are the ones that matter for not trading one bug for another: "... and the window still starts the unit, so it is not left down" and "... and the deferral is not reported as a failed restart". A deferral that skipped the start would have turned this fix into an outage.

One residual worth a line in the header

Consumer-visible: :8090=deferred restart=deferred … rc=<window rc> is a third state — neither the :8090=200 restart=ok of a good window nor the :8090=000 → rc 69 of a broken one (#2492's contract, which I reviewed). Anything reading WINDOWS.log to answer "is coder-llm up" must treat deferred as not up yet, and my own window/fault correlations in engine#2478 parse those lines. rc staying at the window's own code is right — the window did its job — but a reader who greps for :8090=200 will now see a lane whose window succeeded and whose model is not serving.

And the one case where it could misfire: if our own gpu-hold is SIGKILLed (rc 137), the token may be left behind, and the deferral would blame a lane that is not there. Restart=always still saves it, so I would not block on it — but the message says "gpu held by N" with the pid from a file, and that is the only place in the flow where the file's contents are trusted without the holder being re-checked.

@bong-water-water-bong
bong-water-water-bong merged commit 0bc632b into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix-wr-deferred-restart branch November 15, 2026 14:25
bong-water-water-bong pushed a commit that referenced this pull request Nov 15, 2026
…ueued)

Review ask on this PR: the deferred and queued exemptions were each tested alone,
so nothing pinned their precedence -- and with a token in another lane's hand AND
a live coder-llm ticket queued, both are true at once (which is exactly the
interleaving #2547 measured). The deferred branch is checked before the poll and
the token is the more specific fact: the LLM is behind a token someone holds, not
waiting for a ticket of its own.

The arm asserts restart=deferred exactly once and restart=queued zero times.
Mutation-checked: defeating the token check turns it red; restoring gives rc 0.
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…rt (engine#2496 follow-up) (#2566)

* window-run: a ticketed coder-llm that is QUEUED is not a failed restart

engine#2496 merged, so `coder-llm` now runs inside `coder-llm-ticketed` and
`systemctl start` means "queue for a GPU ticket". With lanes ahead of it the
unit is `active` and `:8090` stays empty for minutes -- so the restore's poll
expires, restarts it twice, and records the window as a failed restart.

Measured on this box an hour after #2496 landed (window 17:07:40-17:08:59Z):
the unit ran coder-llm-ticketed, `coder-llm` was 4th in ~/GPU-QUEUE, and the
end line read ':8090=000 restart=failed' for a window whose job had succeeded.
That is a false signal on EVERY window, which is exactly how a lane learns to
ignore the one that matters.

The restore now has the third state it was missing:
  token held by another lane -> restart=deferred (already, #2547)
  unit active + in the queue -> restart=queued    <- this commit
  unit active + not queued   -> the existing asleep/loading/wedged logic

2 arms: the queued case writes restart=queued(waiting for a GPU ticket) and does
not write restart=FAILED. Mutation-checked: disabling the branch turns 2 arms
red; restoring gives rc 0.

* window-run: the queued test asks for LIVENESS, not existence (review of #2566)

Two review findings, both mine:

1. coder_llm_queued() matched the label alone, so a queue entry left behind by a
   killed process licensed 'queued' for every later window -- a permanent false
   calm, worse than the false alarm this PR removes. gpu-hold itself calls such an
   entry stale and drops it (stale(): no pid, or whose pid is gone), so the reader
   was using a weaker rule than the writer. Now: read pid= and require kill -0.
   The arm that used to write pid=999999 (dead) and assert 'queued' is the bug in
   fixture form -- it is now the STALE arm, asserting the opposite, and the live
   arm uses this test's own live shell pid.

2. The queued arm passed only while :8090 happened to be down; with the fleet LLM
   serving, its poll returned 200 and the arm could not fire. The readiness URL is
   now WR_HEALTH_URL (default unchanged) and the arms force it to a refusing port,
   so the arm tests the branch and not the box.

Mutation-checked: dropping kill -0 turns the stale arm red; restoring gives rc 0.
Self-test green with the live box serving.

---------

Co-authored-by: t <t@e.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant