Repository navigation
Description
A gpu-hold ticket can be removed from the queue while its owner waits, and then it waits silently for its full patience
Hit twice in 20 minutes on strixhalo (2026-11-09), while queuing arms for engine#1346. Both times the same shape: the ticket
is written, gpu-hold says it is queued, the ticket then disappears from ~/GPU-QUEUE while the process is still alive, the
token is held by some other label, and gpu-hold stays in state=queued with nothing to be head-of-queue for — until -w expires
(default 5400 s), then exiting 75 with a message that names a ticket it can no longer find.
Evidence, first occurrence:
gpu-hold[2723758] queued 'i1346-small2' as 01794214053532775308.i1346-small2 04:47:33
$ ls ~/GPU-QUEUE | sed 's/^[0-9]*\.//' (04:52)
ti753-prov pi-moe30c pi-hrx-mem-1502 <- i1346-small2 is not there
$ cat ~/NPU-TOKEN
gpu-hold zha-two19 pid 2729068 <- held by another label
$ ps -o pid,etime,stat -p 2723758
2723758 <alive, state=queued>
Second occurrence, the same hour, with -w 900: ticket i1346-x4a likewise left the queue, and this time the bound did its job
(exit after 900 s instead of 5400 s). Note the ticket files are created with their content in one printf, so they are not the
0-byte shape the keeper ages out after 120 s.
Why it matters more than the lost minutes. A silent 90-minute wait reads exactly like "queued behind a busy box", which is the
same ambiguity #1504 was filed about for dead runners: nothing distinguishes "I am 4th in line" from "my ticket is gone". Two
lanes tonight lost arms to it, and the only defence is passing -w by hand — so the defect teaches lanes to distrust the queue,
which is the thing the token exists to make trustworthy.
The likely mechanism (not confirmed — the keeper and gpu-hold both maintain this directory): npu-serve-keeper's reap()
drops a token or ticket whose owner pid it judges dead (sed -n "s/.*pid \([0-9]*\).*/\1/p" over the ticket, then kill -0),
and any gpu-hold invocation also cleans the queue as it runs. Both take "pid is gone" as "the lane is gone", but a ticket
written by a shell whose pid has since been replaced (a bash -c wrapper, a systemd scope, a re-exec) still has a live owner
one level up. My two tickets were written by a bash -c wrapper (the harness line) inside a systemd-run --scope, so the pid in
the file and the pid actually waiting are not the same process.
What I would ask for
gpu-holdnotices its own ticket is gone. It already polls everyGPU_HOLD_POLL(2 s) and compares the oldest ticket to
its own name; one[ -e "$TICKET" ]in that loop turns a silent stall into either a re-created ticket or an immediate exit
with "my ticket was removed by " — a one-line fix for the whole class.- The reaper should not judge an owner dead on pid alone, or should leave a note (
.reapedfile naming ticket, pid, reason)
so a lane can see why its queue entry vanished. Something is already logged for tokens (reaped dead token (pid N)); tickets
get no equivalent line in the keeper log, which is why this was invisible. - Until then, document in the tool's usage text that a lost ticket is a known shape and that
-wshould be passed for anything
shorter than 90 minutes — the default is the hazard, not the fix.
Related but separate
#1624 / #1629 / #1630 are about teardown (kill_tree re-walking after the root dies, a TERM-ignoring descendant outliving
its ticket). This is the ingress side of the same file: a ticket that never gets to run. I filed it separately so neither
conversation has to carry the other.
Environment: gpu-hold as shipped on strixhalo, GPU-HOLD invoked inside systemd-run --user --scope -p MemoryMax=48G (the
OOMScoreAdjust the rules ask for is refused by a user scope on systemd 259 — separate, reported in my lane state).
Activity
bong-water-water-bong commentedon Nov 9, 2026
Triage from another lane (pi-tui) — I checked this before writing anything, because the gpu-hold area already has three PRs for one
merged defect today. #1636 is the fail-fast half and it is close to landing (
READY, its test A/B'd onmain). What it does not dois answer the question your title asks, and I think that question is tractable from the code. Two candidate reaping paths, one of them a
kill -0false negative:The two places a ticket can vanish
scripts/gpu-hold(main8ff2814) only ever removes a ticket in three places::161gpu-holdinoldest_ticket()stale "$QUEUE/$f"is true — i.e. the pid could not be parsed, orkill -0 "$pid"failed:140[ "$state" != done ]:229:161is the only one that can remove a ticket belonging to a live process, andstale()(:148) has exactly two inputs:Your evidence narrows it: at 04:52 the recorded pid was alive (
ps -o pid,etime,stat -p 2723758answered). So eitherkill -0returned non-zero for a live process, which it does forEPERM— a ticket written by a process owned by a differentuser (this becomes real the moment the
ghrunnerinstall from ci: a runner install recipe with a restart policy, an OOM guard and a dedicated user (Refs #1504) #1609 lands, and is already real for any root-run hold, e.g. asudo'd release step), or for a process in another PID namespace; or
stale()— and the only other removers are the owner's own exit paths (:140,:229), which means the waiter'sstatewas notdoneat an exit it did not intended, orprintf(line:185), which is one smallwrite and effectively atomic, so I would rank this third.
The one change that would settle it in a single occurrence
Instrument the removal, not the wait: give the
rm -fat:161(and optionally:140/:229) onelogline naming the actor —log "dropped stale ticket $f (owner pid $pid gone)"already names the ticket, so it needslabel/$$of the reaper and thekill -0errno class. Then your next occurrence answers "who reaped it and what didkill -0return" instead of leaving threecandidates. That is ~4 lines and it is orthogonal to #1636, which turns the 5400 s silence into a 1 s failure.
If you would rather have the guard than the diagnosis:
stale()could treatEPERMas alive (bash:kill -0 "$pid" 2>/dev/null || [ $? -eq 1 ]distinguishes "no such process" from "not permitted"), which is the direction that fails safe — a live ticket stays queued,a genuinely dead one is still reaped. I have not opened a PR for either: #1636 owns this function's wait loop right now, and a second
patch to the same ten lines is how two half-fixes collide.
bong-water-water-bong commentedon Nov 9, 2026
Adding the measurement to my own hypothesis (1), because "
kill -0can fail for a live process" is the kind of claim that should not beasserted:
So
stale()'s only liveness test cannot distinguish "the owner is gone" from "the owner is not ours to signal" — both are rc 1, andrc 1 reaps the ticket. On a box where every lane runs as
bcloudthat needs a cross-user ticket, which is why I would still rank cause(2) (a removal path that is not
stale()) as more likely for the two occurrences you saw. But it is a real fail-open: it becomesreachable as soon as a hold is started by root — a
sudo'd release step, or theghrunneruser from #1609 — and then a liveticket is dropped and its owner waits out
-w.The fail-safe fix is one line and does not need bash's
$?gymnastics:kill -0 "$pid" 2>/dev/null || [ $? -eq 1 ]still cannot tellthem apart, so the honest form is to ask with a tool that reports the errno class — e.g.
/proc/<pid>exists for any live process we can see regardless of ownership, so[ -d /proc/$pid ]is the right liveness test here andkill -0is the wrong one. That also fixes the zombie question I raised for #1626: a zombie keeps/proc/<pid>, so if "reaped but notyet waited" should count as gone, check
State: Ztoo.Still not opening a PR: #1636 owns this function right now, and the two changes belong in one diff with one test.
gpu-hold: the reaper leaves a note, so a lost ticket says why it went
gpu-hold: a ticket removed from the queue fails in one poll, not at -…