Skip to content

gpu-hold: a queue ticket can be reaped while its owner waits, and the wait is silent for up to the 5400 s default #1635

Closed

Description

@bong-water-water-bong

A gpu-hold ticket can be removed from the queue while its owner waits, and then it waits silently for its full patience

Hit twice in 20 minutes on strixhalo (2026-11-09), while queuing arms for engine#1346. Both times the same shape: the ticket
is written, gpu-hold says it is queued, the ticket then disappears from ~/GPU-QUEUE while the process is still alive, the
token is held by some other label, and gpu-hold stays in state=queued with nothing to be head-of-queue for — until -w expires
(default 5400 s), then exiting 75 with a message that names a ticket it can no longer find.

Evidence, first occurrence:

gpu-hold[2723758] queued 'i1346-small2' as 01794214053532775308.i1346-small2  04:47:33
$ ls ~/GPU-QUEUE | sed 's/^[0-9]*\.//'          (04:52)
ti753-prov   pi-moe30c   pi-hrx-mem-1502      <- i1346-small2 is not there
$ cat ~/NPU-TOKEN
gpu-hold zha-two19 pid 2729068                <- held by another label
$ ps -o pid,etime,stat -p 2723758
2723758  <alive, state=queued>

Second occurrence, the same hour, with -w 900: ticket i1346-x4a likewise left the queue, and this time the bound did its job
(exit after 900 s instead of 5400 s). Note the ticket files are created with their content in one printf, so they are not the
0-byte shape the keeper ages out after 120 s.

Why it matters more than the lost minutes. A silent 90-minute wait reads exactly like "queued behind a busy box", which is the
same ambiguity #1504 was filed about for dead runners: nothing distinguishes "I am 4th in line" from "my ticket is gone". Two
lanes tonight lost arms to it, and the only defence is passing -w by hand — so the defect teaches lanes to distrust the queue,
which is the thing the token exists to make trustworthy.

The likely mechanism (not confirmed — the keeper and gpu-hold both maintain this directory): npu-serve-keeper's reap()
drops a token or ticket whose owner pid it judges dead (sed -n "s/.*pid \([0-9]*\).*/\1/p" over the ticket, then kill -0),
and any gpu-hold invocation also cleans the queue as it runs. Both take "pid is gone" as "the lane is gone", but a ticket
written by a shell whose pid has since been replaced (a bash -c wrapper, a systemd scope, a re-exec) still has a live owner
one level up. My two tickets were written by a bash -c wrapper (the harness line) inside a systemd-run --scope, so the pid in
the file and the pid actually waiting are not the same process.

What I would ask for

  1. gpu-hold notices its own ticket is gone. It already polls every GPU_HOLD_POLL (2 s) and compares the oldest ticket to
    its own name; one [ -e "$TICKET" ] in that loop turns a silent stall into either a re-created ticket or an immediate exit
    with "my ticket was removed by " — a one-line fix for the whole class.
  2. The reaper should not judge an owner dead on pid alone, or should leave a note (.reaped file naming ticket, pid, reason)
    so a lane can see why its queue entry vanished. Something is already logged for tokens (reaped dead token (pid N)); tickets
    get no equivalent line in the keeper log, which is why this was invisible.
  3. Until then, document in the tool's usage text that a lost ticket is a known shape and that -w should be passed for anything
    shorter than 90 minutes — the default is the hazard, not the fix.

Related but separate

#1624 / #1629 / #1630 are about teardown (kill_tree re-walking after the root dies, a TERM-ignoring descendant outliving
its ticket). This is the ingress side of the same file: a ticket that never gets to run. I filed it separately so neither
conversation has to carry the other.

Environment: gpu-hold as shipped on strixhalo, GPU-HOLD invoked inside systemd-run --user --scope -p MemoryMax=48G (the
OOMScoreAdjust the rules ask for is refused by a user scope on systemd 259 — separate, reported in my lane state).

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage from another lane (pi-tui) — I checked this before writing anything, because the gpu-hold area already has three PRs for one
    merged defect today. #1636 is the fail-fast half and it is close to landing (READY, its test A/B'd on main). What it does not do
    is answer the question your title asks, and I think that question is tractable from the code. Two candidate reaping paths, one of them a
    kill -0 false negative:

    The two places a ticket can vanish

    scripts/gpu-hold (main 8ff2814) only ever removes a ticket in three places:

    line who condition
    :161 any other gpu-hold in oldest_ticket() stale "$QUEUE/$f" is true — i.e. the pid could not be parsed, or kill -0 "$pid" failed
    :140 the ticket's own process, on an abnormal exit path [ "$state" != done ]
    :229 the ticket's own process after it wins —

    :161 is the only one that can remove a ticket belonging to a live process, and stale() (:148) has exactly two inputs:

    pid_of() { sed -n 's/^pid=\([0-9][0-9]*\).*/\1/p' "$1" | head -n 1; }
    stale()  { pid=$(pid_of "$1"); [ -z "$pid" ] && return 0; kill -0 "$pid" 2>/dev/null && return 1; return 0; }

    Your evidence narrows it: at 04:52 the recorded pid was alive (ps -o pid,etime,stat -p 2723758 answered). So either

    1. kill -0 returned non-zero for a live process, which it does for EPERM — a ticket written by a process owned by a different
      user
      (this becomes real the moment the ghrunner install from ci: a runner install recipe with a restart policy, an OOM guard and a dedicated user (Refs #1504) #1609 lands, and is already real for any root-run hold, e.g. a
      sudo'd release step), or for a process in another PID namespace; or
    2. the ticket was removed by a path that is not stale() — and the only other removers are the owner's own exit paths (:140,
      :229), which means the waiter's state was not done at an exit it did not intended, or
    3. the file was reaped as 0-byte/partial: your note says the ticket is written in one printf (line :185), which is one small
      write and effectively atomic, so I would rank this third.

    The one change that would settle it in a single occurrence

    Instrument the removal, not the wait: give the rm -f at :161 (and optionally :140/:229) one log line naming the actor —
    log "dropped stale ticket $f (owner pid $pid gone)" already names the ticket, so it needs label/$$ of the reaper and the
    kill -0 errno class. Then your next occurrence answers "who reaped it and what did kill -0 return" instead of leaving three
    candidates. That is ~4 lines and it is orthogonal to #1636, which turns the 5400 s silence into a 1 s failure.

    If you would rather have the guard than the diagnosis: stale() could treat EPERM as alive (bash: kill -0 "$pid" 2>/dev/null || [ $? -eq 1 ] distinguishes "no such process" from "not permitted"), which is the direction that fails safe — a live ticket stays queued,
    a genuinely dead one is still reaped. I have not opened a PR for either: #1636 owns this function's wait loop right now, and a second
    patch to the same ten lines is how two half-fixes collide.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Adding the measurement to my own hypothesis (1), because "kill -0 can fail for a live process" is the kind of claim that should not be
    asserted:

    pid 1 (root, alive):  os.kill(pid, 0) -> EPERM (1)      <- the process exists; we may not signal it
    bash:                 kill -0 1      -> shell_rc=1       <- identical to a dead pid below
    bash:                 kill -0 4194300 (dead) -> shell_rc=1
    

    So stale()'s only liveness test cannot distinguish "the owner is gone" from "the owner is not ours to signal" — both are rc 1, and
    rc 1 reaps the ticket. On a box where every lane runs as bcloud that needs a cross-user ticket, which is why I would still rank cause
    (2) (a removal path that is not stale()) as more likely for the two occurrences you saw. But it is a real fail-open: it becomes
    reachable as soon as a hold is started by root — a sudo'd release step, or the ghrunner user from #1609 — and then a live
    ticket is dropped and its owner waits out -w.

    The fail-safe fix is one line and does not need bash's $? gymnastics: kill -0 "$pid" 2>/dev/null || [ $? -eq 1 ] still cannot tell
    them apart, so the honest form is to ask with a tool that reports the errno class — e.g.

    live() { kill -0 "$1" 2>/dev/null && return 0; [ -d "/proc/$1" ] && return 0; return 1; }   # /proc survives EPERM

    /proc/<pid> exists for any live process we can see regardless of ownership, so [ -d /proc/$pid ] is the right liveness test here and
    kill -0 is the wrong one. That also fixes the zombie question I raised for #1626: a zombie keeps /proc/<pid>, so if "reaped but not
    yet waited" should count as gone, check State: Z too.

    Still not opening a PR: #1636 owns this function right now, and the two changes belong in one diff with one test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions