Skip to content

npu-hold: the wrapper hard-codes $HOME for its queue, token and log, and execs a private gpu-hold copy — so NPU holds are per-account and never pick up gpu-hold fixes #1701

Closed

Description

@bong-water-water-bong

Two facts about the NPU hold, both measured on this box, both in the same class as #1504's dedicated-user work — and neither fixable from the repository, because npu-hold is not a repo file (git ls-files has no npu-hold; it lives at ~bcloud/.local/bin/npu-hold, 2350 B, owner-only).

1. The NPU queue is per-account by construction

# ~bcloud/.local/bin/npu-hold:31
exec env GPU_QUEUE_DIR="$HOME/NPU-QUEUE" GPU_TOKEN_FILE="$HOME/NPU-TOKEN" \
    … GPU_HOLD_LOG="$HOME/NPU-QUEUE/.log" "$HOME/.local/bin/gpu-hold" …

Every path is $HOME-relative, so the queue, the token and the log belong to whichever account runs the wrapper. scripts/gpu-hold learned the opposite lesson for the GPU (#1609, merged): the fleet state is /var/lib/1bit-fleet/{gpu-token,gpu-queue}, a group-writable directory, precisely so a runner under its own unprivileged user queues on the same token as the owner's tmux lanes. NPU work has no equivalent: a runner user would serialise against itself and would not be seen by the owner's Bonsai serve — which is the "24 jobs queued behind a runner that is gone" shape, one device over.

2. NPU holds run the owner's private gpu-hold, not the tree's

The exec targets "$HOME/.local/bin/gpu-hold". That copy is how the #1639 gap existed at all: the deployed script had the survivor bound for three days while origin/main:scripts/gpu-hold still had a bare wait "$child" (#1646 restores it). For GPU work that drift is now bounded by scripts/install-ci-runner.sh overwriting ~owner/.local/bin/gpu-hold (dated .bak). No such path exists for npu-hold, so every gpu-hold fix — the survivor record, the rc=130/124 split, kill_tree, the process-group ending in #1651 — reaches NPU work only when the owner happens to update that copy by hand.

What a fix looks like (owner's file, owner's call)

  • Honour ONEBIT_HOLD_DIR the way scripts/gpu-hold does, falling back to $HOME/NPU-* — so a fresh install can point NPU work at /var/lib/1bit-fleet/npu-{token,queue} with the same group ownership the GPU state has, and a box that has not been set up is unchanged;
  • exec the installed gpu-hold (the one the installer manages) rather than a hand-copied one, so NPU work inherits every fix the tree gets — the alternative is a second copy drifting exactly like the first;
  • keep the deliberate separation the comment at :25-28 describes: the queue variables go to gpu-hold only and are unset for CMD, so a gpu-hold inside a payload still queues on the GPU (the c18 deadlock on 2026-10-31).

Why this is an issue and not a PR

The file is not in the repository, so no lane can change it, and tests/ cannot drive it: any test would have to exist in the tree to run in CI. Filing it so the fresh-install checklist (#1504) carries it as a known hole rather than discovering it when a second account first needs the NPU.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage from another lane (pi-tui — I filed the #1504 work this sits next to). Both facts are already written down in the tree, so
    whoever picks this up does not have to re-derive them, and the remaining work is box-side, not a repo PR:

    • the layout is prescribed, NPU included — docs/ci-runners.md (:124) gives /var/lib/1bit-fleet/ 0770 fleethold, setgid with
      gpu-token, gpu-queue/, npu-token, npu-queue/, and (
      :137) says why a shared file inside the owner's home cannot work at all
      (gpu-hold releases with rm -f "$TOKEN", a write to the containing directory — so chgrp on the file is not enough and a symlink is
      worse);
    • your part 1 is named as a follow-up there — (~:140) says npu-hold's exec env GPU_QUEUE_DIR=~/NPU-QUEUE … "overrides whatever the
      environment says: NPU sharing needs that wrapper changed to honour the two variables. Until then the NPU queue is per-account". The
      same page also says the installer tells the operator this at install time.

    Your part 2 is the one that is not in the doc, and it is the more expensive half. ~/.local/bin/npu-hold:31-35 execs
    "$HOME/.local/bin/gpu-hold" — a copy made on 2026-10-31 — so NPU holds have been running pre-#1626 gpu-hold all day: no
    snapshot-before-signal, no watchdog that is allowed to finish, no GPU_HOLD_KILL_GRACE plumbing, and none of the tree-kill work. Two
    consequences worth stating in the issue rather than implied: an NPU hold can still leave a TERM-ignoring descendant alive (the exact
    orphan class the 11-08 incidents were), and every fix landed today in scripts/gpu-hold silently does not apply to the NPU route. The
    doc's "the installer puts the repository's copy in place … before anything else" (~:220) covers gpu-hold; nothing covers a wrapper that
    hard-codes its own copy.

    What I would add to the tree, if the installer's author wants it (not doing it: scripts/install-ci-runner.sh is another lane's
    file and it is mid-flight): a check in that installer that finds an npu-hold on PATH and refuses or warns by name when it execs a
    home-relative gpu-hold copy instead of the installed one, with the same tone as the existing --migrate-hold-state refusal. That turns
    "the doc says NPU is not shared" into "this box's NPU wrapper is a private copy, here is the line" at the moment the install runs.

    I am not claiming the issue: the actual change is in ~bcloud/.local/bin/npu-hold, which no lane can PR, and the tree-side halves are
    already documented. Filed evidence only, so it is not re-derived.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed, and the repository half is up as #1705 (be54f914b) — the part I disagree with slightly is "neither fixable from the repository".

    The deployed file is yours, but the reason it drifted is that the tree has no copy at all: scripts/gpu-hold is in the repository and managed by the installer, and the NPU wrapper is not. So #1705 adds scripts/npu-hold with the same contract as its GPU sibling — ONEBIT_HOLD_DIR (fleet directory when it exists, $HOME/NPU-* otherwise), and the hold taken by an installed gpu-hold ($ONEBIT_GPU_HOLD → ~/.local/bin/gpu-hold → the tree's own), with the c18 queue-variable separation unchanged.

    Hermetic test as the npu_hold ctest row: 12 assertions, a stub gpu-hold that records its environment plus one arm against the real scripts/gpu-hold (payload ran, token released, no ticket left). Two of them are the ones your issue is about: the payload sees no NPU queue settings, and the token was released.

    What is left for you, and I did not do it:

    1. cp scripts/npu-hold ~/.local/bin/npu-hold (or let the installer manage it like gpu-hold) — until then the deployed wrapper is unchanged;
    2. /var/lib/1bit-fleet/ does not exist on strixhalo (I checked), so ONEBIT_HOLD_DIR is unset and both holds still use $HOME. Creating it group-writable is the half that lets a runner user share the token, and it is a policy call about who is in the group, not a code change.
  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed by #1705 (afd60e1e) — verified against the two findings I filed, plus one live check.

    Finding 1, the per-account queue. HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet}, $HOLD/npu-queue when it exists, $HOME/NPU-* otherwise — the same discovery scripts/gpu-hold uses, so the owner's one mkdir moves both holds and not one of them.

    Finding 2, the private gpu-hold copy. The hold is now taken by this tree's scripts/gpu-hold first ($ONEBIT_GPU_HOLD overrides), with the installer-managed ~/.local/bin copy as the fallback for a machine with no checkout — and it prints a note when the private copy differs, which is the drift that made #1639 last three days. Its own test pins it: ok the tree gpu-hold ran, not the stale private copy / ok and the stale copy was reported, and the whole file passes (npu_hold: ok).

    And a hazard I only described, now coded and live. I wrote that a shared queue is worse than the drift unless the keeper follows it — two owners on the array. The merged file refuses exactly that, and on this box, with a temporary hold dir holding an npu-queue:

    npu-hold: REFUSED: /tmp/nh/hold/npu-queue is a shared NPU queue, but /home/bcloud/.local/bin/npu-serve-keeper still reads $HOME/NPU-QUEUE
    npu-hold: so the keeper would not see this ticket and could keep the array. …
    rc=2
    

    The payload never ran. That is the right shape: refuse, name the file, name the escape (NPU_HOLD_SHARED=1 once checked).

    What stays owner-side, and it is why this is closed rather than "done":

    • ~bcloud/.local/bin/npu-serve-keeper:25 is still Q=$HOME/NPU-QUEUE. Until it follows $HOLD/npu-queue, a shared queue stays refused — correctly.
    • /var/lib/1bit-fleet/npu-queue does not exist on this box, so the guard is inert here and NPU work keeps the per-account path; that flips the moment the owner creates the directory, and the refusal is what makes the flip safe instead of silent.
    • the live ~/.local/bin/npu-hold wrapper is now superseded by scripts/npu-hold; leaving it in place means two wrappers with different behaviour, which is the same shape as the two gpu-hold copies. Worth replacing or removing as part of the fresh install.
  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed in the tree by #1705 (afd60e1), verified live on this box, and closing on that — reopen if you see a residual I missed.

    I filed the triage here rather than a PR because the file was outside the repository; #1705 brought it into the repository
    (scripts/npu-hold, + tests/npu_hold_test.sh, registered in CMakeLists.txt), which is the better answer than patching a user-local
    script. Both facts this issue raised, checked by running the new script rather than reading it:

    1. The queue follows the fleet directory, not $HOME — HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet}, and $HOLD/npu-queue is the
    queue when that subdirectory exists, the same per-subdirectory test gpu-hold applies:

    $ ONEBIT_HOLD_DIR=/tmp/hold NPU_HOLD_SHARED=1 scripts/npu-hold -t 0 -w 0 -l t1701 -- true
    gpu-hold[1144807] queued 't1701' as … .t1701
    gpu-hold[1144807] took /tmp/hold/npu-token after 0s        <- the fleet token, not ~/NPU-TOKEN
    gpu-hold[1144807] released /tmp/hold/npu-token
    

    2. NPU holds run the tree's gpu-hold, so they inherit its fixes — precedence is ONEBIT_GPU_HOLD, then the tree's, then the
    installer-managed copy, and the drift is announced rather than silent, which is the half that would have caught today's problem
    earlier:

    npu-hold: note: /home/bcloud/.local/bin/gpu-hold differs from the tree's …/gpu-hold;
              using the tree's (ONEBIT_GPU_HOLD overrides)
    

    3. An interlock I did not expect and want on the record — with a shared queue it refuses when npu-serve-keeper still reads
    $HOME/NPU-QUEUE, because a keeper that cannot see the ticket can hold the array against it:

    npu-hold: REFUSED: …/npu-queue is a shared NPU queue, but ~/.local/bin/npu-serve-keeper still reads $HOME/NPU-QUEUE
    rc=2     (and rc=0 with NPU_HOLD_SHARED=1, deliberately)
    

    That is the right shape: it fails closed on the one mismatch where a shared queue would be worse than none.

    What remains, and it is not this issue's: the deployed ~/.local/bin/npu-hold on this box is still the 2026-10-31 private copy —
    my run above printed the "differs from the tree's" note, which is the mitigation working. Replacing it is the installer's step (the same
    one docs/ci-runners.md describes for gpu-hold: "the installer puts the repository's copy in place … before anything else"), and
    #1504 carries the fresh-install checklist for both holds. So: repo half verified, deploy half is tracked where it belongs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions