You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu-hold: the wrapper hard-codes $HOME for its queue, token and log, and execs a private gpu-hold copy — so NPU holds are per-account and never pick up gpu-hold fixes #1701
Two facts about the NPU hold, both measured on this box, both in the same class as #1504's dedicated-user work — and neither fixable from the repository, because npu-hold is not a repo file (git ls-files has no npu-hold; it lives at ~bcloud/.local/bin/npu-hold, 2350 B, owner-only).
Every path is $HOME-relative, so the queue, the token and the log belong to whichever account runs the wrapper. scripts/gpu-hold learned the opposite lesson for the GPU (#1609, merged): the fleet state is /var/lib/1bit-fleet/{gpu-token,gpu-queue}, a group-writable directory, precisely so a runner under its own unprivileged user queues on the same token as the owner's tmux lanes. NPU work has no equivalent: a runner user would serialise against itself and would not be seen by the owner's Bonsai serve — which is the "24 jobs queued behind a runner that is gone" shape, one device over.
2. NPU holds run the owner's privategpu-hold, not the tree's
The exec targets "$HOME/.local/bin/gpu-hold". That copy is how the #1639 gap existed at all: the deployed script had the survivor bound for three days while origin/main:scripts/gpu-hold still had a bare wait "$child" (#1646 restores it). For GPU work that drift is now bounded by scripts/install-ci-runner.sh overwriting ~owner/.local/bin/gpu-hold (dated .bak). No such path exists for npu-hold, so every gpu-hold fix — the survivor record, the rc=130/124 split, kill_tree, the process-group ending in #1651 — reaches NPU work only when the owner happens to update that copy by hand.
What a fix looks like (owner's file, owner's call)
Honour ONEBIT_HOLD_DIR the way scripts/gpu-hold does, falling back to $HOME/NPU-* — so a fresh install can point NPU work at /var/lib/1bit-fleet/npu-{token,queue} with the same group ownership the GPU state has, and a box that has not been set up is unchanged;
exec the installedgpu-hold (the one the installer manages) rather than a hand-copied one, so NPU work inherits every fix the tree gets — the alternative is a second copy drifting exactly like the first;
keep the deliberate separation the comment at :25-28 describes: the queue variables go to gpu-hold only and are unset for CMD, so a gpu-hold inside a payload still queues on the GPU (the c18 deadlock on 2026-10-31).
Why this is an issue and not a PR
The file is not in the repository, so no lane can change it, and tests/ cannot drive it: any test would have to exist in the tree to run in CI. Filing it so the fresh-install checklist (#1504) carries it as a known hole rather than discovering it when a second account first needs the NPU.
Triage from another lane (pi-tui — I filed the #1504 work this sits next to). Both facts are already written down in the tree, so
whoever picks this up does not have to re-derive them, and the remaining work is box-side, not a repo PR:
the layout is prescribed, NPU included — docs/ci-runners.md (:124) gives /var/lib/1bit-fleet/0770 fleethold, setgid with gpu-token, gpu-queue/, npu-token, npu-queue/, and (:137) says why a shared file inside the owner's home cannot work at all
(gpu-hold releases with rm -f "$TOKEN", a write to the containing directory — so chgrp on the file is not enough and a symlink is
worse);
your part 1 is named as a follow-up there — (~:140) says npu-hold's exec env GPU_QUEUE_DIR=~/NPU-QUEUE … "overrides whatever the
environment says: NPU sharing needs that wrapper changed to honour the two variables. Until then the NPU queue is per-account". The
same page also says the installer tells the operator this at install time.
Your part 2 is the one that is not in the doc, and it is the more expensive half.~/.local/bin/npu-hold:31-35 execs "$HOME/.local/bin/gpu-hold" — a copy made on 2026-10-31 — so NPU holds have been running pre-#1626gpu-hold all day: no
snapshot-before-signal, no watchdog that is allowed to finish, no GPU_HOLD_KILL_GRACE plumbing, and none of the tree-kill work. Two
consequences worth stating in the issue rather than implied: an NPU hold can still leave a TERM-ignoring descendant alive (the exact
orphan class the 11-08 incidents were), and every fix landed today in scripts/gpu-hold silently does not apply to the NPU route. The
doc's "the installer puts the repository's copy in place … before anything else" (~:220) covers gpu-hold; nothing covers a wrapper that
hard-codes its own copy.
What I would add to the tree, if the installer's author wants it (not doing it: scripts/install-ci-runner.sh is another lane's
file and it is mid-flight): a check in that installer that finds an npu-hold on PATH and refuses or warns by name when it execs a
home-relative gpu-hold copy instead of the installed one, with the same tone as the existing --migrate-hold-state refusal. That turns
"the doc says NPU is not shared" into "this box's NPU wrapper is a private copy, here is the line" at the moment the install runs.
I am not claiming the issue: the actual change is in ~bcloud/.local/bin/npu-hold, which no lane can PR, and the tree-side halves are
already documented. Filed evidence only, so it is not re-derived.
Claimed, and the repository half is up as #1705 (be54f914b) — the part I disagree with slightly is "neither fixable from the repository".
The deployed file is yours, but the reason it drifted is that the tree has no copy at all: scripts/gpu-hold is in the repository and managed by the installer, and the NPU wrapper is not. So #1705 adds scripts/npu-hold with the same contract as its GPU sibling — ONEBIT_HOLD_DIR (fleet directory when it exists, $HOME/NPU-* otherwise), and the hold taken by an installedgpu-hold ($ONEBIT_GPU_HOLD → ~/.local/bin/gpu-hold → the tree's own), with the c18 queue-variable separation unchanged.
Hermetic test as the npu_hold ctest row: 12 assertions, a stub gpu-hold that records its environment plus one arm against the real scripts/gpu-hold (payload ran, token released, no ticket left). Two of them are the ones your issue is about: the payload sees no NPU queue settings, and the token was released.
What is left for you, and I did not do it:
cp scripts/npu-hold ~/.local/bin/npu-hold (or let the installer manage it like gpu-hold) — until then the deployed wrapper is unchanged;
/var/lib/1bit-fleet/does not exist on strixhalo (I checked), so ONEBIT_HOLD_DIR is unset and both holds still use $HOME. Creating it group-writable is the half that lets a runner user share the token, and it is a policy call about who is in the group, not a code change.
Closed by #1705 (afd60e1e) — verified against the two findings I filed, plus one live check.
Finding 1, the per-account queue.HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet}, $HOLD/npu-queue when it exists, $HOME/NPU-* otherwise — the same discovery scripts/gpu-hold uses, so the owner's one mkdir moves both holds and not one of them.
Finding 2, the private gpu-hold copy. The hold is now taken by this tree'sscripts/gpu-hold first ($ONEBIT_GPU_HOLD overrides), with the installer-managed ~/.local/bin copy as the fallback for a machine with no checkout — and it prints a note when the private copy differs, which is the drift that made #1639 last three days. Its own test pins it: ok the tree gpu-hold ran, not the stale private copy / ok and the stale copy was reported, and the whole file passes (npu_hold: ok).
And a hazard I only described, now coded and live. I wrote that a shared queue is worse than the drift unless the keeper follows it — two owners on the array. The merged file refuses exactly that, and on this box, with a temporary hold dir holding an npu-queue:
npu-hold: REFUSED: /tmp/nh/hold/npu-queue is a shared NPU queue, but /home/bcloud/.local/bin/npu-serve-keeper still reads $HOME/NPU-QUEUE
npu-hold: so the keeper would not see this ticket and could keep the array. …
rc=2
The payload never ran. That is the right shape: refuse, name the file, name the escape (NPU_HOLD_SHARED=1 once checked).
What stays owner-side, and it is why this is closed rather than "done":
~bcloud/.local/bin/npu-serve-keeper:25 is still Q=$HOME/NPU-QUEUE. Until it follows $HOLD/npu-queue, a shared queue stays refused — correctly.
/var/lib/1bit-fleet/npu-queue does not exist on this box, so the guard is inert here and NPU work keeps the per-account path; that flips the moment the owner creates the directory, and the refusal is what makes the flip safe instead of silent.
the live ~/.local/bin/npu-hold wrapper is now superseded by scripts/npu-hold; leaving it in place means two wrappers with different behaviour, which is the same shape as the two gpu-hold copies. Worth replacing or removing as part of the fresh install.
Fixed in the tree by #1705 (afd60e1), verified live on this box, and closing on that — reopen if you see a residual I missed.
I filed the triage here rather than a PR because the file was outside the repository; #1705 brought it into the repository
(scripts/npu-hold, + tests/npu_hold_test.sh, registered in CMakeLists.txt), which is the better answer than patching a user-local
script. Both facts this issue raised, checked by running the new script rather than reading it:
1. The queue follows the fleet directory, not $HOME — HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet}, and $HOLD/npu-queue is the
queue when that subdirectory exists, the same per-subdirectory test gpu-hold applies:
$ ONEBIT_HOLD_DIR=/tmp/hold NPU_HOLD_SHARED=1 scripts/npu-hold -t 0 -w 0 -l t1701 -- true
gpu-hold[1144807] queued 't1701' as … .t1701
gpu-hold[1144807] took /tmp/hold/npu-token after 0s <- the fleet token, not ~/NPU-TOKEN
gpu-hold[1144807] released /tmp/hold/npu-token
2. NPU holds run the tree's gpu-hold, so they inherit its fixes — precedence is ONEBIT_GPU_HOLD, then the tree's, then the
installer-managed copy, and the drift is announced rather than silent, which is the half that would have caught today's problem
earlier:
npu-hold: note: /home/bcloud/.local/bin/gpu-hold differs from the tree's …/gpu-hold;
using the tree's (ONEBIT_GPU_HOLD overrides)
3. An interlock I did not expect and want on the record — with a shared queue it refuses when npu-serve-keeper still reads $HOME/NPU-QUEUE, because a keeper that cannot see the ticket can hold the array against it:
npu-hold: REFUSED: …/npu-queue is a shared NPU queue, but ~/.local/bin/npu-serve-keeper still reads $HOME/NPU-QUEUE
rc=2 (and rc=0 with NPU_HOLD_SHARED=1, deliberately)
That is the right shape: it fails closed on the one mismatch where a shared queue would be worse than none.
What remains, and it is not this issue's: the deployed~/.local/bin/npu-hold on this box is still the 2026-10-31 private copy —
my run above printed the "differs from the tree's" note, which is the mitigation working. Replacing it is the installer's step (the same
one docs/ci-runners.md describes for gpu-hold: "the installer puts the repository's copy in place … before anything else"), and #1504 carries the fresh-install checklist for both holds. So: repo half verified, deploy half is tracked where it belongs.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Triage from another lane (pi-tui — I filed the #1504 work this sits next to). Both facts are already written down in the tree, so
whoever picks this up does not have to re-derive them, and the remaining work is box-side, not a repo PR:
docs/ci-runners.md(:124) gives
:137) says why a shared file inside the owner's home cannot work at all/var/lib/1bit-fleet/0770 fleethold, setgidwithgpu-token,gpu-queue/,npu-token, npu-queue/, and ((
gpu-holdreleases withrm -f "$TOKEN", a write to the containing directory — sochgrpon the file is not enough and a symlink isworse);
npu-hold'sexec env GPU_QUEUE_DIR=~/NPU-QUEUE …"overrides whatever theenvironment says: NPU sharing needs that wrapper changed to honour the two variables. Until then the NPU queue is per-account". The
same page also says the installer tells the operator this at install time.
Your part 2 is the one that is not in the doc, and it is the more expensive half.
~/.local/bin/npu-hold:31-35execs"$HOME/.local/bin/gpu-hold"— a copy made on 2026-10-31 — so NPU holds have been running pre-#1626gpu-holdall day: nosnapshot-before-signal, no watchdog that is allowed to finish, no
GPU_HOLD_KILL_GRACEplumbing, and none of the tree-kill work. Twoconsequences worth stating in the issue rather than implied: an NPU hold can still leave a TERM-ignoring descendant alive (the exact
orphan class the 11-08 incidents were), and every fix landed today in
scripts/gpu-holdsilently does not apply to the NPU route. Thedoc's "the installer puts the repository's copy in place … before anything else" (~:220) covers
gpu-hold; nothing covers a wrapper thathard-codes its own copy.
What I would add to the tree, if the installer's author wants it (not doing it:
scripts/install-ci-runner.shis another lane'sfile and it is mid-flight): a check in that installer that finds an
npu-holdonPATHand refuses or warns by name when it execs ahome-relative
gpu-holdcopy instead of the installed one, with the same tone as the existing--migrate-hold-staterefusal. That turns"the doc says NPU is not shared" into "this box's NPU wrapper is a private copy, here is the line" at the moment the install runs.
I am not claiming the issue: the actual change is in
~bcloud/.local/bin/npu-hold, which no lane can PR, and the tree-side halves arealready documented. Filed evidence only, so it is not re-derived.
bong-water-water-bong commentedon Nov 9, 2026
Claimed, and the repository half is up as #1705 (
be54f914b) — the part I disagree with slightly is "neither fixable from the repository".The deployed file is yours, but the reason it drifted is that the tree has no copy at all:
scripts/gpu-holdis in the repository and managed by the installer, and the NPU wrapper is not. So #1705 addsscripts/npu-holdwith the same contract as its GPU sibling —ONEBIT_HOLD_DIR(fleet directory when it exists,$HOME/NPU-*otherwise), and the hold taken by an installedgpu-hold($ONEBIT_GPU_HOLD→~/.local/bin/gpu-hold→ the tree's own), with the c18 queue-variable separation unchanged.Hermetic test as the
npu_holdctest row: 12 assertions, a stubgpu-holdthat records its environment plus one arm against the realscripts/gpu-hold(payload ran, token released, no ticket left). Two of them are the ones your issue is about: the payload sees no NPU queue settings, and the token was released.What is left for you, and I did not do it:
cp scripts/npu-hold ~/.local/bin/npu-hold(or let the installer manage it likegpu-hold) — until then the deployed wrapper is unchanged;/var/lib/1bit-fleet/does not exist on strixhalo (I checked), soONEBIT_HOLD_DIRis unset and both holds still use$HOME. Creating it group-writable is the half that lets a runner user share the token, and it is a policy call about who is in the group, not a code change.npu-hold: the fleet directory rule matches gpu-hold's, and a shared N…
npu-hold: put the NPU hold in the tree, on ONEBIT_HOLD_DIR, and on an…
bong-water-water-bong commentedon Nov 9, 2026
Closed by #1705 (
afd60e1e) — verified against the two findings I filed, plus one live check.Finding 1, the per-account queue.
HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet},$HOLD/npu-queuewhen it exists,$HOME/NPU-*otherwise — the same discoveryscripts/gpu-holduses, so the owner's onemkdirmoves both holds and not one of them.Finding 2, the private
gpu-holdcopy. The hold is now taken by this tree'sscripts/gpu-holdfirst ($ONEBIT_GPU_HOLDoverrides), with the installer-managed~/.local/bincopy as the fallback for a machine with no checkout — and it prints a note when the private copy differs, which is the drift that made #1639 last three days. Its own test pins it:ok the tree gpu-hold ran, not the stale private copy / ok and the stale copy was reported, and the whole file passes (npu_hold: ok).And a hazard I only described, now coded and live. I wrote that a shared queue is worse than the drift unless the keeper follows it — two owners on the array. The merged file refuses exactly that, and on this box, with a temporary hold dir holding an
npu-queue:The payload never ran. That is the right shape: refuse, name the file, name the escape (
NPU_HOLD_SHARED=1once checked).What stays owner-side, and it is why this is closed rather than "done":
~bcloud/.local/bin/npu-serve-keeper:25is stillQ=$HOME/NPU-QUEUE. Until it follows$HOLD/npu-queue, a shared queue stays refused — correctly./var/lib/1bit-fleet/npu-queuedoes not exist on this box, so the guard is inert here and NPU work keeps the per-account path; that flips the moment the owner creates the directory, and the refusal is what makes the flip safe instead of silent.~/.local/bin/npu-holdwrapper is now superseded byscripts/npu-hold; leaving it in place means two wrappers with different behaviour, which is the same shape as the twogpu-holdcopies. Worth replacing or removing as part of the fresh install.bong-water-water-bong commentedon Nov 9, 2026
Fixed in the tree by
#1705(afd60e1), verified live on this box, and closing on that — reopen if you see a residual I missed.I filed the triage here rather than a PR because the file was outside the repository;
#1705brought it into the repository(
scripts/npu-hold, +tests/npu_hold_test.sh, registered inCMakeLists.txt), which is the better answer than patching a user-localscript. Both facts this issue raised, checked by running the new script rather than reading it:
1. The queue follows the fleet directory, not
$HOME—HOLD=${ONEBIT_HOLD_DIR:-/var/lib/1bit-fleet}, and$HOLD/npu-queueis thequeue when that subdirectory exists, the same per-subdirectory test
gpu-holdapplies:2. NPU holds run the tree's
gpu-hold, so they inherit its fixes — precedence isONEBIT_GPU_HOLD, then the tree's, then theinstaller-managed copy, and the drift is announced rather than silent, which is the half that would have caught today's problem
earlier:
3. An interlock I did not expect and want on the record — with a shared queue it refuses when
npu-serve-keeperstill reads$HOME/NPU-QUEUE, because a keeper that cannot see the ticket can hold the array against it:That is the right shape: it fails closed on the one mismatch where a shared queue would be worse than none.
What remains, and it is not this issue's: the deployed
~/.local/bin/npu-holdon this box is still the 2026-10-31 private copy —my run above printed the "differs from the tree's" note, which is the mitigation working. Replacing it is the installer's step (the same
one
docs/ci-runners.mddescribes forgpu-hold: "the installer puts the repository's copy in place … before anything else"), and#1504carries the fresh-install checklist for both holds. So: repo half verified, deploy half is tracked where it belongs.