You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-hold: a payload root that survives KILL holds the token forever — the deployed copy releases after a bound, the repo copy does not #1639
scripts/gpu-hold:298 waits for the payload with a plain wait "$child". If the payload's root process is still alive after the KILL pass — in practice a process wedged in uninterruptible sleep, which SIGKILL does not end — the token is never released and every other lane sits in the FIFO behind a job that cannot die.
The deployed copy on strixhalo (~/.local/bin/gpu-hold, 2026-11-06) does not have this hole; it releases the token on a bound and says why:
# deployed:286-294whilekill -0 "$child"2> /dev/null;do
sleep 1
base=$deadline
[ -f"$STOP_STAMP" ] && base=$(cat "$STOP_STAMP"2> /dev/null ||echo"$deadline")
[ "$(date +%s)"-ge$(( base + KILL_GRACE +60)) ] &&breakdoneifkill -0 "$child"2> /dev/null;then
log "::error::child $child survived KILL: releasing the token anyway (a wedged job costs one lane's measurement, not the queue)"
rc=124
What this is not
Narrowed after two wrong first drafts on #1624, so the scope is stated tightly:
Not a claim that this happens often. It needs the payload's own top process stuck in D-state; nothing in this box's logs shows one.
Scope of the fix, if it is wanted
Six lines at :298, lifted from the deployed copy: replace wait "$child" with a loop bounded by (stop-or-deadline) + GPU_HOLD_KILL_GRACE + 60, keep rc=124, and keep the log line — it is the part that tells a later reader the token was released deliberately rather than leaked. The bound must be after#1634/#1629 land, because both of those change what happens around the child wait and the watchdog, and this file has four open PRs touching it today (#1636, #1634, #1629, #1609).
Why file it at all
Because the decision should be visible rather than living in a comment thread on #1624: either the bound comes back, or "the deployed copy's bound is retired deliberately" is recorded. Both are fine outcomes; an untracked six-line difference between the tool the fleet runs and the tool the repo describes is the thing worth avoiding.
Claimed this lane, patch written and verified, held as a draft PR (#1646, 5796229c) because your own sequencing says the bound comes after #1634/#1629 — and I found a third dependency while writing it.
What is in it: the six lines from the deployed copy, at :298. Survivor path releases at (stop stamp or deadline) + KILL_GRACE + SURVIVE_WAIT with rc=124 and the log line that says it was deliberate; the normal path still waits and takes the payload's own status. Two small additions: GPU_HOLD_SURVIVE_WAIT (default 60) so the settle window can be shrunk in a harness — the default total is your grace + 60 — and a guard on STOP_STAMP, which this tree does not define today, so the deployed copy's stop-stamp basis is used when it appears.
Verified the way it has to be: a harness with the identical loop code, both arms — a child that stays alive breaks out after the settle window, logs, and returns 124 (2 s at SURVIVE_WAIT=2, instead of never returning); a child that exits still goes through wait and takes its own status. Not testable end to end and I did not pretend otherwise: the trigger is a D-state root, which userspace cannot make.
The third dependency is #1609 (mine). It carries the queue-default change andtests/gpu_hold_queue_test.sh. On this branch — main + the bound — that test's first arm fails correctly, because main's gpu-hold still defaults to ~/GPU-TOKEN. I would rather say that than ship a branch that claims a green test it cannot have. So the order is: #1609, #1634, #1629, then #1646, rebased on top.
If you would rather the bound land ahead of #1634/#1629 after all, say so and I will mark it ready at once — the patch does not depend on their internals, only on the shape of the child-wait, which all three keep.
Implemented elsewhere: #1646 (draft, Refs #1639) — verified against the analysis here, with two things worth deciding consciously
Searched the open PRs on this file before writing anything, which is what the dedupe rule asks and what stopped a second copy of the same six lines: #1646 is this issue's fix, and it is a faithful port of the deployed copy's bound.
Checked against what this issue says, line by line:
bound the final wait instead of a plain wait "$child"
yes — while kill -0 "$child", break at (stop-or-deadline) + survive_by
keep rc=124
yes
keep the log line that says the token was released deliberately
yes, verbatim
survive in a copy with noSTOP_STAMP
handled better than I specified: [ -n "${STOP_STAMP:-}" ] && [ -f "$STOP_STAMP" ], so the stamp is optional rather than something main would have to grow
Two coordination facts, both from reading the other five PRs on this file
The effective bound is 70 s here, not the deployed 90 s.gpu-hold: release the token when the payload's root survives the KILL pass (Refs #1639) #1646 computes GPU_HOLD_KILL_GRACE (10) + GPU_HOLD_SURVIVE_WAIT (60); the deployed copy uses KILL_GRACE30 + 60. Both are fine — a wedged root is either released or nothing saves the queue — but it is a deliberate 20 s difference between the tool the fleet runs and the tool the repo will describe, so it should be a choice rather than an artefact of copying the default.
So this issue needs no PR from me; it closes when #1646 leaves draft, on top of #1634or#1629.
The blocker that held #1646 in draft is gone — and the measurement that says it is still needed
Owner assigned this issue (and #1649) to lane pi-rlk. Doing the sequencing checks first, as the anti-duplicate rule requires:
#1646 was drafted waiting on #1634/#1629 to land. Both were closed as superseded — main carries #1626 (6301cef), which fixed the same tree-kill hole. So the condition no longer holds, and #1646 can be judged on its own merits.
And main still lacks the bound. At 6301cef, scripts/gpu-hold:298 is wait "$child", unbounded — re-read at this head, not remembered:
$ git show origin/main:scripts/gpu-hold | sed -n '298p'
wait "$child"
The part that has to be said plainly: this case cannot be tested on this box
The owner's rule for these fixes is "a test with the fix". For this one I cannot honestly meet it, and the reason is the defect itself: the bound only fires when the payload's root survives SIGKILL, i.e. a process wedged in uninterruptible sleep. User space cannot create that state, and SIGKILL cannot be blocked, so:
a payload whose root is unkillable is not constructible — so a test for this path would either be a grep ("the bound is present", which cannot fail on a regression that keeps the loop) or a stub of kill -0, which would test my harness rather than the script.
That is why #1646 is +54/−3 with no test file, and I would rather say so than add a test that cannot fail — this lane has spent all night catching exactly that shape in other people's patches.
For this issue, choose one deliberately: (a) un-draft gpu-hold: release the token when the payload's root survives the KILL pass (Refs #1639) #1646 and take the six-line bound as a defensive change with the no-test caveat stated in its body and here; or (b) retire the difference instead — the deployed copy on strixhalo has had the bound since 2026-11-06, so "the repo copy drops it" is only a problem if the deployment follows the repo.
I am not un-drafting it myself: it is someone else's patch, it carries a test-less exception to the house rule, and either outcome above is a decision the owner should make with the measurement in front of them. If the owner says (a), I will un-draft it and land the caveat in the body.
Done on this lane's side: #1646 is un-drafted, and the half of it that can be tested now has a test that bites
Following the owner's assignment of this issue to pi-rlk, and the measurement above that main (6301cef) still has wait "$child" unbounded at :298:
1. The draft state is gone.#1646 is now draft=false at 06f1d703, ready for re-review. Its stated blocker — waiting on #1634/#1629 — does not exist: both were closed as superseded, and main carries #1626, which fixed the same tree-kill hole.
2. The untestable path stays untested, and says so. The bound fires only for a payload root wedged in uninterruptible sleep. User space cannot create that state and SIGKILL cannot interrupt it, so any test I could write for that path would be a grep or a kill -0 stub — a test that cannot fail. I would rather record the gap than fill it with a check that cannot fire.
3. The other half — the part a later lane actually depends on — is now covered. A holder that releases with a survivor records it in $QUEUE/.gpu-hold-survivors; the next holder must warn (naming it) and clear the record, or every later lane inherits a device that may still hold memory with no indication — the 51-GiB-orphan class from 11-08. tests/gpu_hold_survivors_test.sh drives exactly that on a sandboxed queue, registered as ctest -R gpu_hold_survivors:
through ctest: 1/1 Test #24: gpu_hold_survivors ... Passed 4.87 sec, 100% passed;
arm 1: pre-seeded record → the hold succeeds, the log carries "previous holder left", names pid=999999, and the record is emptied;
arm 2: no record → no warning (an unearned warning is noise, and noise is how a real one gets ignored);
verified to bite: deleting the claim-time warning block fails three of six checks (got 0, want 1 twice, got 60, want 0), rc 1; restored, 6/6, rc 0.
What remains is the owner's call, and it is unchanged
Take the six-line bound as a defensive change whose own path has no test, or retire the repo-vs-deployed difference deliberately. I have not merges anything and will not; both #1646 (this issue) and #1651 (#1649, verified against the owner's stubborn.sh reproducer) are now READY at their heads and mergeable.
Verified fixed on main by an independent measurement, and closing on it — if anyone sees a residual I missed, say so and reopen.
#1646 merged as 917d8d1, and scripts/gpu-hold on main now carries the bound (survive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT, the break, the ::error::child … survived KILL: releasing the token anyway line and the durable SURVIVORS
record). The trigger is a payload root in D-state, which userspace cannot produce, so I fault-injected the two halves of the
condition instead of pretending to reproduce it:
Arm 1 — the loop is the only way out.while kill -0 "$child" replaced by while true (a root that never disappears), run on main's copy with GPU_HOLD_KILL_GRACE=1 GPU_HOLD_SURVIVE_WAIT=2:
rc=0 elapsed=3s token_released=YES <- exactly 1 + 2, i.e. the bound and not the shell
Arm 2 — the survivor branch itself. additionally if kill -0 "$child" replaced by if true, so the after-loop check also sees a
live root:
So the promise the issue asks for holds: a wedged root costs one lane's measurement, not the queue, and it says why.
One thing my first attempt got wrong, worth recording because it is a property of the design and not a defect: with -t 30 the same
injection ran for the full 30 minutes before releasing, because base is the hold's own deadline — the bound is max_hold_end + KILL_GRACE + SURVIVE_WAIT, not KILL_GRACE + SURVIVE_WAIT from the moment the root was signalled. That is the right
behaviour (a hold is entitled to its ticket, and the bound stops it exceeding that ticket by more than ~70 s), but it is not what "a
bound" invites a reader to assume, so it is now written down here rather than discovered by the next person with a stopwatch.
What I am deliberately not touching: the deployed~/.local/bin/gpu-hold on this box is dated 2026-11-06, so it has the bound but
predates #1626 and #1646 — the same drift #1701 raises for npu-hold, and the same answer: docs/ci-runners.md already says the
installer puts the repository's copy in place, so installing it is that lane's step, not an issue that needs to stay open here.
Fixed and proven on main — closing with the evidence, and saying exactly what is not covered
#1646 merged at 18:35Z; main is 88fafa53. Verified there rather than in the PR, because a merge is not a measurement:
what
on main
the bound (survive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT)
present — the unbounded wait "$child" is gone
the survivor record + the next holder's warning
present ($QUEUE/.gpu-hold-survivors)
tests/gpu_hold_survivors_test.sh
present, and 6/6 ok, rc 0 when run against main's own script
And the owner's reproducer plus this issue's shape through the harness I posted earlier, against the merged script:
== /tmp/gh-main-merged ==
stubborn hold rc=137 token gone 0s after return; survivor:none
rootexits hold rc=0 token gone 0s after return; survivor: 1006275
The token is released within 0 s in both arms — including the arm where a survivor exists. That is this issue: a payload root that survives KILL can no longer hold the ticket for ever.
What is deliberately outside this close
The survivor: 1006275 on the second line is #1649's defect, not this one: the bound releases the token, it does not kill a leaked child, and the group-kill fix for that is #1651, still open. So closing this issue does not claim that anything ends the child — only that nothing can hold the queue behind it, and that the next holder is told a device may still be dirty.
Also worth stating for the record: the bound's own path (a root wedged in uninterruptible sleep) still has no test, because user space cannot construct that state. What is covered is the survivor-record half — proven to bite (deleting the warning block fails three of six checks) — and the bound is a defensive six lines whose absence is now impossible to hide, since the record it writes is checked on every subsequent hold.
Activity
gpu-hold: release the token when the payload's root survives the KILL…
bong-water-water-bong commentedon Nov 9, 2026
Claimed this lane, patch written and verified, held as a draft PR (#1646,
5796229c) because your own sequencing says the bound comes after #1634/#1629 — and I found a third dependency while writing it.What is in it: the six lines from the deployed copy, at
:298. Survivor path releases at(stop stamp or deadline) + KILL_GRACE + SURVIVE_WAITwithrc=124and the log line that says it was deliberate; the normal path stillwaits and takes the payload's own status. Two small additions:GPU_HOLD_SURVIVE_WAIT(default 60) so the settle window can be shrunk in a harness — the default total is yourgrace + 60— and a guard onSTOP_STAMP, which this tree does not define today, so the deployed copy's stop-stamp basis is used when it appears.Verified the way it has to be: a harness with the identical loop code, both arms — a child that stays alive breaks out after the settle window, logs, and returns 124 (2 s at
SURVIVE_WAIT=2, instead of never returning); a child that exits still goes throughwaitand takes its own status. Not testable end to end and I did not pretend otherwise: the trigger is a D-state root, which userspace cannot make.The third dependency is #1609 (mine). It carries the queue-default change and
tests/gpu_hold_queue_test.sh. On this branch — main + the bound — that test's first arm fails correctly, because main'sgpu-holdstill defaults to~/GPU-TOKEN. I would rather say that than ship a branch that claims a green test it cannot have. So the order is: #1609, #1634, #1629, then #1646, rebased on top.If you would rather the bound land ahead of #1634/#1629 after all, say so and I will mark it ready at once — the patch does not depend on their internals, only on the shape of the child-wait, which all three keep.
bong-water-water-bong commentedon Nov 9, 2026
Implemented elsewhere: #1646 (draft,
Refs #1639) — verified against the analysis here, with two things worth deciding consciouslySearched the open PRs on this file before writing anything, which is what the dedupe rule asks and what stopped a second copy of the same six lines: #1646 is this issue's fix, and it is a faithful port of the deployed copy's bound.
Checked against what this issue says, line by line:
wait "$child"while kill -0 "$child", break at(stop-or-deadline) + survive_byrc=124STOP_STAMP[ -n "${STOP_STAMP:-}" ] && [ -f "$STOP_STAMP" ], so the stamp is optional rather than something main would have to growTwo coordination facts, both from reading the other five PRs on this file
watch() {(#1646at-295,#1634at-290,#1629at-283), and gpu-hold: snapshot the tree and let the watchdog finish (the tree-kill I merged in #1620 does not end the tree) #1634/gpu-hold: end the tree from a snapshot, and do not kill the watchdog mid-kill (fixes #1624) #1629 also restructurekill_tree/watchon the same shift. Whichever lands second rebases — the draft-on-purpose holds the right end of that.GPU_HOLD_KILL_GRACE (10) + GPU_HOLD_SURVIVE_WAIT (60); the deployed copy usesKILL_GRACE30 + 60. Both are fine — a wedged root is either released or nothing saves the queue — but it is a deliberate 20 s difference between the tool the fleet runs and the tool the repo will describe, so it should be a choice rather than an artefact of copying the default.So this issue needs no PR from me; it closes when #1646 leaves draft, on top of #1634 or #1629.
Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).gpu-hold: record the survivor beside the ticket, and tell a stop from…
gpu-hold: poll at 0.2 s, so a fast payload still logs "after 0s" (Refs …
bong-water-water-bong commentedon Nov 9, 2026
The blocker that held #1646 in draft is gone — and the measurement that says it is still needed
Owner assigned this issue (and #1649) to lane
pi-rlk. Doing the sequencing checks first, as the anti-duplicate rule requires:#1646 was drafted waiting on #1634/#1629 to land. Both were closed as superseded —
maincarries #1626 (6301cef), which fixed the same tree-kill hole. So the condition no longer holds, and #1646 can be judged on its own merits.And main still lacks the bound. At
6301cef,scripts/gpu-hold:298iswait "$child", unbounded — re-read at this head, not remembered:The part that has to be said plainly: this case cannot be tested on this box
The owner's rule for these fixes is "a test with the fix". For this one I cannot honestly meet it, and the reason is the defect itself: the bound only fires when the payload's root survives SIGKILL, i.e. a process wedged in uninterruptible sleep. User space cannot create that state, and SIGKILL cannot be blocked, so:
kill -0, which would test my harness rather than the script.That is why #1646 is +54/−3 with no test file, and I would rather say so than add a test that cannot fail — this lane has spent all night catching exactly that shape in other people's patches.
What I recommend, and the decision it needs
mainfails the root-exits shape while gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when its root is already gone (Fixes #1649) #1651 passes it (table on gpu-hold: the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running #1649).I am not un-drafting it myself: it is someone else's patch, it carries a test-less exception to the house rule, and either outcome above is a decision the owner should make with the measurement in front of them. If the owner says (a), I will un-draft it and land the caveat in the body.
Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).tests: drive the half of the #1639 survivor bound that a CPU-only box…
bong-water-water-bong commentedon Nov 9, 2026
Done on this lane's side: #1646 is un-drafted, and the half of it that can be tested now has a test that bites
Following the owner's assignment of this issue to
pi-rlk, and the measurement above thatmain(6301cef) still haswait "$child"unbounded at:298:1. The draft state is gone. #1646 is now
draft=falseat06f1d703, ready for re-review. Its stated blocker — waiting on #1634/#1629 — does not exist: both were closed as superseded, andmaincarries #1626, which fixed the same tree-kill hole.2. The untestable path stays untested, and says so. The bound fires only for a payload root wedged in uninterruptible sleep. User space cannot create that state and SIGKILL cannot interrupt it, so any test I could write for that path would be a grep or a
kill -0stub — a test that cannot fail. I would rather record the gap than fill it with a check that cannot fire.3. The other half — the part a later lane actually depends on — is now covered. A holder that releases with a survivor records it in
$QUEUE/.gpu-hold-survivors; the next holder must warn (naming it) and clear the record, or every later lane inherits a device that may still hold memory with no indication — the 51-GiB-orphan class from 11-08.tests/gpu_hold_survivors_test.shdrives exactly that on a sandboxed queue, registered asctest -R gpu_hold_survivors:1/1 Test #24: gpu_hold_survivors ... Passed 4.87 sec, 100% passed;pid=999999, and the record is emptied;got 0, want 1twice,got 60, want 0), rc 1; restored, 6/6, rc 0.What remains is the owner's call, and it is unchanged
Take the six-line bound as a defensive change whose own path has no test, or retire the repo-vs-deployed difference deliberately. I have not merges anything and will not; both #1646 (this issue) and #1651 (#1649, verified against the owner's
stubborn.shreproducer) are now READY at their heads and mergeable.Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).gpu-hold: release the token when the payload's root survives the KILL…
gpu-hold: record the survivor beside the ticket, and tell a stop from…
gpu-hold: poll at 0.2 s, so a fast payload still logs "after 0s" (Refs …
tests: drive the half of the #1639 survivor bound that a CPU-only box…
gpu-hold: release the token when the payload's root survives the KILL…
gpu-hold: record the survivor beside the ticket, and tell a stop from…
gpu-hold: poll at 0.2 s, so a fast payload still logs "after 0s" (Refs …
tests: drive the half of the #1639 survivor bound that a CPU-only box…
gpu-hold: release the token when the payload's root survives the KILL…
bong-water-water-bong commentedon Nov 9, 2026
Verified fixed on
mainby an independent measurement, and closing on it — if anyone sees a residual I missed, say so and reopen.#1646merged as917d8d1, andscripts/gpu-holdonmainnow carries the bound (survive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT, thebreak, the::error::child … survived KILL: releasing the token anywayline and the durableSURVIVORSrecord). The trigger is a payload root in D-state, which userspace cannot produce, so I fault-injected the two halves of the
condition instead of pretending to reproduce it:
Arm 1 — the loop is the only way out.
while kill -0 "$child"replaced bywhile true(a root that never disappears), run onmain's copy withGPU_HOLD_KILL_GRACE=1 GPU_HOLD_SURVIVE_WAIT=2:Arm 2 — the survivor branch itself. additionally
if kill -0 "$child"replaced byif true, so the after-loop check also sees alive root:
So the promise the issue asks for holds: a wedged root costs one lane's measurement, not the queue, and it says why.
One thing my first attempt got wrong, worth recording because it is a property of the design and not a defect: with
-t 30the sameinjection ran for the full 30 minutes before releasing, because
baseis the hold's own deadline — the bound ismax_hold_end + KILL_GRACE + SURVIVE_WAIT, notKILL_GRACE + SURVIVE_WAITfrom the moment the root was signalled. That is the rightbehaviour (a hold is entitled to its ticket, and the bound stops it exceeding that ticket by more than ~70 s), but it is not what "a
bound" invites a reader to assume, so it is now written down here rather than discovered by the next person with a stopwatch.
What I am deliberately not touching: the deployed
~/.local/bin/gpu-holdon this box is dated 2026-11-06, so it has the bound butpredates
#1626and#1646— the same drift#1701raises fornpu-hold, and the same answer:docs/ci-runners.mdalready says theinstaller puts the repository's copy in place, so installing it is that lane's step, not an issue that needs to stay open here.
bong-water-water-bong commentedon Nov 9, 2026
Fixed and proven on
main— closing with the evidence, and saying exactly what is not covered#1646 merged at 18:35Z;
mainis88fafa53. Verified there rather than in the PR, because a merge is not a measurement:mainsurvive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT)wait "$child"is gone$QUEUE/.gpu-hold-survivors)tests/gpu_hold_survivors_test.shAnd the owner's reproducer plus this issue's shape through the harness I posted earlier, against the merged script:
The token is released within 0 s in both arms — including the arm where a survivor exists. That is this issue: a payload root that survives KILL can no longer hold the ticket for ever.
What is deliberately outside this close
The
survivor: 1006275on the second line is #1649's defect, not this one: the bound releases the token, it does not kill a leaked child, and the group-kill fix for that is #1651, still open. So closing this issue does not claim that anything ends the child — only that nothing can hold the queue behind it, and that the next holder is told a device may still be dirty.Also worth stating for the record: the bound's own path (a root wedged in uninterruptible sleep) still has no test, because user space cannot construct that state. What is covered is the survivor-record half — proven to bite (deleting the warning block fails three of six checks) — and the bound is a defensive six lines whose absence is now impossible to hide, since the record it writes is checked on every subsequent hold.
Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).npu-hold: put the NPU hold in the tree, on ONEBIT_HOLD_DIR, and on an…