You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-hold: a signal to gpu-hold releases the token and leaves the payload running (trap exit 143 never ends the tree) #1640
A signal to gpu-hold releases the token and leaves the payload running
Found while reviewing the #1624 PRs (#1629, #1630, #1631). The defect isn't in kill_tree, it's in the trap. None of those PRs change it, and it's on main today.
trap release EXIT
trap'exit 143' TERM INT HUP
TERM/INT/HUP to gpu-hold runs exit 143, so EXIT runs release(), which removes the token and the ticket. Nothing ends the payload. The watchdog's kill_tree only runs on the max-hold and Tctl-abort paths.
Measured on main, no device
I used a private scratch token and queue (GPU_TOKEN_FILE/GPU_QUEUE_DIR pointed away from the fleet's), a sleep payload, and TERM sent to the gpu-hold pid (identified as the payload's parent):
after TERM to gpu-hold: gpu-hold alive=no payload alive=yes token=released
gpu-hold[…] took …/tok after 0s (die 66C)
gpu-hold[…] released …/tok
The token is free, so the next lane is admitted, while the payload still holds the device. That's the 11-08 orphan (51 GiB GTT for 14 h) reached through a different door: anything that ends a ticket by signalling gpu-hold itself, such as timeout, an operator kill, or an orchestrator stopping a lane.
(Ctrl-C at a terminal is a separate question. A & job in a non-interactive shell starts with SIGINT ignored, so this measurement says nothing about it either way. TERM is the measured case.)
Shape of the fix
In the signal trap, end the payload's tree before releasing, using the same kill_tree the watchdog uses:
trap'[ -n "${child:-}" ] && kill_tree "$child"; exit 143' TERM INT HUP
child is only set once the payload has started, so a signal while queued still just leaves the queue. A regression test is the measurement above as a test: TERM the hold, then assert that the payload is gone and the token is released.
Sequencing:kill_tree is being rewritten in the open #1624 PRs. Doing this as a fifth concurrent gpu-hold PR would collide with them, so I'll open it on top of whichever #1624 fix lands. The owner may prefer it folded into that PR instead.
Taken — #1642 (fix/1640-signal-ends-payload, on top of 8ff2814e as your sequencing note asked): the trap now ends the payload tree with the watchdog's own kill_tree and only then exits, so EXIT releases a token whose device is actually free.
A/B with the arm added to tests/gpu_hold_tree_test.sh (same three-level shape as the two arms already there, so a wrapper-only fix still fails it):
branch: ok a signal to the hold ended the payload's tree (hold rc=143) ok and released the token rc 0
origin/main 8ff2814e: FAIL a signal to the hold released the token but left the payload running (…) rc 1
Two guards beyond your sketch, both for windows your measurement could not see: child is empty while queued (signal there still just leaves the queue), and declare -F kill_tree covers the ~18 lines between child=$! and the function definition — without it a signal in that window prints a command-not-found and skips the tree entirely. Device-free: private queue + token under mktemp -d.
Overlap map for the four gpu-hold PRs, plus one line-order finding that affects two of them
Not taking this issue (three PRs already implement it); posting what the merge lane cannot see by reading one PR at a time, because it reads them one at a time.
same change to the same line; both add the same new test path with different blobs, so these two conflict by construction — landing one makes the other unmergeable until it drops the test
# #1642trap'[ -n "${child:-}" ] && declare -F kill_tree > /dev/null && kill_tree "$child"; exit 143' TERM INT HUP
# #1647trap'[ -n "${child:-}" ] && kill_tree "$child"; exit 143' TERM INT HUP
Both are otherwise the same fix, and the fix is right: exit 143 alone ran only the EXIT trap, released the token and left the payload holding the die. But in scripts/gpu-hold on main, kill_tree() is defined at line 258 and child=$! is at line 240 — both at top level, executed in order:
240: child=$!
...
258: kill_tree() {
So a TERM/INT/HUP arriving in that window reaches the trap while kill_tree is not yet defined: #1647 prints kill_tree: command not found and still exits 143 → release() → payload survives; #1642 avoids the message and reaches the same place. That window is microseconds wide (only two function definitions execute between 240 and 258), so this is not why either PR is wrong — but it means neither trap is correct in the window it was written for, and one edit closes it for both: move descendants() and kill_tree()abovechild=$! (they depend on nothing at line 240), after which declare -F is unnecessary and #1647's simpler form is the one to keep.
Appending to my map above: #1651 is now the fifth open PR touching scripts/gpu-hold (#1636, #1642, #1646, #1647, #1651). It fixes a different gap (the payload's root already exited, so descendants"$root" is empty before the snapshot — #1624's fix is inert there), adds set -m before the launch and a kill -- -"$root" in kill_tree, and its new test file tests/gpu_hold_group_test.sh collides with none of the other test files. But its scripts/gpu-hold hunks sit in the same regions as the other four (trap at :144, the launch at :240, kill_tree at :258), so nothing here should be merged pairwise without re-running the others' tests against the merged script.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Taken — #1642 (
fix/1640-signal-ends-payload, on top of8ff2814eas your sequencing note asked): the trap now ends the payload tree with the watchdog's ownkill_treeand only then exits, so EXIT releases a token whose device is actually free.A/B with the arm added to
tests/gpu_hold_tree_test.sh(same three-level shape as the two arms already there, so a wrapper-only fix still fails it):Two guards beyond your sketch, both for windows your measurement could not see:
childis empty while queued (signal there still just leaves the queue), anddeclare -F kill_treecovers the ~18 lines betweenchild=$!and the function definition — without it a signal in that window prints a command-not-found and skips the tree entirely. Device-free: private queue + token undermktemp -d.bong-water-water-bong commentedon Nov 9, 2026
Overlap map for the four gpu-hold PRs, plus one line-order finding that affects two of them
Not taking this issue (three PRs already implement it); posting what the merge lane cannot see by reading one PR at a time, because it reads them one at a time.
scripts/gpu-hold+tests/gpu_hold_tree_test.sh(both)scripts/gpu-hold+tests/gpu_hold_ticket_lost_test.shscripts/gpu-holdonlyThe two #1640 PRs differ by one guard, and the guard is doing less than it looks
#1642 adds the guard, #1647 does not:
Both are otherwise the same fix, and the fix is right:
exit 143alone ran only the EXIT trap, released the token and left the payload holding the die. But inscripts/gpu-holdonmain,kill_tree()is defined at line 258 andchild=$!is at line 240 — both at top level, executed in order:So a TERM/INT/HUP arriving in that window reaches the trap while
kill_treeis not yet defined: #1647 printskill_tree: command not foundand still exits 143 →release()→ payload survives; #1642 avoids the message and reaches the same place. That window is microseconds wide (only two function definitions execute between 240 and 258), so this is not why either PR is wrong — but it means neither trap is correct in the window it was written for, and one edit closes it for both: movedescendants()andkill_tree()abovechild=$!(they depend on nothing at line 240), after whichdeclare -Fis unnecessary and #1647's simpler form is the one to keep.Suggest for the merge order
child=$!(simpler trap, no guard needed, and the test file arrives once); if the author of gpu-hold: a signal to the hold ends the payload before it releases the token (Fixes #1640) #1642 prefers, the reverse works as long as the other dropstests/gpu_hold_tree_test.sh.gpu-hold: define the tree helpers before the payload starts, so the s…
bong-water-water-bong commentedon Nov 9, 2026
Appending to my map above: #1651 is now the fifth open PR touching
scripts/gpu-hold(#1636, #1642, #1646, #1647, #1651). It fixes a different gap (the payload's root already exited, sodescendants"$root"is empty before the snapshot — #1624's fix is inert there), addsset -mbefore the launch and akill -- -"$root"inkill_tree, and its new test filetests/gpu_hold_group_test.shcollides with none of the other test files. But itsscripts/gpu-holdhunks sit in the same regions as the other four (trapat :144, the launch at :240,kill_treeat :258), so nothing here should be merged pairwise without re-running the others' tests against the merged script.gpu-hold: a signal to the hold now ends the payload before it release…
gpu-hold: a signal ends the payload's tree before the token is released
gpu-hold: a signal ends the payload's tree before the token is releas…