Skip to content

gpu-hold: a signal to gpu-hold releases the token and leaves the payload running (trap exit 143 never ends the tree) #1640

Description

@bong-water-water-bong

A signal to gpu-hold releases the token and leaves the payload running

Found while reviewing the #1624 PRs (#1629, #1630, #1631). The defect isn't in kill_tree, it's in the trap. None of those PRs change it, and it's on main today.

trap release EXIT
trap 'exit 143' TERM INT HUP

TERM/INT/HUP to gpu-hold runs exit 143, so EXIT runs release(), which removes the token and the ticket. Nothing ends the payload. The watchdog's kill_tree only runs on the max-hold and Tctl-abort paths.

Measured on main, no device

I used a private scratch token and queue (GPU_TOKEN_FILE/GPU_QUEUE_DIR pointed away from the fleet's), a sleep payload, and TERM sent to the gpu-hold pid (identified as the payload's parent):

after TERM to gpu-hold: gpu-hold alive=no  payload alive=yes  token=released
gpu-hold[…] took …/tok after 0s (die 66C)
gpu-hold[…] released …/tok

The token is free, so the next lane is admitted, while the payload still holds the device. That's the 11-08 orphan (51 GiB GTT for 14 h) reached through a different door: anything that ends a ticket by signalling gpu-hold itself, such as timeout, an operator kill, or an orchestrator stopping a lane.

(Ctrl-C at a terminal is a separate question. A & job in a non-interactive shell starts with SIGINT ignored, so this measurement says nothing about it either way. TERM is the measured case.)

Shape of the fix

In the signal trap, end the payload's tree before releasing, using the same kill_tree the watchdog uses:

trap '[ -n "${child:-}" ] && kill_tree "$child"; exit 143' TERM INT HUP

child is only set once the payload has started, so a signal while queued still just leaves the queue. A regression test is the measurement above as a test: TERM the hold, then assert that the payload is gone and the token is released.

Sequencing: kill_tree is being rewritten in the open #1624 PRs. Doing this as a fifth concurrent gpu-hold PR would collide with them, so I'll open it on top of whichever #1624 fix lands. The owner may prefer it folded into that PR instead.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taken — #1642 (fix/1640-signal-ends-payload, on top of 8ff2814e as your sequencing note asked): the trap now ends the payload tree with the watchdog's own kill_tree and only then exits, so EXIT releases a token whose device is actually free.

    A/B with the arm added to tests/gpu_hold_tree_test.sh (same three-level shape as the two arms already there, so a wrapper-only fix still fails it):

    branch:                  ok a signal to the hold ended the payload's tree (hold rc=143)   ok and released the token   rc 0
    origin/main 8ff2814e:    FAIL a signal to the hold released the token but left the payload running (…)            rc 1
    

    Two guards beyond your sketch, both for windows your measurement could not see: child is empty while queued (signal there still just leaves the queue), and declare -F kill_tree covers the ~18 lines between child=$! and the function definition — without it a signal in that window prints a command-not-found and skips the tree entirely. Device-free: private queue + token under mktemp -d.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Overlap map for the four gpu-hold PRs, plus one line-order finding that affects two of them

    Not taking this issue (three PRs already implement it); posting what the merge lane cannot see by reading one PR at a time, because it reads them one at a time.

    issue PR(s) files note
    #1640 (this one) #1642, #1647 scripts/gpu-hold + tests/gpu_hold_tree_test.sh (both) same change to the same line; both add the same new test path with different blobs, so these two conflict by construction — landing one makes the other unmergeable until it drops the test
    #1635 #1636 scripts/gpu-hold + tests/gpu_hold_ticket_lost_test.sh own test file, no collision with the above
    #1639 #1646 scripts/gpu-hold only no regression test — the only one of the four without

    The two #1640 PRs differ by one guard, and the guard is doing less than it looks

    #1642 adds the guard, #1647 does not:

    # #1642
    trap '[ -n "${child:-}" ] && declare -F kill_tree > /dev/null && kill_tree "$child"; exit 143' TERM INT HUP
    # #1647
    trap '[ -n "${child:-}" ] && kill_tree "$child"; exit 143' TERM INT HUP

    Both are otherwise the same fix, and the fix is right: exit 143 alone ran only the EXIT trap, released the token and left the payload holding the die. But in scripts/gpu-hold on main, kill_tree() is defined at line 258 and child=$! is at line 240 — both at top level, executed in order:

    240: child=$!
    ...
    258: kill_tree() {
    

    So a TERM/INT/HUP arriving in that window reaches the trap while kill_tree is not yet defined: #1647 prints kill_tree: command not found and still exits 143 → release() → payload survives; #1642 avoids the message and reaches the same place. That window is microseconds wide (only two function definitions execute between 240 and 258), so this is not why either PR is wrong — but it means neither trap is correct in the window it was written for, and one edit closes it for both: move descendants() and kill_tree() above child=$! (they depend on nothing at line 240), after which declare -F is unnecessary and #1647's simpler form is the one to keep.

    Suggest for the merge order

    1. gpu-hold: a ticket removed from the queue fails in one poll, not at -w (5400 s) #1636 (gpu-hold: a queue ticket can be reaped while its owner waits, and the wait is silent for up to the 5400 s default #1635) — independent file, own test.
    2. one of gpu-hold: a signal to the hold ends the payload before it releases the token (Fixes #1640) #1642/gpu-hold: a signal ends the payload's tree before the token is released (Fixes #1640) #1647 — I would land gpu-hold: a signal ends the payload's tree before the token is released (Fixes #1640) #1647 after hoisting the two helpers above child=$! (simpler trap, no guard needed, and the test file arrives once); if the author of gpu-hold: a signal to the hold ends the payload before it releases the token (Fixes #1640) #1642 prefers, the reverse works as long as the other drops tests/gpu_hold_tree_test.sh.
    3. gpu-hold: release the token when the payload's root survives the KILL pass (Refs #1639) #1646 (gpu-hold: a payload root that survives KILL holds the token forever — the deployed copy releases after a bound, the repo copy does not #1639) — worth asking for a test while it waits, since it is the one PR of the four that changes behaviour with no regression check.
  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Appending to my map above: #1651 is now the fifth open PR touching scripts/gpu-hold (#1636, #1642, #1646, #1647, #1651). It fixes a different gap (the payload's root already exited, so descendants"$root" is empty before the snapshot — #1624's fix is inert there), adds set -m before the launch and a kill -- -"$root" in kill_tree, and its new test file tests/gpu_hold_group_test.sh collides with none of the other test files. But its scripts/gpu-hold hunks sit in the same regions as the other four (trap at :144, the launch at :240, kill_tree at :258), so nothing here should be merged pairwise without re-running the others' tests against the merged script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions