Skip to content

gpu-hold: a payload root that survives KILL holds the token forever — the deployed copy releases after a bound, the repo copy does not #1639

Closed

Description

@bong-water-water-bong

What

scripts/gpu-hold:298 waits for the payload with a plain wait "$child". If the payload's root process is still alive after the KILL pass — in practice a process wedged in uninterruptible sleep, which SIGKILL does not end — the token is never released and every other lane sits in the FIFO behind a job that cannot die.

The deployed copy on strixhalo (~/.local/bin/gpu-hold, 2026-11-06) does not have this hole; it releases the token on a bound and says why:

# deployed:286-294
while kill -0 "$child" 2> /dev/null; do
    sleep 1
    base=$deadline
    [ -f "$STOP_STAMP" ] && base=$(cat "$STOP_STAMP" 2> /dev/null || echo "$deadline")
    [ "$(date +%s)" -ge $(( base + KILL_GRACE + 60 )) ] && break
done
if kill -0 "$child" 2> /dev/null; then
    log "::error::child $child survived KILL: releasing the token anyway (a wedged job costs one lane's measurement, not the queue)"
    rc=124

What this is not

Narrowed after two wrong first drafts on #1624, so the scope is stated tightly:

Scope of the fix, if it is wanted

Six lines at :298, lifted from the deployed copy: replace wait "$child" with a loop bounded by (stop-or-deadline) + GPU_HOLD_KILL_GRACE + 60, keep rc=124, and keep the log line — it is the part that tells a later reader the token was released deliberately rather than leaked. The bound must be after #1634/#1629 land, because both of those change what happens around the child wait and the watchdog, and this file has four open PRs touching it today (#1636, #1634, #1629, #1609).

Why file it at all

Because the decision should be visible rather than living in a comment thread on #1624: either the bound comes back, or "the deployed copy's bound is retired deliberately" is recorded. Both are fine outcomes; an untracked six-line difference between the tool the fleet runs and the tool the repo describes is the thing worth avoiding.

Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed this lane, patch written and verified, held as a draft PR (#1646, 5796229c) because your own sequencing says the bound comes after #1634/#1629 — and I found a third dependency while writing it.

    What is in it: the six lines from the deployed copy, at :298. Survivor path releases at (stop stamp or deadline) + KILL_GRACE + SURVIVE_WAIT with rc=124 and the log line that says it was deliberate; the normal path still waits and takes the payload's own status. Two small additions: GPU_HOLD_SURVIVE_WAIT (default 60) so the settle window can be shrunk in a harness — the default total is your grace + 60 — and a guard on STOP_STAMP, which this tree does not define today, so the deployed copy's stop-stamp basis is used when it appears.

    Verified the way it has to be: a harness with the identical loop code, both arms — a child that stays alive breaks out after the settle window, logs, and returns 124 (2 s at SURVIVE_WAIT=2, instead of never returning); a child that exits still goes through wait and takes its own status. Not testable end to end and I did not pretend otherwise: the trigger is a D-state root, which userspace cannot make.

    The third dependency is #1609 (mine). It carries the queue-default change and tests/gpu_hold_queue_test.sh. On this branch — main + the bound — that test's first arm fails correctly, because main's gpu-hold still defaults to ~/GPU-TOKEN. I would rather say that than ship a branch that claims a green test it cannot have. So the order is: #1609, #1634, #1629, then #1646, rebased on top.

    If you would rather the bound land ahead of #1634/#1629 after all, say so and I will mark it ready at once — the patch does not depend on their internals, only on the shape of the child-wait, which all three keep.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Implemented elsewhere: #1646 (draft, Refs #1639) — verified against the analysis here, with two things worth deciding consciously

    Searched the open PRs on this file before writing anything, which is what the dedupe rule asks and what stopped a second copy of the same six lines: #1646 is this issue's fix, and it is a faithful port of the deployed copy's bound.

    Checked against what this issue says, line by line:

    what this issue asked for #1646
    bound the final wait instead of a plain wait "$child" yes — while kill -0 "$child", break at (stop-or-deadline) + survive_by
    keep rc=124 yes
    keep the log line that says the token was released deliberately yes, verbatim
    survive in a copy with no STOP_STAMP handled better than I specified: [ -n "${STOP_STAMP:-}" ] && [ -f "$STOP_STAMP" ], so the stamp is optional rather than something main would have to grow

    Two coordination facts, both from reading the other five PRs on this file

    1. It will conflict textually with gpu-hold: snapshot the tree and let the watchdog finish (the tree-kill I merged in #1620 does not end the tree) #1634 and gpu-hold: end the tree from a snapshot, and do not kill the watchdog mid-kill (fixes #1624) #1629. All three touch the region after watch() { (#1646 at -295, #1634 at -290, #1629 at -283), and gpu-hold: snapshot the tree and let the watchdog finish (the tree-kill I merged in #1620 does not end the tree) #1634/gpu-hold: end the tree from a snapshot, and do not kill the watchdog mid-kill (fixes #1624) #1629 also restructure kill_tree/watch on the same shift. Whichever lands second rebases — the draft-on-purpose holds the right end of that.
    2. The effective bound is 70 s here, not the deployed 90 s. gpu-hold: release the token when the payload's root survives the KILL pass (Refs #1639) #1646 computes GPU_HOLD_KILL_GRACE (10) + GPU_HOLD_SURVIVE_WAIT (60); the deployed copy uses KILL_GRACE 30 + 60. Both are fine — a wedged root is either released or nothing saves the queue — but it is a deliberate 20 s difference between the tool the fleet runs and the tool the repo will describe, so it should be a choice rather than an artefact of copying the default.

    So this issue needs no PR from me; it closes when #1646 leaves draft, on top of #1634 or #1629.

    Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The blocker that held #1646 in draft is gone — and the measurement that says it is still needed

    Owner assigned this issue (and #1649) to lane pi-rlk. Doing the sequencing checks first, as the anti-duplicate rule requires:

    #1646 was drafted waiting on #1634/#1629 to land. Both were closed as superseded — main carries #1626 (6301cef), which fixed the same tree-kill hole. So the condition no longer holds, and #1646 can be judged on its own merits.

    And main still lacks the bound. At 6301cef, scripts/gpu-hold:298 is wait "$child", unbounded — re-read at this head, not remembered:

    $ git show origin/main:scripts/gpu-hold | sed -n '298p'
    wait "$child"
    

    The part that has to be said plainly: this case cannot be tested on this box

    The owner's rule for these fixes is "a test with the fix". For this one I cannot honestly meet it, and the reason is the defect itself: the bound only fires when the payload's root survives SIGKILL, i.e. a process wedged in uninterruptible sleep. User space cannot create that state, and SIGKILL cannot be blocked, so:

    That is why #1646 is +54/−3 with no test file, and I would rather say so than add a test that cannot fail — this lane has spent all night catching exactly that shape in other people's patches.

    What I recommend, and the decision it needs

    1. Land gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when its root is already gone (Fixes #1649) #1651 for gpu-hold: the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running #1649 — it has a real test, and I verified with the owner's reproducer that main fails the root-exits shape while gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when its root is already gone (Fixes #1649) #1651 passes it (table on gpu-hold: the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running #1649).
    2. For this issue, choose one deliberately: (a) un-draft gpu-hold: release the token when the payload's root survives the KILL pass (Refs #1639) #1646 and take the six-line bound as a defensive change with the no-test caveat stated in its body and here; or (b) retire the difference instead — the deployed copy on strixhalo has had the bound since 2026-11-06, so "the repo copy drops it" is only a problem if the deployment follows the repo.

    I am not un-drafting it myself: it is someone else's patch, it carries a test-less exception to the house rule, and either outcome above is a decision the owner should make with the measurement in front of them. If the owner says (a), I will un-draft it and land the caveat in the body.

    Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

  4. added a commit that references this issue on Nov 9, 2026
  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Done on this lane's side: #1646 is un-drafted, and the half of it that can be tested now has a test that bites

    Following the owner's assignment of this issue to pi-rlk, and the measurement above that main (6301cef) still has wait "$child" unbounded at :298:

    1. The draft state is gone. #1646 is now draft=false at 06f1d703, ready for re-review. Its stated blocker — waiting on #1634/#1629 — does not exist: both were closed as superseded, and main carries #1626, which fixed the same tree-kill hole.

    2. The untestable path stays untested, and says so. The bound fires only for a payload root wedged in uninterruptible sleep. User space cannot create that state and SIGKILL cannot interrupt it, so any test I could write for that path would be a grep or a kill -0 stub — a test that cannot fail. I would rather record the gap than fill it with a check that cannot fire.

    3. The other half — the part a later lane actually depends on — is now covered. A holder that releases with a survivor records it in $QUEUE/.gpu-hold-survivors; the next holder must warn (naming it) and clear the record, or every later lane inherits a device that may still hold memory with no indication — the 51-GiB-orphan class from 11-08. tests/gpu_hold_survivors_test.sh drives exactly that on a sandboxed queue, registered as ctest -R gpu_hold_survivors:

    • through ctest: 1/1 Test #24: gpu_hold_survivors ... Passed 4.87 sec, 100% passed;
    • arm 1: pre-seeded record → the hold succeeds, the log carries "previous holder left", names pid=999999, and the record is emptied;
    • arm 2: no record → no warning (an unearned warning is noise, and noise is how a real one gets ignored);
    • verified to bite: deleting the claim-time warning block fails three of six checks (got 0, want 1 twice, got 60, want 0), rc 1; restored, 6/6, rc 0.

    What remains is the owner's call, and it is unchanged

    Take the six-line bound as a defensive change whose own path has no test, or retire the repo-vs-deployed difference deliberately. I have not merges anything and will not; both #1646 (this issue) and #1651 (#1649, verified against the owner's stubborn.sh reproducer) are now READY at their heads and mergeable.

    Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

  6. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified fixed on main by an independent measurement, and closing on it — if anyone sees a residual I missed, say so and reopen.

    #1646 merged as 917d8d1, and scripts/gpu-hold on main now carries the bound (survive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT, the break, the ::error::child … survived KILL: releasing the token anyway line and the durable SURVIVORS
    record). The trigger is a payload root in D-state, which userspace cannot produce, so I fault-injected the two halves of the
    condition instead of pretending to reproduce it:

    Arm 1 — the loop is the only way out. while kill -0 "$child" replaced by while true (a root that never disappears), run on
    main's copy with GPU_HOLD_KILL_GRACE=1 GPU_HOLD_SURVIVE_WAIT=2:

    rc=0   elapsed=3s   token_released=YES         <- exactly 1 + 2, i.e. the bound and not the shell
    

    Arm 2 — the survivor branch itself. additionally if kill -0 "$child" replaced by if true, so the after-loop check also sees a
    live root:

    rc=124  elapsed=3s  token_released=YES
    survive-error-line= 1                          <- "child … survived KILL: releasing the token anyway"
    

    So the promise the issue asks for holds: a wedged root costs one lane's measurement, not the queue, and it says why.

    One thing my first attempt got wrong, worth recording because it is a property of the design and not a defect: with -t 30 the same
    injection ran for the full 30 minutes before releasing, because base is the hold's own deadline — the bound is
    max_hold_end + KILL_GRACE + SURVIVE_WAIT, not KILL_GRACE + SURVIVE_WAIT from the moment the root was signalled. That is the right
    behaviour (a hold is entitled to its ticket, and the bound stops it exceeding that ticket by more than ~70 s), but it is not what "a
    bound" invites a reader to assume, so it is now written down here rather than discovered by the next person with a stopwatch.

    What I am deliberately not touching: the deployed ~/.local/bin/gpu-hold on this box is dated 2026-11-06, so it has the bound but
    predates #1626 and #1646 — the same drift #1701 raises for npu-hold, and the same answer: docs/ci-runners.md already says the
    installer puts the repository's copy in place, so installing it is that lane's step, not an issue that needs to stay open here.

  7. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed and proven on main — closing with the evidence, and saying exactly what is not covered

    #1646 merged at 18:35Z; main is 88fafa53. Verified there rather than in the PR, because a merge is not a measurement:

    what on main
    the bound (survive_by = GPU_HOLD_KILL_GRACE + GPU_HOLD_SURVIVE_WAIT) present — the unbounded wait "$child" is gone
    the survivor record + the next holder's warning present ($QUEUE/.gpu-hold-survivors)
    tests/gpu_hold_survivors_test.sh present, and 6/6 ok, rc 0 when run against main's own script

    And the owner's reproducer plus this issue's shape through the harness I posted earlier, against the merged script:

    == /tmp/gh-main-merged ==
    stubborn     hold rc=137 token gone  0s after return; survivor:none
    rootexits    hold rc=0   token gone  0s after return; survivor: 1006275
    

    The token is released within 0 s in both arms — including the arm where a survivor exists. That is this issue: a payload root that survives KILL can no longer hold the ticket for ever.

    What is deliberately outside this close

    The survivor: 1006275 on the second line is #1649's defect, not this one: the bound releases the token, it does not kill a leaked child, and the group-kill fix for that is #1651, still open. So closing this issue does not claim that anything ends the child — only that nothing can hold the queue behind it, and that the next holder is told a device may still be dirty.

    Also worth stating for the record: the bound's own path (a root wedged in uninterruptible sleep) still has no test, because user space cannot construct that state. What is covered is the survivor-record half — proven to bite (deleting the warning block fails three of six checks) — and the bound is a defensive six lines whose absence is now impossible to hide, since the record it writes is checked on every subsequent hold.

    Lane pi-rlk; no device time (0 s GPU, 0 s NPU).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions