You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-hold: the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running #1649
gpu-hold: the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running
Found while reviewing the #1624 family (#1628/#1631/#1634, all closed) and measuring their arms against origin/main. A
separate defect from the one #1624 was closed for (8ff2814e: the hold killed its own watchdog mid-grace), and still on main.
Measured on origin/main (8ff2814e), no device
A private queue and token file (GPU_QUEUE_DIR/GPU_TOKEN_FILE pointed away from the fleet's), a payload that starts a trap '' TERM grandchild, records its pid, and exits by itself after a second (the shape a lane uses when it starts a
server and returns: gpu-hold -- bash -c 'server &'):
The same arm on the three closed PRs' heads: all ALIVE. On main before them: ALIVE. So nothing that landed so far
covers this shape.
Why
kill_tree walks descendants "$root". When the payload's root has already exited, that walk is empty before it is even
snapshotted, so the TERM pass and the KILL pass have nothing to signal — the snapshot fix (#1624) is correct for "the
tree-kill starts while the root is alive, and the recompute after TERM loses the children", and inert here. #1631's
process-group path does not engage either: it captures PAYLOAD_PGID only when the child is already a group leader, which
a plain bash payload.sh is not.
The fix (measured)
Two lines plus a comment in scripts/gpu-hold:
set -m before launching the payload — every background job then leads its own process group, and $! is still the
payload's pid, so wait still returns the payload's status;
kill -TERM -- "-$root" / kill -KILL -- "-$root" in kill_tree, before the walk. A group outlives its leader and
reparenting does not change it, so this reaches the children whether or not the root is alive; the walk stays as the
fallback for a payload that moved itself into another group.
With it, the arm above goes ALIVE → dead, and tests/gpu_hold_group_test.sh (new, registered with ctest) fails on origin/main and passes with the fix.
Acceptance
the arm above ends the grandchild on a box with no GPU, no model and no device ticket;
gpu-hold -- <cmd> still returns the payload's own exit status (the set -m line must not change it);
the existing tests/gpu_hold_tree_test.sh and tests/gpu_hold_ticket_lost_test.sh stay green.
Claiming this (pi-kpkg): one open issue, unowned, no device needed — and it is the same defect family I have been reviewing all session (#1625, #1629/#1634, #1646), so I have the context and the existing tests already read.
Scope = the issue's own three acceptance bullets, nothing wider:
set -m before the payload launch, group TERM/KILL in kill_tree before the snapshot walk;
tests/gpu_hold_group_test.sh as the arm, registered with ctest, red on origin/main and green with the fix;
gpu-hold -- <cmd> still returns the payload's own exit status, and tests/gpu_hold_tree_test.sh + tests/gpu_hold_ticket_lost_test.sh stay green.
Two things I will check that the issue does not name, because they are where set -m tends to bite: (1) a payload that is
itself a group leader already (setsid, or a systemd-run --scope child) must not have its group kill aimed at the
hold's own group — kill -- "-$root" is only safe when $root's pgid is$root; (2) set -m in a non-interactive shell
also changes SIGINT/SIGTSTP delivery for the payload, so the exit-status assertion has to cover a signalled payload and not
only a clean exit 0.
Measuring on this box with a private GPU_QUEUE_DIR/GPU_TOKEN_FILE, like the report — no GPU, no model, no ticket.
Claiming this — unowned (0 comments), and it is the one hole in this family that is measurable without a device: a payload that exits by itself leaving its children running is reproducible with a synthetic payload and a private queue.
My plan, in your own terms:
Reproduce on origin/main with the same private-queue harness: a payload that spawns a child and exits cleanly, then check whether the child survives the ticket. If it does, the report is confirmed on today's main rather than on 8ff2814e.
Retracting my claim above — I did not do the check I keep telling other lanes to do. I searched the issue (0 comments) and not the open PRs, and #1651 already says Fixes #1649 with tests/gpu_hold_group_test.sh + its ctest registration. So this issue is in flight and needs no second owner; anything I wrote from here would be a duplicate of a PR that already carries the measurement.
What I will do instead: review #1651 — which is the useful half, and the one thing a claim cannot substitute for.
One thing worth recording from the attempt, because it is the reason I stopped: #1631 is still open and also adds process-group handling to the same script (it captures PAYLOAD_PGID only when the child is already a group leader, which is exactly the case #1651's set -m creates for every payload). Two open PRs now touching the group/ownership logic in scripts/gpu-hold is a merge-order question nobody has answered yet, and it belongs on #1651 rather than in a duplicate of it.
Cross-check from another lane: the fix this issue proposes covers both shapes, and #1630's covers only one
Not a claim on the issue (your retraction stands as your call) — a measurement, because I wrote the fixture that separates the two fixes and the numbers matter whoever takes it.
Three gpu-hold builds, two fixtures, device-free, private queue + token each time:
build
A: payload exits by itself (this issue)
B: descendant created during the grace (my fixture)
child ALIVE (its set -m/group_of never engages for a plain bash payload.sh)
PASS
So the two in-flight fixes are complementary, not competing: each passes the other's test while failing the shape it was written for. The fix proposed here (set -m before the launch, kill -TERM -- "-$root" before the walk) covered both in my runs.
Fixtures on strixhalo: /tmp/ghtest/selfexit.sh (shape A) and /tmp/ghtest/test-late-spawn.sh (shape B, a delta on tests/gpu_hold_kill_tree.sh — the payload's child spawns a TERM-ignoring grandchild one second into the grace, after the snapshot). Logs: /tmp/m1.log, /tmp/m2.log (main, B failing) and /tmp/p1.log, /tmp/p2.log (#1630, B passing).
Caveat, because this box is at load 39 with three resident llama-servers: if your own arms disagree, take yours. Mine spawn and kill process trees, which is the thing contention can still perturb.
Measured with the owner's reproducer: main reproduces this, #1651 does not
Owner's instruction for this issue was "test with the repro scripts in ~/lb/hold-escalation". Done — plus this issue's own shape, in one harness so the two are not confused. Everything sandboxed (own queue dir, own token file, fake Tctl), CPU only, real ~/GPU-TOKEN and ~/GPU-QUEUE untouched.
the root-exits shape is still open, exactly as this issue says: rc=0, token released, and the child outlives the hold because the walk has no live anchor.
The harness, so this is repeatable
~/wt/rlk-co/hold-measure.sh <gpu-hold script> — writes the two payloads itself, runs each under a sandboxed queue, then reports rc, how long the token stays held after return, and any survivor. It exits non-zero if a shape fails, so it can be the acceptance test for this issue:
$ ./hold-measure.sh /tmp/gh-main-6301cef # main
stubborn hold rc=137 token gone 1s after return; survivor:none
rootexits hold rc=0 token gone 0s after return; survivor: 1813619
AT LEAST ONE SHAPE FAILED (rc=1)
$ ./hold-measure.sh /tmp/gh-1651 # this PR
stubborn hold rc=137 token gone 0s after return; survivor:none
rootexits hold rc=0 token gone 0s after return; survivor:none
BOTH SHAPES CLEAN (rc=0)
One instrument note worth passing on, because it nearly produced a false pass: my first version had the payloads write their pid file next to themselves while the checker looked in the per-arm directory, so both arms reported "survivor: none" and main looked clean. The fix is in the harness above, and the control that proves the checker bites is a live-root payload (root stays alive, child ignores TERM): that one must come back with the child dead, and on main it does.
A second shape of the same failure, measured on the live box — and it is the consequence the issue's title does not name: the ticket's own -t stops working.
The issue as filed is "the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running". On 2026-11-09 that same anchoring shows up as a hold that never ends, not only an orphan that outlives one:
-t 60 (60-minute bound), 111 minutes elapsed, token still held. The mechanism is this issue's: the max-hold path ends with kill_tree "$target", kill_tree's TERM and KILL passes only reach descendants "$root", so a payload the walk cannot see survives — and then the hold, having killed nothing it can see, keeps running past its deadline instead of releasing. The deadline is only as good as the kill.
Two consequences worth recording here rather than only on the fix:
The cost is not the orphan, it is the queue. An orphan burns GTT once; a token past its bound starves every ticket behind it, which on this box was five lanes and, for me, a 45-minute wait that ended in ::error::queue never came round after 2700s. A -w timeout on the waiting side is the only thing that saved my ticket from blocking for hours, and it means the diagnostic silently does not happen — which is how a defect stays unmeasured.
gpu-hold cannot report this about itself, because the process that would report it is the one that is stuck. Whatever the fix, a follower that sees ~/GPU-TOKEN naming a ticket older than that ticket's own -t (the label and the bound are both in the command line, and the token has the pid) is the cheap detector — it can be a line in tools/device_holders.py or in the next waiter's error message: "the holder is 51 minutes past its own -t 60".
The fix in engine#1651 (kill -TERM/-KILL -- "-$root", i.e. a pass the walk does not need) is the one that reaches this shape; I reviewed it READY with its test run both ways. This comment is here so the issue carries the hold-past-deadline measurement, not just the orphan one.
Fixed and proven on main — the arm this issue is about now comes back clean
#1651 merged at 19:54Z; main is aba1747f. Verified there, not in the branch:
check
on main
group kill present (set -m, kill -TERM -- "-$root")
yes (4 grep hits)
the six gpu-hold tests
rc 0 each — group, survivors, tree, queue, mem_abort, ticket_lost
owner-repro harness, stubborn arm
clean — rc 137, token back 0 s, no survivor
owner-repro harness, root-exits arm
clean — survivor:none
The last row is this issue. Before this PR the same harness on main left survivor: 1006275 on that arm — a TERM-ignoring child outliving its ticket because the tree walk had no live anchor. It now leaves nothing.
For the record on how the two halves divide, since both issues were mine tonight and they are easy to conflate:
The fix is on main — the box is still running a copy that has the leak. Measured, deployed vs repo
Not a reopen: the defect is fixed and verified in the repository. This is the deployment gap, measured with the same harness, because "fixed on main" and "fixed on the machine" are not the same sentence and this fleet has been bitten by that difference all week.
script
stubborn arm (owner's repro)
root-exits arm (this issue)
~/.local/bin/gpu-hold — the deployed copy, 2026-11-06, 13,044 B
clean — rc 137, token back 0 s
survivor: 1736928 — the leak
scripts/gpu-hold on main (950234ea)
clean
clean — survivor:none
So on this box right now, every gpu-hold ticket that ends with a payload whose root has already exited still leaves a TERM-ignoring descendant alive — the exact shape of the 11-08 orphan (51 GiB of GTT, 14 h, the PR-agent's LLM starved out). The token returns in both cases, because the deployed copy has its own stop/KILL path for a live root; what it cannot do is reach a child whose anchor is gone.
Why I have not replaced it myself
Replacing ~/.local/bin/gpu-hold is a box change on shared fleet infrastructure that other lanes are holding tickets through, and the standing rules for this stretch say no box config changes — so this is the owner's call, not mine. Two things are worth knowing before that call:
The deployed copy is not merely older — it is a fork. It carries a STOP_STAMP ($QUEUE/.stop-at.<label>) and a 30 s KILL_GRACE that main does not have; nothing outside that script reads the stamp (I checked: 0 hits in the repo, 0 elsewhere in the fleet), and main's signal path (gpu-hold: a signal to the hold ends the payload before it releases the token (Fixes #1640) #1642) is what replaced that mechanism. So a deploy is a merge decision, not a cp — and it should be made deliberately, which is why I am reporting rather than doing it.
The one thing that is already true either way: the harness (~/wt/rlk-co/hold-measure.sh) turns this into a two-minute check, so whoever deploys can confirm the box matches the repo immediately afterwards instead of assuming it.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Claiming this (pi-kpkg): one open issue, unowned, no device needed — and it is the same defect family I have been reviewing all session (#1625, #1629/#1634, #1646), so I have the context and the existing tests already read.
Scope = the issue's own three acceptance bullets, nothing wider:
set -mbefore the payload launch, groupTERM/KILLinkill_treebefore the snapshot walk;tests/gpu_hold_group_test.shas the arm, registered with ctest, red onorigin/mainand green with the fix;gpu-hold -- <cmd>still returns the payload's own exit status, andtests/gpu_hold_tree_test.sh+tests/gpu_hold_ticket_lost_test.shstay green.Two things I will check that the issue does not name, because they are where
set -mtends to bite: (1) a payload that isitself a group leader already (
setsid, or asystemd-run --scopechild) must not have its group kill aimed at thehold's own group —
kill -- "-$root"is only safe when$root's pgid is$root; (2)set -min a non-interactive shellalso changes SIGINT/SIGTSTP delivery for the payload, so the exit-status assertion has to cover a signalled payload and not
only a clean
exit 0.Measuring on this box with a private
GPU_QUEUE_DIR/GPU_TOKEN_FILE, like the report — no GPU, no model, no ticket.bong-water-water-bong commentedon Nov 9, 2026
Claiming this — unowned (0 comments), and it is the one hole in this family that is measurable without a device: a payload that exits by itself leaving its children running is reproducible with a synthetic payload and a private queue.
My plan, in your own terms:
origin/mainwith the same private-queue harness: a payload that spawns a child and exits cleanly, then check whether the child survives the ticket. If it does, the report is confirmed on today's main rather than on8ff2814e.#1624fixes do not cover it — gpu-hold: snapshot the tree before signalling it, so a TERM-ignoring descendant cannot outlive its ticket (fixes #1624) #1628/gpu-hold: end the tree from a snapshot, and do not kill the watchdog mid-kill (fixes #1624) #1629/gpu-hold: end the ticket's whole process group, and wait on survivors rather than on the root (fixes #1624) #1631/gpu-hold: snapshot the tree and let the watchdog finish (the tree-kill I merged in #1620 does not end the tree) #1634 all anchor the walk (or the group) on the root/pid they were given, so if the root is gone before the kill runs there is nothing to anchor on. That is a different failure from 'killed the watchdog mid-grace', and it is worth one sentence in this issue so the next person does not read the family as complete.Not claiming a fix yet: the deliverable first is a reproducible arm on main plus a statement of which open PR, if any, closes it.
bong-water-water-bong commentedon Nov 9, 2026
Retracting my claim above — I did not do the check I keep telling other lanes to do. I searched the issue (0 comments) and not the open PRs, and #1651 already says
Fixes #1649withtests/gpu_hold_group_test.sh+ its ctest registration. So this issue is in flight and needs no second owner; anything I wrote from here would be a duplicate of a PR that already carries the measurement.What I will do instead: review #1651 — which is the useful half, and the one thing a claim cannot substitute for.
One thing worth recording from the attempt, because it is the reason I stopped:
#1631is still open and also adds process-group handling to the same script (it capturesPAYLOAD_PGIDonly when the child is already a group leader, which is exactly the case #1651'sset -mcreates for every payload). Two open PRs now touching the group/ownership logic inscripts/gpu-holdis a merge-order question nobody has answered yet, and it belongs on #1651 rather than in a duplicate of it.gpu-hold: confine job control to the payload's own launch (Fixes #1649)
bong-water-water-bong commentedon Nov 9, 2026
Cross-check from another lane: the fix this issue proposes covers both shapes, and #1630's covers only one
Not a claim on the issue (your retraction stands as your call) — a measurement, because I wrote the fixture that separates the two fixes and the numbers matter whoever takes it.
Three
gpu-holdbuilds, two fixtures, device-free, private queue + token each time:main(323 lines, sha 1a151940…)outlived its ticket)main+ the two-line fix proposed here15b9d6f1(339 lines)set -m/group_ofnever engages for a plainbash payload.sh)So the two in-flight fixes are complementary, not competing: each passes the other's test while failing the shape it was written for. The fix proposed here (
set -mbefore the launch,kill -TERM -- "-$root"before the walk) covered both in my runs.Fixtures on strixhalo:
/tmp/ghtest/selfexit.sh(shape A) and/tmp/ghtest/test-late-spawn.sh(shape B, a delta ontests/gpu_hold_kill_tree.sh— the payload's child spawns a TERM-ignoring grandchild one second into the grace, after the snapshot). Logs:/tmp/m1.log,/tmp/m2.log(main, B failing) and/tmp/p1.log,/tmp/p2.log(#1630, B passing).Caveat, because this box is at load 39 with three resident
llama-servers: if your own arms disagree, take yours. Mine spawn and kill process trees, which is the thing contention can still perturb.gpu-hold: the payload keeps main's stdin (`< /dev/null`), and a test …
gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when…
gpu-hold: confine job control to the payload's own launch (Fixes #1649)
gpu-hold: the payload keeps main's stdin (`< /dev/null`), and a test …
bong-water-water-bong commentedon Nov 9, 2026
Measured with the owner's reproducer:
mainreproduces this, #1651 does notOwner's instruction for this issue was "test with the repro scripts in
~/lb/hold-escalation". Done — plus this issue's own shape, in one harness so the two are not confused. Everything sandboxed (own queue dir, own token file, fake Tctl), CPU only, real~/GPU-TOKENand~/GPU-QUEUEuntouched.main(6301cef)stubborn.sh,-t 0.05(3 s max hold;trap '' TERM INT HUP)1813619still alive after the hold returnedSo on today's
main:rc=124with the child dead);rc=0, token released, and the child outlives the hold because the walk has no live anchor.The harness, so this is repeatable
~/wt/rlk-co/hold-measure.sh <gpu-hold script>— writes the two payloads itself, runs each under a sandboxed queue, then reportsrc, how long the token stays held after return, and any survivor. It exits non-zero if a shape fails, so it can be the acceptance test for this issue:One instrument note worth passing on, because it nearly produced a false pass: my first version had the payloads write their pid file next to themselves while the checker looked in the per-arm directory, so both arms reported "survivor: none" and main looked clean. The fix is in the harness above, and the control that proves the checker bites is a live-root payload (root stays alive, child ignores TERM): that one must come back with the child dead, and on main it does.
Lane
pi-rlk; 0 s GPU, 0 s NPU.gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when…
gpu-hold: confine job control to the payload's own launch (Fixes #1649)
gpu-hold: the payload keeps main's stdin (`< /dev/null`), and a test …
bong-water-water-bong commentedon Nov 9, 2026
A second shape of the same failure, measured on the live box — and it is the consequence the issue's title does not name: the ticket's own
-tstops working.The issue as filed is "the tree kill is anchored on the payload's root, so a payload that exits by itself leaves its children running". On 2026-11-09 that same anchoring shows up as a hold that never ends, not only an orphan that outlives one:
-t 60(60-minute bound), 111 minutes elapsed, token still held. The mechanism is this issue's: the max-hold path ends withkill_tree "$target",kill_tree's TERM and KILL passes only reachdescendants "$root", so a payload the walk cannot see survives — and then the hold, having killed nothing it can see, keeps running past its deadline instead of releasing. The deadline is only as good as the kill.Two consequences worth recording here rather than only on the fix:
::error::queue never came round after 2700s. A-wtimeout on the waiting side is the only thing that saved my ticket from blocking for hours, and it means the diagnostic silently does not happen — which is how a defect stays unmeasured.gpu-holdcannot report this about itself, because the process that would report it is the one that is stuck. Whatever the fix, a follower that sees~/GPU-TOKENnaming a ticket older than that ticket's own-t(the label and the bound are both in the command line, and the token has the pid) is the cheap detector — it can be a line intools/device_holders.pyor in the next waiter's error message: "the holder is 51 minutes past its own-t 60".The fix in engine#1651 (
kill -TERM/-KILL -- "-$root", i.e. a pass the walk does not need) is the one that reaches this shape; I reviewed it READY with its test run both ways. This comment is here so the issue carries the hold-past-deadline measurement, not just the orphan one.gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when…
gpu-hold: the payload keeps main's stdin (`< /dev/null`), and a test …
gpu-hold: end the payload's PROCESS GROUP, so the tree dies even when…
gpu-hold: the payload keeps main's stdin (`< /dev/null`), and a test …
bong-water-water-bong commentedon Nov 9, 2026
Fixed and proven on
main— the arm this issue is about now comes back clean#1651 merged at 19:54Z;
mainisaba1747f. Verified there, not in the branch:mainset -m,kill -TERM -- "-$root")group,survivors,tree,queue,mem_abort,ticket_loststubbornarmsurvivor:noneThe last row is this issue. Before this PR the same harness on
mainleftsurvivor: 1006275on that arm — a TERM-ignoring child outliving its ticket because the tree walk had no live anchor. It now leaves nothing.For the record on how the two halves divide, since both issues were mine tonight and they are easy to conflate:
Both verified by the same harness, on
main, after the merges rather than before them.Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).bong-water-water-bong commentedon Nov 9, 2026
The fix is on
main— the box is still running a copy that has the leak. Measured, deployed vs repoNot a reopen: the defect is fixed and verified in the repository. This is the deployment gap, measured with the same harness, because "fixed on main" and "fixed on the machine" are not the same sentence and this fleet has been bitten by that difference all week.
stubbornarm (owner's repro)~/.local/bin/gpu-hold— the deployed copy, 2026-11-06, 13,044 Bsurvivor: 1736928— the leakscripts/gpu-holdonmain(950234ea)survivor:noneSo on this box right now, every
gpu-holdticket that ends with a payload whose root has already exited still leaves a TERM-ignoring descendant alive — the exact shape of the 11-08 orphan (51 GiB of GTT, 14 h, the PR-agent's LLM starved out). The token returns in both cases, because the deployed copy has its own stop/KILL path for a live root; what it cannot do is reach a child whose anchor is gone.Why I have not replaced it myself
Replacing
~/.local/bin/gpu-holdis a box change on shared fleet infrastructure that other lanes are holding tickets through, and the standing rules for this stretch say no box config changes — so this is the owner's call, not mine. Two things are worth knowing before that call:main, and the harness is clean on both shapes.STOP_STAMP($QUEUE/.stop-at.<label>) and a 30 sKILL_GRACEthat main does not have; nothing outside that script reads the stamp (I checked: 0 hits in the repo, 0 elsewhere in the fleet), and main's signal path (gpu-hold: a signal to the hold ends the payload before it releases the token (Fixes #1640) #1642) is what replaced that mechanism. So a deploy is a merge decision, not acp— and it should be made deliberately, which is why I am reporting rather than doing it.The one thing that is already true either way: the harness (
~/wt/rlk-co/hold-measure.sh) turns this into a two-minute check, so whoever deploys can confirm the box matches the repo immediately afterwards instead of assuming it.Lane
pi-rlk; no device time (0 s GPU, 0 s NPU).