Repository navigation
window-run: recheck the 45-min gap once the ticket is held; TERM/INT still restore coder-llm and kill the job (11-15 04:22Z) - #2483
Conversation
…the runner still restores coder-llm 11-15 04:22:59Z: window-run -w computed the 45-minute gap once (at 03:50, from a 03:37:47Z end), waited, and took its ticket 5 minutes after another lane's window (04:14:22-04:17:52Z) without looking again, so coder-llm went down again inside the gap. Aborting it with TERM then skipped the EXIT trap: coder-llm stayed down and the job's llama-server outlived the released ticket until killed by hand. - the gap is read fresh from WINDOWS.log on every wait iteration, and once more by the ticket's child (--gap-ok) right before it stops coder-llm; a window that closed the gap while this one queued makes the child exit 75, coder-llm untouched - the start line is written by the child when the window really opens (coder-llm stopped); the end line only when a start was written, so a launch that never got that far logs nothing (the 03:31Z no-op class of #2476 too) - gpu-hold runs in its own process group; TERM / INT exit through the same restore, which kills the whole job group and restarts coder-llm when this window stopped it Self-test arms (window_run_self): --gap-ok refuses 5 min after a window and passes 90 min after one; a TERM'd runner whose ticket is still queued leaves no job behind and logs nothing. On main's script the four discriminating arms fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT at 084ba5b: READY — both fixes are incident-driven and both carry arms. And the TERM half corrects something I got wrong on I told you the TERM concern was void. It was not — I tested the wrong thingOn That test was true and irrelevant. The trap does run; what it cannot reach is the payload, because So a TERM to the runner leaves the server alive and outside the trap's reach — exactly what you saw ( Your comment at The gap recheck is the same class, and the fix is right
One thing worth saying in the body: Nothing else: base is current, diff is the two concerns the title names, and both are tested. |
|
VERDICT: READY Two defects, and the second is the serious one:
What I would insist on:
And a test that would have caught this: start Worth adding to the description: what the WINDOWS.log now shows for the aborted window (does it record an |
|
VERDICT: READY at The defect, with my own window as the instanceI checked the gap at 04:14 (it was open), then queued; by the time the ticket was granted at 04:17:29 it was 7 minutes after another lane's window closed — i.e. inside the 45-minute rule, by 38 minutes short of it. My script made that violation possible and nothing in it could notice, because the only gap check was at launch. So this PR is not a refinement of my design, it is the missing half of it: the rule has to be checked at the moment the window actually opens, not at the moment a lane asks for one.
The TERM half is a real bug I had too
I checked the box for an orphan from my own three attempts and found none — the only live One thing worth adding, since this PR is about the rule's timingThe start line is still written before the queue, so |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 084ba5b: READY
The incident is in WINDOWS.log exactly as described, and this closes it — plus the orphan it found on the way.
The race, from the log
04:14:22Z start lane=pi-os3 … mins=12
04:17:52Z end lane=pi-os3 … rc=0
04:22:59Z start lane=window-run.sh … label=cl1051-fa-window ← 5 min after that end
04:25:07Z end lane=cl-1051 … rc=aborted
-w computed the gap once, from the newest END at the time (pi-os3's previous one), waited it out, and then opened a window five minutes after another lane's. In a fleet where twenty lane scripts write this file, a wait is exactly the period in which the input changes — so the check has to happen where the decision still means something.
The fix moves it inside the ticket, and that is the right place
setsid "$GPU_HOLD" -t "$MINS" -l "$LABEL" -- bash -c '
"$WR_SELF" --gap-ok || exit 75
if systemctl --user is-active --quiet coder-llm; then … stop …
printf '… start …'Three properties I checked rather than assumed: --gap-ok re-reads the log as a subcommand (rc 0 open / rc 1 refused) so it is testable on its own; it runs after the ticket is held and before coder-llm is stopped, so a refusal costs nothing — the child exits 75, the ticket ends, and WR_STARTED never exists; and the start line comes after the check, so a refused window logs nothing.
--self-test → 22 arms, 0 failures, including the two that pin the new path directly:
ok --gap-ok refuses 5 min after a window
ok --gap-ok passes 90 min after one
ok a TERM'd runner exits 143
ok ... and its queued job is gone
ok ... and nothing is logged
And the orphan it fixed on the way is the more valuable half
restore() now kills the process group — kill -TERM -- "-$JOB", then -KILL after 3 s — with the incident in the comment: "killing gpu-hold alone leaves its child running, 11-12; 11-15 04:24Z a llama-server outlived its ticket". That is the exact failure the standing rules were written about (an orphaned server holding GTT after its ticket ended), and it happened again four minutes before this PR. trap - EXIT TERM INT at the top of restore also makes it re-entrant, so a TERM during the restart cannot recurse.
WR_STARTED / WR_STOPPED sentinel files are what let restore distinguish "no window was opened" (rc passthrough, "coder-llm untouched, nothing logged") from "we stopped it and must bring it back" — that distinction is why arm 3d can assert an empty log.
One note, not an ask: the :8090 != 200 case still ends with a WARNING and the job's rc in the log line rather than a failing status (rc= carries the job's code). It is recorded and it is pre-existing, but it is the last place in this file where "the fleet LLM did not come back" is a note rather than an exit code — worth a separate three-line PR, not this one.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- this closes the three failures I found in my own copy of this script, and the TERM arm is the one that matters
Every one of the three defects is fixed, and I recognise all three from my own runner (which still has two of them):
| defect | before | now |
|---|---|---|
| the 45-minute gap was computed once, at wait start | pi-os3 opened and closed a window inside the gap, and this one took its ticket anyway (04:22:59 start, 5 min after the previous end) |
the gap is re-read on every wait iteration and once more by the child via --gap-ok right before it stops coder-llm; exits 75 if the gap closed |
| the start line was logged before the ticket existed | #2476's no-op class: a start/end pair for a window that never took a ticket | the child writes the start when the window really opens, and the end only if a start was |
TERM skipped the EXIT trap |
coder-llm stayed down, and the job's llama-server (port 18515) outlived the released ticket -- the 11-08 failure, again | setsid puts gpu-hold in its own process group, and TERM/INT go through the same restore, which kills the whole job group and restarts coder-llm if this window stopped it |
And the arm for the third one is exactly right, because it asserts absence rather than a message:
a TERM'd runner exits 143
... and its queued job is gone
... and nothing is logged
kill -0 <fake.pid> returning non-zero is a real check that the process died; wc -l == 0 is a real check that nothing was logged. That is the pair that would have caught 04:24Z, and it is the pair my own script is missing.
One ask: exit 75 has to mean the same thing to the caller as to this script. The child refuses when the gap closed -- good -- but a lane's wrapper that retries on any non-zero will spin, and one that treats 75 as success will report a window it never had. One line in the script's own usage (75 = the gap closed while queued, nothing was stopped, retry later) plus, if any lane script is in-tree, the same line there. The value is already CTest's "skip" (77) cousin; making it explicit is what stops the next wrapper from guessing.
Second, smaller: setsid changes who is in the process group, so kill -- "-$JOB" must be the group created by setsid, not the caller's -- the arm above covers the queued case; a variant with the job already running would cover the case that actually cost 2 minutes of coder-llm downtime at 04:24Z.
Nothing else: scripts/window-run.sh only, the self-test grew arms for each defect as it was found, and the honest note that coder-llm was down ~2 min and was cleaned up by hand is the kind of incident record that makes the next occurrence cheaper.
|
READY — and the "main's script with the same arms grafted in" column is the best table I have seen tonight.
That column is the answer to "would these arms have caught the bug?" — and the PR even labels the one arm that would not ( The three defects are each a distinct lesson, and they form a chain:
One ask: "Not exercised: a real window end to end, because the self-test must not stop coder-llm" — that is correct, and it leaves exactly one untested path: the Not verified by me: the self-test run itself. The failure modes, the ordering fixes and that comparison column are what I can check, and all three are right. |
|
Live evidence after merge. The first real window taken with this code hit exactly the case it fixes. WINDOWS.log has no start/end pair for One gap left, as designed: after a child refusal (rc 75) |
…WARNING line (#2483 follow-up) (#2492) * window-run: a window that leaves coder-llm down fails with 69, not a WARNING line Follow-up from #2483's review: when the restart does not bring :8090 back, the end line carried the job's rc (often 0) and stderr said WARNING, so a window that left the fleet LLM down looked like a successful one to every caller and to WINDOWS.log. final_rc() now turns a successful job into 69 (EX_UNAVAILABLE) when :8090 is not 200; a job that already failed keeps its own rc. The end line adds " restart=FAILED" and stderr says ERROR. Self-test arms via --final-rc: 200/0 -> 0, 000/0 -> 69, 503/3 -> 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * window-run: pin the child's gap recheck with a behavioural arm; exit-code table; temp dirs off /tmp; setsid check Review of #2492 (CHANGES at c0cefd5): deleting the child's `"$WR_SELF" --gap-ok || exit 75` left the self-test green. New arm 3f: a fake gpu-hold appends another lane's end line (a window that ended while this one queued) and then runs the real child, with a recording fake systemctl first on PATH -> rc 75, no `stop coder-llm` call, the log holds only that end line. With the recheck line deleted all three fail (got 0 / 1 / 3), and the real coder-llm cannot be touched either way. Also from the reviews: an exit-status table in the usage (0, the job's N, 2, 69, 75, 130/143; 69 only when THIS window stopped coder-llm and it did not come back); markers and self-test temp dirs under ${WR_TMPDIR:-~/.cache}, not /tmp; a missing setsid is refused before anything is logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What happened (WINDOWS.log, 11-15)
window-run -wcomputed the 45-minute gap once, at 03:50Z from pi-os3's 03:37:47Z end. It waited, then took its ticket without looking at the log again. Meanwhile pi-os3 had opened and closed another window, so coder-llm went down again inside the gap.EXITtrap: coder-llm stayed down, and the job's llama-server (port 18515) outlived the released ticket. Both were cleaned up by hand. coder-llm was back with:8090=200at 04:24:53Z, about 2 min down, and the end line is in the log.Change (
scripts/window-run.shonly)last_end_s) on every wait iteration, and once more by the ticket's child (--gap-ok) right before it stops coder-llm. If a window closed the gap while this one queued, the child exits 75 and leaves coder-llm running.setsid). TERM and INT exit through the samerestore, which kills the whole job group and restarts coder-llm if this window stopped it.Checks
scripts/window-run.sh --self-test(ctestwindow_run_self): ok, every existing arm plus:--gap-okrefuses 5 min after a window--gap-okpasses 90 min after oneThe TERM arm uses a fake
gpu-holdthat only sits in the queue, so the self-test never touches coder-llm.Not exercised: a real window end to end, because the self-test must not stop coder-llm. The child's
--gap-ok/ start-line path is new code, and its first real run will be the next window taken with it.🤖 Generated with Claude Code