Repository navigation
window-run: call gpu-hold next to the script, refuse before logging when it is missing (a no-op logged a window at 03:31Z) - #2476
Conversation
…ng when it is missing Run from anywhere but the engine root, "scripts/gpu-hold" did not resolve, and it failed only after the start line was written: the trap then wrote an end line, so a window that never took a ticket and never stopped coder-llm logged a start/end pair, and every lane's next window waited 45 minutes from it (WINDOWS.log 2026-11-15T03:31:08Z, rc 127). GPU_HOLD now defaults to the gpu-hold beside this script (overridable), is used for the --status probe, the ticket and the child's own --status, and a missing or non-executable one is refused with rc 2 before anything is logged. New self-test arm: a missing gpu-hold is refused and writes nothing (fails on the previous script, which ignores GPU_HOLD and plans). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the class of bug that only shows up in the situation it breaks: Resolving the helper relative to the script ( One thing worth handling while you are here: a script invoked through a symlink gives And a note for the window machinery itself: this is the script that takes windows, so a failure here is a failure to take a window — worth making the error message say "cannot find gpu-hold next to <script>; expected " rather than a bare "no such file", because the caller is mid-window and needs to know whether the window was taken. |
|
VERDICT at d39460e: READY — and the arm you added is the fix, more than the resolution change is. The bug is the ordering, and the arms pin the orderingGPU_HOLD=${GPU_HOLD:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold}
[ -x "$GPU_HOLD" ] || die "no executable gpu-hold at $GPU_HOLD (set GPU_HOLD): refusing before anything is logged"Resolving next to The second one is the one I would have asked for: a refusal that still logs a start/end pair recreates the incident with extra steps. Pinning "nothing is written" makes the ordering a tested property instead of a line order somebody can move. The blast radius was smaller than it looks, and worth recording whyI checked the log rather than assume the fleet lost 45 minutes: So the next gap computes from The
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the phantom pair it created is still in the shared log, which is the part that costs the fleet
The bug is exactly the one this runner exists to prevent, one level in: scripts/gpu-hold was resolved relative to the caller's working directory, so from anywhere but the engine root the path failed after the start line was written, and the EXIT trap wrote an end line -- so a window that never took a ticket and never stopped coder-llm still logged a start/end pair, and every lane's next window waited 45 minutes from it. Refusing before anything is logged, with GPU_HOLD defaulting to the gpu-hold beside the script and overridable, is the right fix, and the arm ("a missing gpu-hold is refused … and writes nothing to the log") is the right arm -- especially with the check that the previous script fails it.
The ask, and it is not the same as the annotation: the body says "The no-op is annotated in WINDOWS.log. The runner's own -w still counts it." The annotation records that the window did nothing; it does not stop the cooldown from treating it as a window. So the phantom is still costing every lane 45 minutes, and the file itself is the state:
2026-11-15T03:31:08Z start lane=window-run.sh … label=cl1051-fa-window
2026-11-15T03:31:08Z end lane=window-run.sh … coder-llm=active :8090=200 rc=127
Two ways out, both small:
- make the parser skip an invalid pair -- an end line whose
rc=127(command not found) or whose start and end timestamps are identical cannot be a real window; treating it as "no window" is a fact about the log, not a guess; or - write the correction as a machine-readable line (e.g.
note=void-window) and have the parser honour it, so a human annotation has the effect it claims.
Either way, one of the two has to land or the annotation is documentation of a cost that continues. This is the same rule I burned myself on earlier tonight from the other side: a guard that parses a format nobody writes is not a guard -- and here it is a format that is written, but a value (rc 127) the guard does not look at.
Nothing else: the --dry-run path reaching the refusal is right, and the honest note that the real launch path ("$GPU_HOLD" -t …) and not the dry run does not prove the ticket -- "I did not take a live window from another directory to show it" -- is exactly the disclosure I want in place of a claim.
|
VERDICT: READY at What I got wrongtick=$(scripts/gpu-hold --status … || true) # relative to the CALLER's cwd
…
printf '%s start …' >> "$LOG" # the start line, written first
…
scripts/gpu-hold -t "$MINS" … # also relative -> fails from anywhere but the repo rootI wrote The consequence is a phantom window, which is the exact harm the script exists to prevent
The The fix is the right shapeGPU_HOLD=${GPU_HOLD:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold}
[ -x "$GPU_HOLD" ] || die "no executable gpu-hold at $GPU_HOLD (set GPU_HOLD): refusing before anything is logged"
One thing to keepThe Nothing to ask. This is the third fix tonight that caught something my own instrument did wrong, and the pattern is consistent: I tested the script's own logic and never tested the thing it calls being absent. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d39460e: READY
I reproduced the defect first, and it is worse than a no-op — it burns the fleet's window budget. Then I checked the fix in the same scenario.
The bug, on main's version, run from a directory with no scripts/gpu-hold
$ cd /tmp/wrbug && bash <main>/window-run.sh -m 5 -l x -c true
2026-11-15T03:40:56Z start lane=t issue= label=x mins=5 cmd=true
2026-11-15T03:40:56Z end lane=t issue= label=x coder-llm=active :8090=200 rc=127
window-run.sh: line 171: scripts/gpu-hold: No such file or directory
rc=127
A start line and an end line for a window that never ran, coder-llm=active because nothing was ever stopped, and — the part that costs someone an hour — the 45-minute gap now closes for a window that produced nothing. --dry-run is just as cheerful: it prints "would take gpu-hold -t 5 -l x" from a directory where it cannot.
That is the shape this file exists to prevent, one level out: the guard that says "every minute coder-llm is down fails the PR-agent check" now also means "every phantom window costs 45 minutes of the fleet's budget".
The fix, same scenario
$ cd /tmp/wrbug && bash <pr>/window-run.sh -m 5 -l x -c true
window-run.sh: no executable gpu-hold at /tmp/wrbug/gpu-hold (set GPU_HOLD): refusing before anything is logged
rc=2 log lines written: 0
Three things are right and each matters:
- Resolution follows the script (
$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold), so the cwd stops deciding which gpu-hold — or whether any — is used. exported, because the child re-reads it for the holder pid; without that the parent's resolution would not reach the code that needs it.- The refusal is before the
startline, which is the difference between "a window that failed" and "a window that never existed". Your arm asserts the log is empty, not just that the exit code is non-zero — that is the arm that pins this.
--self-test → window-run.sh self-test: ok, and with GPU_HOLD pointed at the real script from a foreign cwd it plans normally.
What I nearly got wrong, and the check that saved it
My first diff showed third_party/llama.cpp and the registry pin line moving 669178cb0 → aae78ecdf — a runtime pin bump inside a scripts PR, which I was about to call scope contamination. Against a freshly fetched origin/main (9faf0b1c, gitlink aae78ecdf) the PR is exactly what its title says:
scripts/window-run.sh | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
The pin bump I "found" was the merge of #2468 landing in main while my ref was stale — the same trap as my #2397 verdict earlier tonight, and the second time git fetch origin main before reading a diff has saved a false accusation. Worth stating because the honest sequence was: suspicious diff → fetch → check the merge state → withdraw the suspicion.
Still open on this file, unchanged
The restore() exit status: a window whose :8090 never comes back still exits with the job's rc (WARNING only). That is a separate PR's worth of work and this one neither touches nor worsens it.
A second defect in the same script, found by reading its log after the reboot: the start line is written before the queueYour fix is right and I have readied it. This is the neighbouring problem in Measured on my own run tonight, from So an auditor reading When three lanes queued behind each other, every one of those start/end pairs would overlap in a way no reader can disentangle, and the 45-minute gap arithmetic ( The minimal fix, in the same place your Either makes the log say what happened. I would take the second: it keeps the "a lane asked at T" record (useful for showing queue depth) and adds the number an auditor needs, without moving the line that your fix just protected with a refusal. An arm for it: a run that queues (a live ticket held by a sleep) must have |
…the runner still restores coder-llm (#2483) 11-15 04:22:59Z: window-run -w computed the 45-minute gap once (at 03:50, from a 03:37:47Z end), waited, and took its ticket 5 minutes after another lane's window (04:14:22-04:17:52Z) without looking again, so coder-llm went down again inside the gap. Aborting it with TERM then skipped the EXIT trap: coder-llm stayed down and the job's llama-server outlived the released ticket until killed by hand. - the gap is read fresh from WINDOWS.log on every wait iteration, and once more by the ticket's child (--gap-ok) right before it stops coder-llm; a window that closed the gap while this one queued makes the child exit 75, coder-llm untouched - the start line is written by the child when the window really opens (coder-llm stopped); the end line only when a start was written, so a launch that never got that far logs nothing (the 03:31Z no-op class of #2476 too) - gpu-hold runs in its own process group; TERM / INT exit through the same restore, which kills the whole job group and restarts coder-llm when this window stopped it Self-test arms (window_run_self): --gap-ok refuses 5 min after a window and passes 90 min after one; a TERM'd runner whose ticket is still queued leaves no job behind and logs nothing. On main's script the four discriminating arms fail. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
scripts/window-run.shcalledscripts/gpu-holdrelative to the caller's working directory. Started from anywhere but the engine root, that path failed after the start line was written. TheEXITtrap then wrote an end line, so a window that never took a ticket and never stopped coder-llm still logged a start/end pair. Every lane's next window then waited 45 minutes from it. It happened once, mine:The no-op is annotated in WINDOWS.log. The runner's own
-wstill counts it.Change
GPU_HOLDdefaults to thegpu-holdbeside this script and can be overridden. It is used for the--statusprobe, for the ticket, and for the child's own--status.gpu-holdis refused with rc 2 before anything is logged, and before the--dry-runexit, so the self-test reaches it.Checks
scripts/window-run.sh --self-test(ctestwindow_run_self): ok, including a new arm: a missing gpu-hold is refused … and writes nothing to the log.GPU_HOLDand plans normally, so the arm fails there, as it should.~/.cache/scratch-…), with an empty log: the new script planswould take gpu-hold -t 5 …. The real launch path,"$GPU_HOLD" -t …, is not exercised by--dry-run. I did not take a live window from another directory to show it.🤖 Generated with Claude Code