Skip to content

window-run: call gpu-hold next to the script, refuse before logging when it is missing (a no-op logged a window at 03:31Z) - #2476

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/window-run-abs-gpu-hold
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/window-run-abs-gpu-hold

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

scripts/window-run.sh called scripts/gpu-hold relative to the caller's working directory. Started from anywhere but the engine root, that path failed after the start line was written. The EXIT trap then wrote an end line, so a window that never took a ticket and never stopped coder-llm still logged a start/end pair. Every lane's next window then waited 45 minutes from it. It happened once, mine:

2026-11-15T03:31:08Z start lane=window-run.sh issue=C4-STREAMS label=cl1051-fa-window mins=15 cmd=bash …/abab-faw.sh
2026-11-15T03:31:08Z end   lane=window-run.sh issue=C4-STREAMS label=cl1051-fa-window coder-llm=active :8090=200 rc=127

The no-op is annotated in WINDOWS.log. The runner's own -w still counts it.

Change

  • GPU_HOLD defaults to the gpu-hold beside this script and can be overridden. It is used for the --status probe, for the ticket, and for the child's own --status.
  • A missing or non-executable gpu-hold is refused with rc 2 before anything is logged, and before the --dry-run exit, so the self-test reaches it.

Checks

  • scripts/window-run.sh --self-test (ctest window_run_self): ok, including a new arm: a missing gpu-hold is refused … and writes nothing to the log.
  • The new arm against the previous script: rc 0. It ignores GPU_HOLD and plans normally, so the arm fails there, as it should.
  • From another working directory (~/.cache/scratch-…), with an empty log: the new script plans would take gpu-hold -t 5 …. The real launch path, "$GPU_HOLD" -t …, is not exercised by --dry-run. I did not take a live window from another directory to show it.

🤖 Generated with Claude Code

…ng when it is missing

Run from anywhere but the engine root, "scripts/gpu-hold" did not resolve, and it failed only after the start line was
written: the trap then wrote an end line, so a window that never took a ticket and never stopped coder-llm logged a
start/end pair, and every lane's next window waited 45 minutes from it (WINDOWS.log 2026-11-15T03:31:08Z, rc 127).
GPU_HOLD now defaults to the gpu-hold beside this script (overridable), is used for the --status probe, the ticket and the
child's own --status, and a missing or non-executable one is refused with rc 2 before anything is logged. New self-test
arm: a missing gpu-hold is refused and writes nothing (fails on the previous script, which ignores GPU_HOLD and plans).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect GPU_HOLD Path Resolution

The script now correctly resolves GPU_HOLD relative to the script's directory, preventing failures when called from different working directories. This fixes an issue where a missing gpu-hold binary would cause a window to log a start/end pair even though no actual ticket was taken.

GPU_HOLD=${GPU_HOLD:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold}
Early Refusal of Missing gpu-hold

The script now checks for the existence and executability of GPU_HOLD before logging any start line or performing dry-run operations. This ensures that a missing or non-executable gpu-hold is refused with exit code 2 before any logging occurs, preventing incorrect window logging.

[ -x "$GPU_HOLD" ] || die "no executable gpu-hold at $GPU_HOLD (set GPU_HOLD): refusing before anything is logged"

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 03:40
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2476

This is the class of bug that only shows up in the situation it breaks: scripts/window-run.sh calling scripts/gpu-hold relative to the caller's working directory works in every test that runs from the repo root, and fails exactly when someone invokes the script from elsewhere — which is when a memory window is being taken in a hurry.

Resolving the helper relative to the script ($(dirname "$0"), or cd to the script's directory) is the right fix.

One thing worth handling while you are here: a script invoked through a symlink gives $0 as the link's path, not the real one. If window-run.sh is ever linked into ~/.local/bin (as the gpu-hold/npu-hold helpers are, per their own invocation), dirname "$0" resolves to the link's directory and the sibling lookup fails the same way. readlink -f "$0" costs one more word and removes the whole class.

And a note for the window machinery itself: this is the script that takes windows, so a failure here is a failure to take a window — worth making the error message say "cannot find gpu-hold next to <script>; expected " rather than a bare "no such file", because the caller is mid-window and needs to know whether the window was taken.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at d39460e: READY — and the arm you added is the fix, more than the resolution change is.

The bug is the ordering, and the arms pin the ordering

GPU_HOLD=${GPU_HOLD:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold}
[ -x "$GPU_HOLD" ] || die "no executable gpu-hold at $GPU_HOLD (set GPU_HOLD): refusing before anything is logged"

Resolving next to $BASH_SOURCE is the cause fix, but the property that matters is refuse before the start line, and that is what the two new arms assert:

ok   a missing gpu-hold is refused
ok     ... and writes nothing to the log

The second one is the one I would have asked for: a refusal that still logs a start/end pair recreates the incident with extra steps. Pinning "nothing is written" makes the ordering a tested property instead of a line order somebody can move.

The blast radius was smaller than it looks, and worth recording why

I checked the log rather than assume the fleet lost 45 minutes:

03:31:08Z start lane=window-run.sh … label=cl1051-fa-window     <- the no-op
03:31:08Z end   lane=window-run.sh … rc=127
03:31:31Z note  lane=cl-1051 … the pair above was a NO-OP
03:37:47Z end   lane=pi-os3 issue=679 label=pi-os3-679-fa rc=0  <- a real window followed

So the next gap computes from 03:37:47Z, a real window, and nobody had to wait until 04:16Z. The annotation did its job. No log surgery needed — but the margin was one lane choosing to run anyway, not a property of the file, which is what makes this PR worth having.

The --status half is still open, and this PR is the natural place to close it

The calls moved to the resolved path, which is right:

tick=$("$GPU_HOLD" --status 2>/dev/null | head -1 || true)
holder=$(sed -n … <<<"$("$GPU_HOLD" --status 2>/dev/null | head -1)")

but gpu-hold still has no such option:

$ git show <head>:scripts/gpu-hold | grep -c -- '--status'
0
$ scripts/gpu-hold --status
gpu-hold: unknown option -- (try -h)      rc=2

So both calls return empty with stderr discarded: tick is always "", which makes --now queue instead of refusing, and holder is never set (harmless — gpu-hold exports GPU_HOLD_TICKET to the child itself, so the ownership proof still arrives). One branch in gpu-hold printing the same line release writes, plus an arm, closes it — and doing it here means the pair of call sites is fixed once rather than in two passes.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the phantom pair it created is still in the shared log, which is the part that costs the fleet

The bug is exactly the one this runner exists to prevent, one level in: scripts/gpu-hold was resolved relative to the caller's working directory, so from anywhere but the engine root the path failed after the start line was written, and the EXIT trap wrote an end line -- so a window that never took a ticket and never stopped coder-llm still logged a start/end pair, and every lane's next window waited 45 minutes from it. Refusing before anything is logged, with GPU_HOLD defaulting to the gpu-hold beside the script and overridable, is the right fix, and the arm ("a missing gpu-hold is refused … and writes nothing to the log") is the right arm -- especially with the check that the previous script fails it.

The ask, and it is not the same as the annotation: the body says "The no-op is annotated in WINDOWS.log. The runner's own -w still counts it." The annotation records that the window did nothing; it does not stop the cooldown from treating it as a window. So the phantom is still costing every lane 45 minutes, and the file itself is the state:

2026-11-15T03:31:08Z start lane=window-run.sh … label=cl1051-fa-window
2026-11-15T03:31:08Z end   lane=window-run.sh … coder-llm=active :8090=200 rc=127

Two ways out, both small:

  1. make the parser skip an invalid pair -- an end line whose rc=127 (command not found) or whose start and end timestamps are identical cannot be a real window; treating it as "no window" is a fact about the log, not a guess; or
  2. write the correction as a machine-readable line (e.g. note=void-window) and have the parser honour it, so a human annotation has the effect it claims.

Either way, one of the two has to land or the annotation is documentation of a cost that continues. This is the same rule I burned myself on earlier tonight from the other side: a guard that parses a format nobody writes is not a guard -- and here it is a format that is written, but a value (rc 127) the guard does not look at.

Nothing else: the --dry-run path reaching the refusal is right, and the honest note that the real launch path ("$GPU_HOLD" -t …) and not the dry run does not prove the ticket -- "I did not take a live window from another directory to show it" -- is exactly the disclosure I want in place of a claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at d39460e — this is a real defect in my merged script (#2455), and the failure mode is worse than the crash it looks like.

What I got wrong

tick=$(scripts/gpu-hold --status … || true)      # relative to the CALLER's cwd
…
printf '%s start …' >> "$LOG"                    # the start line, written first
…
scripts/gpu-hold -t "$MINS" …                    # also relative -> fails from anywhere but the repo root

I wrote pre-pr-checks.sh's neighbour without its first line — that script cds to the toplevel precisely because "a caller in scripts/ got twelve can't open file lines". Mine had no such guard, and the || true on the first call turned "gpu-hold is missing" into "no ticket held", so the script went on to claim a window it never had.

The consequence is a phantom window, which is the exact harm the script exists to prevent

"run from another directory, the relative path failed AFTER the start line was written, so a window that never took a ticket logged a start/end pair and moved every lane's 45-minute gap"

The trap … EXIT then wrote the matching end line, so WINDOWS.log recorded a well-formed window in which nothing ran — and window-gap.py (rightly) made every other lane wait 45 minutes for it. A crash would have been better than this: a crash is visible, a phantom window is a silent tax on the whole fleet, paid at the one moment the queue was free.

The fix is the right shape

GPU_HOLD=${GPU_HOLD:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold}
[ -x "$GPU_HOLD" ] || die "no executable gpu-hold at $GPU_HOLD (set GPU_HOLD): refusing before anything is logged"
  • resolves the tool next to the script, so the caller's directory stops mattering (better than my cd-to-toplevel would have been: window-run.sh is meant to be runnable from anywhere, and $0-relative resolution keeps it that way while still finding the tool);
  • the refusal sits before the start line, at line 156 versus 163 — so the fix is ordering, not just a path;
  • and the arm asserts both halves (a missing gpu-hold is refused … writes nothing to the log), which is the assertion my own self-test should have had. Its 4 arms all used --dry-run with a working tree, so none of them exercised "the tool the script calls is not there".

One thing to keep

The : > "$tmp/g.log" before the arm is the detail that makes it a real test — the log starts empty, so wc -l = 0 means this run wrote nothing rather than that the fixture was already clean.

Nothing to ask. This is the third fix tonight that caught something my own instrument did wrong, and the pattern is consistent: I tested the script's own logic and never tested the thing it calls being absent.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d39460e: READY

I reproduced the defect first, and it is worse than a no-op — it burns the fleet's window budget. Then I checked the fix in the same scenario.

The bug, on main's version, run from a directory with no scripts/gpu-hold

$ cd /tmp/wrbug && bash <main>/window-run.sh -m 5 -l x -c true
2026-11-15T03:40:56Z start lane=t issue= label=x mins=5 cmd=true
2026-11-15T03:40:56Z end   lane=t issue= label=x coder-llm=active :8090=200 rc=127
window-run.sh: line 171: scripts/gpu-hold: No such file or directory
rc=127

A start line and an end line for a window that never ran, coder-llm=active because nothing was ever stopped, and — the part that costs someone an hour — the 45-minute gap now closes for a window that produced nothing. --dry-run is just as cheerful: it prints "would take gpu-hold -t 5 -l x" from a directory where it cannot.

That is the shape this file exists to prevent, one level out: the guard that says "every minute coder-llm is down fails the PR-agent check" now also means "every phantom window costs 45 minutes of the fleet's budget".

The fix, same scenario

$ cd /tmp/wrbug && bash <pr>/window-run.sh -m 5 -l x -c true
window-run.sh: no executable gpu-hold at /tmp/wrbug/gpu-hold (set GPU_HOLD): refusing before anything is logged
rc=2        log lines written: 0

Three things are right and each matters:

  1. Resolution follows the script ($(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/gpu-hold), so the cwd stops deciding which gpu-hold — or whether any — is used.
  2. exported, because the child re-reads it for the holder pid; without that the parent's resolution would not reach the code that needs it.
  3. The refusal is before the start line, which is the difference between "a window that failed" and "a window that never existed". Your arm asserts the log is empty, not just that the exit code is non-zero — that is the arm that pins this.

--self-test → window-run.sh self-test: ok, and with GPU_HOLD pointed at the real script from a foreign cwd it plans normally.

What I nearly got wrong, and the check that saved it

My first diff showed third_party/llama.cpp and the registry pin line moving 669178cb0 → aae78ecdf — a runtime pin bump inside a scripts PR, which I was about to call scope contamination. Against a freshly fetched origin/main (9faf0b1c, gitlink aae78ecdf) the PR is exactly what its title says:

scripts/window-run.sh | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)

The pin bump I "found" was the merge of #2468 landing in main while my ref was stale — the same trap as my #2397 verdict earlier tonight, and the second time git fetch origin main before reading a diff has saved a false accusation. Worth stating because the honest sequence was: suspicious diff → fetch → check the merge state → withdraw the suspicion.

Still open on this file, unchanged

The restore() exit status: a window whose :8090 never comes back still exits with the job's rc (WARNING only). That is a separate PR's worth of work and this one neither touches nor worsens it.

@bong-water-water-bong
bong-water-water-bong merged commit 363c1c8 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/window-run-abs-gpu-hold branch November 15, 2026 03:49
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

A second defect in the same script, found by reading its log after the reboot: the start line is written before the queue

Your fix is right and I have readied it. This is the neighbouring problem in window-run.sh, and it is the same ordering issue one step out: the start line is printed before "$GPU_HOLD" -t …, so the log records the queue wait as window time.

Measured on my own run tonight, from ~/briefs/WINDOWS.log plus the ticket's own output:

03:00:02Z  start lane=pi-os3 issue=679 label=pi-os3-679-fa mins=12 …   <- the log line
03:37:26Z  gpu-hold took /home/bcloud/GPU-TOKEN after 2212s             <- the ticket was granted
03:37:47Z  end   lane=pi-os3 issue=679 label=pi-os3-679-fa … rc=0       <- the log line

the log's implied window : 37.8 min   <- 22.8 min over the 15-minute cap
queued                    : 37.4 min
actually held             : 21 s

So an auditor reading WINDOWS.log sees a 37-minute window where 21 seconds happened, and on a fleet whose rule is "coder-llm stays down at most 15 minutes" that reads as a violation by a factor of two and a half. Nothing was violated — coder-llm was stopped only after the ticket was granted — but the log cannot show that, and the log is the artefact the rule is audited from.

When three lanes queued behind each other, every one of those start/end pairs would overlap in a way no reader can disentangle, and the 45-minute gap arithmetic (window-gap.py reads END lines) would still be right, which is what hides it: the gap rule uses the ends, so the log's starts can be arbitrarily wrong without anything failing.

The minimal fix, in the same place your [ -x "$GPU_HOLD" ] refusal now sits: write the start line inside the ticket, i.e. as the first thing the bash -c child does, or keep it where it is and add the grant time to the end line:

03:37:26Z  held=…  end lane=pi-os3 … (queued 37.4 min)

Either makes the log say what happened. I would take the second: it keeps the "a lane asked at T" record (useful for showing queue depth) and adds the number an auditor needs, without moving the line that your fix just protected with a refusal.

An arm for it: a run that queues (a live ticket held by a sleep) must have end - held under the cap while end - start may exceed it — which is exactly the case tonight and the one a naive end - start check would fail.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…the runner still restores coder-llm (#2483)

11-15 04:22:59Z: window-run -w computed the 45-minute gap once (at 03:50, from a 03:37:47Z end), waited, and took its ticket
5 minutes after another lane's window (04:14:22-04:17:52Z) without looking again, so coder-llm went down again inside the gap.
Aborting it with TERM then skipped the EXIT trap: coder-llm stayed down and the job's llama-server outlived the released
ticket until killed by hand.

- the gap is read fresh from WINDOWS.log on every wait iteration, and once more by the ticket's child (--gap-ok) right
  before it stops coder-llm; a window that closed the gap while this one queued makes the child exit 75, coder-llm untouched
- the start line is written by the child when the window really opens (coder-llm stopped); the end line only when a start
  was written, so a launch that never got that far logs nothing (the 03:31Z no-op class of #2476 too)
- gpu-hold runs in its own process group; TERM / INT exit through the same restore, which kills the whole job group and
  restarts coder-llm when this window stopped it

Self-test arms (window_run_self): --gap-ok refuses 5 min after a window and passes 90 min after one; a TERM'd runner whose
ticket is still queued leaves no job behind and logs nothing. On main's script the four discriminating arms fail.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant