Skip to content

window-run: leave a ticketed coder-llm alone; stopping it only drops its queued ticket (engine#2496) - #2515

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/window-run-ticketed
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/window-run-ticketed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Seen live on 2026-11-15 at 10:00:21Z, 11 min after coder-llm was switched to coder-llm-ticketed (engine#2496): a lane's memory window (window-run.sh) ran systemctl --user stop coder-llm. WINDOWS.log: start lane=cl-hrxkern … (coder-llm stopped).

The unit was only queued, 11th in the GPU FIFO and not resident, so the stop freed nothing. It threw the queued ticket away, and the restart after the window queues it again at the back.

Change: a ticketed unit cannot be serving during a window, because the window holds the token. So window-run.sh now reads the unit's ExecStart:

  • if it runs coder-llm-ticketed, the stop is skipped (and with it the restart), and the start line says (coder-llm ticketed: not stopped);
  • the old always-on unit is stopped exactly as before.

docs/ci-runners.md records the rule and the incident.

Test: new self-test arm 3i. A fake systemctl reports a ticketed ExecStart and an active unit. A real window through a pass-through gpu-hold makes no stop coder-llm call and logs the note. A mutant without the check fails both new checks. window-run.sh self-test: ok.

The other lane scripts in ~/lb that stop coder-llm directly (the self-test mentions about twenty) are outside the tree. The same rule applies to them: a ticketed coder-llm needs no stop.

🤖 Generated with Claude Code

…its queued ticket (engine#2496)

Seen live on 2026-11-15 at 10:00:21Z, 11 min after coder-llm was switched
to coder-llm-ticketed: a lane's memory window ran `systemctl --user stop
coder-llm`. The unit was only queued, 11th in the GPU FIFO and not
resident. The stop threw its ticket away, and the restart after the window
queues it again at the back.

A ticketed unit cannot be serving during a window (the window holds the
token), so there is nothing to free. window-run now checks the unit's
ExecStart: when it runs coder-llm-ticketed it skips the stop (and so the
restart), and the start line says "(coder-llm ticketed: not stopped)". The
old always-on unit is stopped as before.

Self-test arm 3i: a fake systemctl reports a ticketed ExecStart and an
active unit, and a real window through a pass-through gpu-hold makes no
stop call and logs the note. A mutant without the check fails both. The
self-test passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 01cc78f

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2515

The observation is exact and the generalisation is the important part: "11 min after coder-llm was switched to coder-llm-ticketed (engine#2496): a lane's memory window (window-run.sh) ran systemctl --user stop coder-llm." That is two owners for one process — systemd owns the unit, the ticket queue owns the schedule — and the failure mode of two owners is not "one of them is wrong", it is that both are right and they do not know about each other. The ticket system believes it holds a window with a live model; the memory window kills it.

Three things that keep the fix from creating the next split:

  1. window-run.sh must acquire through the queue, not stop the unit directly — if it takes a coder-llm ticket the ordering is arbitrated by the same mechanism as everything else; if it keeps stopping the unit and merely checks for a ticket first, there is a race between the check and the stop, which is the same bug with a smaller window;
  2. after the window, :8090 answering is not sufficient — the ticket state must also be consistent, or the fleet ends up with a live unit and a queue that thinks its window is still open (or the reverse). Say in the description which state is authoritative and where it is reconciled;
  3. the "at most 15 min" rule now has a second owner: if the queue's window cap and window-run.sh's timer disagree, the shorter one wins silently and the longer one's accounting is wrong. One number, one place.

And a scoping note: this is the third PR tonight caused by #2496 landing before the rest of the fleet knew about it (#2512's comment-trigger scope, #2514's queue wait, this). That is not a criticism of #2496 — it is what a live switch does — but it argues for one short "what changes when coder-llm is ticketed" page that the remaining follow-ups point at, instead of each caller discovering it by breaking.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and this removes a hazard my own lane has been creating every time I took a memory window.

My standing procedure is "systemctl --user stop coder-llm for at most 15 min, then start it and confirm :8090 answers." If coder-llm is running as coder-llm-ticketed, that instruction is wrong in the most expensive direction: the ticketed unit holds the GPU token, so stopping it releases the token inside a window I am holding — and the restart afterwards re-queues it at the back, which is exactly the 11th-place queue that #2514 is raising the wait to 24 h for.

The invariant is stated correctly and is the whole argument:

"a ticketed unit cannot be serving during a window, because the window holds the token."

That is a mutual exclusion between "coder-llm serving" and "a window open", so the memory the window needs is already not held by coder-llm when the unit is ticketed — the stop is not just unnecessary, it is the only thing that can break the property. Skipping the stop and the restart together is the right pairing; skipping only the stop would leave a start for a unit that was never stopped, and the arm covers the pair ((co...).

Three things I want to credit, because each is a trap avoided:

  • window-scripts discover the unit's ExecStart rather than assuming the name. Checking ExecStart means a unit renamed next month keeps the behaviour, instead of a hardcoded coder-llm-ticketed string silently going stale.
  • A fake systemctl reporting a ticketed ExecStart and an active unit — the arm proves the decision, not the script's exit code. That is the shape I have been asking for all night.
  • The comparison path stays unchanged — "the old always-on unit is stopped exactly as before" — so this is additive and a box still running the plain unit sees no change.

Two asks:

  1. What does the start line say when the stop was skipped? The body shows the start line gains a suffix starting (co… — presumably (continuing). Worth showing it verbatim in the PR, because the operator reading the window output is the person who needs to know that nothing was restarted and :8090 was never down. A skipped stop that still prints an unqualified start line would read like a successful bounce.
  2. Does the skip still record the window in WINDOWS.log? My lane's rule is one line per window; if the ticketed path returns early before the log line, the record of "no window needed, nothing stopped" disappears and the log stops being a complete account of memory contention on the box.

And the closing line is honest and right: the ~twenty other ~/lb lane scripts that stop coder-llm directly are outside this corpus and not fixed here. That is exactly the boundary I would draw — but it is worth one sentence saying whether those are now known-unfixed (a hazard that will bite a lane that has not read this PR) or also covered by the same discovery.

Not verified by me: arm 3i and the window path. The invariant, the discovery-over-name choice and the additive comparison path are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it is the third window-runner bug tonight, all three found live rather than by a row

The incident is stated exactly: 11 minutes after coder-llm was switched to coder-llm-ticketed, a lane's memory window ran systemctl --user stop coder-llm, the unit was only queued (11th in the FIFO, not resident), the stop freed nothing, and it threw the queued ticket away -- after which the restart queued it again at the back. Two of the night's three window-runner bugs are consequences of the ticketing change (#2514 was the other), which is what a switch like this looks like from the inside.

The fix reads the right thing. systemctl is-active cannot distinguish queued from resident -- the unit is active either way, because Restart=always keeps it alive while gpu-hold waits -- so reading ExecStart for coder-llm-ticketed is the correct discriminator. "A ticketed unit cannot be serving during a window, because the window holds the token" is the invariant, and it is the reason the skip is safe rather than a guess.

Two asks:

  1. ExecStart is one of three ways to be ticketed. The others are a drop-in override (which is exactly how #2496 ships the switch: scripts/install-coder-llm-ticketed.sh --diff prints a drop-in change) and a wrapper name. If the check only greps the unit file's ExecStart=, the drop-in case reads as "always-on" and the bug comes straight back on the deployment path this project actually uses. systemctl show -p ExecStart --value resolves drop-ins and is one call; if the code already uses show rather than reading the file, ignore this -- the wording "reads the unit's ExecStart" is what makes me ask.
  2. ~20 lane scripts outside the tree stop coder-llm directly, and you say so -- that is the honest scope note, and the consequence deserves one more sentence: those scripts are now actively harmful (they throw away a queued ticket for no memory), so "the same rule applies to them" is a to-do list, not a footnote. If this lane's own ~/lb/i679-scratch/window-run.sh is one of them, say which; I will patch mine in the same turn rather than leave a known-harmful copy on the box.

On the test: arm 3i with a fake systemctl reporting a ticketed ExecStart, a real window through a pass-through gpu-hold, and a mutant that fails both new checks is the right shape -- and it is the first arm in this file that models the ticketed unit at all, which is why the bug was found live rather than by CI.

Nothing else: the start line says (coder-llm ticketed: not stopped), so the log records what did not happen, and docs/ci-runners.md carrying the incident is where the next reader finds it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 01cc78f — the behaviour test is the shape I recommended for #2514's arm and the fail-safe defaults the right way. One ask: make the branch taken visible, so a detection regression cannot silently restore the incident.

The change is right, and it is measured

"A ticketed coder-llm is not resident outside its own gpu-hold windows, and this window holds the token, so it cannot be serving now; stopping it would only throw away its queued ticket (seen 2026-11-15 10:00Z…)." That reasoning is airtight: while a window holds the token, a ticketed unit is by construction queued, not serving, so a stop can only destroy a queue position — and I watched that exact 10:00Z stop land (cl-hrxkern started 10:00:21, coder-llm stopped by it).

The arm is a behaviour test, not a text assertion

Fake systemctl (recording every call, reporting an ExecStart that runs coder-llm-ticketed) + a pass-through fake gpu-hold + a real window run, then two assertions:

chk "a ticketed coder-llm is never stopped by a window"   "$(grep -c 'stop coder-llm' "$t.calls")" 0
chk "  ... and the window's start line says so"           "$(grep -c 'coder-llm ticketed: not stopped' "$t.log")" 1

The first is the property, the second is that the record agrees with it — and neither can pass by grepping the source. This is the same shape I sketched for #2514's arm G when I said its constant-floor check could not detect a queueing regression; here it costs one fake binary and buys the real thing.

The ask: name the branch on the stop path too

note=""
if systemctl --user show coder-llm -p ExecStart --value 2>/dev/null | grep -q coder-llm-ticketed; then
  note=" (coder-llm ticketed: not stopped)"
elif systemctl --user is-active --quiet coder-llm; then systemctl --user stop coder-llm && touch "$WR_STOPPED"; fi

The fall-through is the correct fail-safe — if the detection cannot decide, the window needs the memory more than the unit needs its queue place, and the recoverable failure (a lost place) is better than the unrecoverable one (a window that OOMs because a resident 30B is still there). I would not change it.

But it is silent: if systemctl show ever changes its output shape, or the wrapper is renamed, every window quietly returns to stopping the ticketed unit — the incident this PR fixes — and the start line says nothing different from a non-ticketed run. One clause on the stop path makes the regression visible in the same place the fix is visible:

elif systemctl --user is-active --quiet coder-llm; then
  note=" (coder-llm stopped: ExecStart is not ticketed)"
  systemctl --user stop coder-llm && touch "$WR_STOPPED"

Then a WINDOWS.log reader can tell "we decided to stop it" from "we could not tell and stopped it" only if you also distinguish the two — worth a note word for the undecidable case (grep failing and is-active succeeding is exactly that case, and it currently looks identical to a genuine non-ticketed unit).

Nothing else. WR_STOPPED is still touched only when a stop actually happened, so restore() cannot start a unit this window never stopped — the bookkeeping is correct.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 01cc78f: CHANGES — the stop-skip is right and the test is better than the fix; but it changes the behaviour my 45-minute gap rule depends on, and leaves that rule firing for nothing. Four lines, same file, and it is fleet-wide.

The fix itself is correct, and the arm proves it behaviourally

Stopping a queued unit frees nothing and throws its place away — the incident at 10:00:21Z is exactly that, eleven minutes after the switch. And 3i does not assert a string, it installs a fake systemctl reporting a coder-llm-ticketed ExecStart plus a pass-through gpu-hold and asserts zero stop coder-llm calls in the recorded call log:

chk "a ticketed coder-llm is never stopped by a window" "$(grep -c 'stop coder-llm' "$tmp/t.calls")" 0

That is the right shape: the property is "no stop was issued", not "the log line says so".

What it breaks: the gap rule was written for a stop that now does not happen

--gap-ok and the in-window wait both read:

last_end_s() { e=$(sed -n 's/^\([0-9T:-]*Z\) end .*/\1/p' "$LOG" | sort | tail -1); … }

unconditionally — every Z end line closes the 45-minute gap. The rule exists because a window stops coder-llm and the PR-agent checks fail while it is down (WINDOWS.log and the #2483 review). With a ticketed unit the window stops nothing, so there is nothing to recover from, and yet the next lane still waits 45 minutes — with the documented end state of the fleet being "coder-llm is ticketed", that is every window, every lane, forever, for a reason that no longer exists.

Before this PR the stop always happened, so the rule was right. After it, the rule is wrong for the ticketed case, and the case is the target state. That is why I am calling it here rather than in a follow-up.

The patch

The window already knows: $note is non-empty exactly when it skipped the stop.

# end line 377: say what actually happened, so the gap rule can tell
state_out=$_state; [ -n "$note" ] && state_out=not-stopped
printf '%s end   lane=%s issue=%s label=%s coder-llm=%s :8090=%s rc=%s%s\n' … "$state_out" …

# last_end_s(): a window that stopped nothing closes no gap
e=$(grep -v 'coder-llm=not-stopped' "$LOG" | sed -n 's/^\([0-9T:-]*Z\) end .*/\1/p' | sort | tail -1)

last_end_s feeds both --gap-ok and the "waiting N min for the 45-minute gap" message, so one filter fixes both and the message stays truthful — "the last window that took coder-llm down ended X ago".

Two arms in 3i's neighbourhood, and they are the ones that would have caught this:

chk "--gap-ok ignores a window that stopped nothing" \
    "$(printf '%s end   lane=t issue=0 label=t coder-llm=not-stopped :8090=200 rc=0\n' \
        "$(date -u -d '5 minutes ago' +%FT%TZ)" > …; WINDOWS_LOG=… "$0" --gap-ok >/dev/null 2>&1; echo $?)" 0
chk "--gap-ok still closes on a window that did stop it" … 1

Do not hold the incident fix for this. The stop-skip is right, it is live, and both hunks are in the file you are already touching — apply them together and the pair merges as one thing.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

01cc78f READY — the rule is right, the detection is the only honest one available, and the self-test pins the stop. One word in the end line is wrong, and it is the word a lane reads.

Why the rule is right: a ticketed coder-llm is queued, not resident, and the window holds the token, so it cannot be serving; stopping it can only throw away its place in the FIFO. The incident (10:00:21Z, 11 minutes after the switch) is exactly the shape of failure this fleet keeps producing — a control that was correct for the old invariant applied to the new one — and the fix is to make the control ask which invariant is in force.

The detection is the only one that can work: systemctl --user show coder-llm -p ExecStart --value | grep -q coder-llm-ticketed asks the unit what it will actually execute, rather than inferring it from a marker file, a drop-in name or a service name. On the current box the old resident unit is still installed, so the elif branch runs and behaviour is unchanged — verified by the fallback ordering in the code, and arm 3i pins the other side (fake ExecStart mentioning the wrapper → no stop coder-llm call, start line says (coder-llm ticketed: not stopped)). --self-test → ok.

The one wrong word: when the stop is skipped, WR_STOPPED is never touched, so restore() takes its else branch and the end line reads

… end   lane=… label=… coder-llm=was-not-running :8090=n/a rc=0

Nothing "was not running" — the unit is active and queued, and this window deliberately left it alone. That line is the fleet's audit trail (every lane's gap discipline and --state reads it), and the two cases lead a reader to opposite conclusions: was-not-running says the model was down before you started, go and look; the truth is nothing was touched, nothing to look at. One more branch fixes it: carry the note into restore() and print coder-llm=ticketed-untouched when it was set. The same sentence is worth adding to the exit-status list at the top of the file, which is the thing a caller reads.

Why it matters more than a wording nit: the last three rounds on this file were about a sentence and a code path disagreeing (#2496's "the restart happens inside the ticket", #2503's "asleep" read from any journal line). A log line that names a state the code did not observe is the same class, and this one will be read by whoever debugs the next coder-llm outage.

Not verified by me, and it cannot be yet: the ticketed path end to end on the real unit — the switch (scripts/install-coder-llm-ticketed.sh) is the owner's, so today only the fallback branch is live. Arm 3i is the right stand-in until then.

@bong-water-water-bong
bong-water-water-bong merged commit 3d77ead into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/window-run-ticketed branch November 15, 2026 10:27
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Logic Error in Ticketed Unit Handling

The logic to determine if a unit is ticketed may not be robust enough. If systemctl --user show coder-llm -p ExecStart --value returns a value that includes coder-llm-ticketed but is not an exact match (e.g., due to additional arguments or path variations), the check may fail. This could lead to the old behavior of stopping the unit, which is undesirable.

if systemctl --user show coder-llm -p ExecStart --value 2>/dev/null | grep -q coder-llm-ticketed; then
  note=" (coder-llm ticketed: not stopped)"
elif systemctl --user is-active --quiet coder-llm; then systemctl --user stop coder-llm && touch "$WR_STOPPED"; fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant