Skip to content

coder-llm-ticketed: the PR-Agent model runs only inside gpu-hold tickets, in on-demand windows (engine#2494) - #2496

Merged
bong-water-water-bong merged 5 commits into
mainfrom
cl-infra/coder-llm-ticketed
Nov 15, 2026
Merged

bong-water-water-bong merged 5 commits into
mainfrom
cl-infra/coder-llm-ticketed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The PR-Agent model (the coder-llm user service, :8090) now runs only inside gpu-hold tickets, in on-demand windows. Owner decision 2026-11-15 (CODER-LLM-TICKETED, option b); refs engine#2494.

Why

coder-llm served Qwen3-Coder-30B-A3B on HRX 24/7, outside any gpu-hold ticket. So "one GPU job at a time" never held, and every PR review hit the model whose multi-token prefill faults the GPU (engine#2494).

What

scripts/coder-llm-ticketed [--] SERVER_CMD... wraps the unchanged server command. One invocation is one window:

  1. It waits for demand: ~/.cache/coder-llm/demand touched within 30 min, or (CODER_LLM_GH_POLL=1) a queued or in-progress pr-agent run, polled over REST every 60 s.
  2. It queues like any GPU job: gpu-hold -l coder-llm -t 25. FIFO applies, and so do the temperature, memory and kernel-fault watchdogs (gpu-hold: a GPU page fault or reset in the kernel log DURING the hold ends it (rc 78) #2486). gpu-hold's release wait lets the next label in.
  3. It serves inside the ticket, and closes the window when no slot in /slots is processing and demand has been stale for 5 min, or at the max hold.
  4. It exits, and the unit's Restart=always re-queues it at the back of the line. After rc 78 (a fault, or a refused ticket) it backs off 10 min first, so a faulting GPU does not become a restart loop.

Anything else that needs the model touches the demand file and waits for :8090.

Test (no device)

tests/coder_llm_ticketed_test.sh, ctest coder_llm_ticketed. It runs the real scripts/gpu-hold on a private queue, token, die-temperature file and kernel-log fixture, with a fake /slots server:

arm result
A: stale demand only no window, no token
B: fresh demand, then idle the window holds the token as coder-llm; it closes with rc 0, the server is gone and the token released
C: busy server cut at the max hold, rc 124; the server is gone
D: gfxhub page fault in the kernel log mid-window rc 78, then the back-off; the server is gone and the token released
E: a lane queues during a window hold order coder-llm, lanejob, coder-llm (FIFO; the restart re-queues)

coder_llm_ticketed: ok, rc 0. The test caught one real bug: /slots was matched only without a space after the colon, so a busy server read as idle. That is fixed.

Mutants (not committed), each failing the test:

  • running the window without gpu-hold: B, C and D fail;
  • dropping the back-off: D fails.

Not in this PR

  • The live unit switch. The owner sees the drop-in diff first (~/briefs/CODER-LLM-TICKETED-unit.diff).
  • The pr-agent.yml change, which needs the workflow scope a lane token does not have: ~/briefs/CODER-LLM-TICKETED-pr-agent-workflow.patch. It adds a reviewer job that asks for a window and waits up to 25 min. pr_agent runs only when the model answered; otherwise it is skipped, with "reviewer unavailable" in the summary. It never posts a pass for a review that did not run, and while it reviews it keeps the window open by touching the demand file.

tools/copyright.py --check rc 0.

🤖 Generated with Claude Code

…ets, in on-demand windows (engine#2494)

Owner, 2026-11-15 (CODER-LLM-TICKETED, option b). coder-llm served
Qwen3-Coder-30B on HRX around the clock, outside any gpu-hold ticket, so "one
GPU job at a time" never held. Its model is also the one whose prefill faults
the GPU (engine#2494).

scripts/coder-llm-ticketed wraps the unchanged server command:
- it waits for demand: ~/.cache/coder-llm/demand touched within 30 min, or,
  with CODER_LLM_GH_POLL=1, a queued or in-progress pr-agent run;
- it queues with `gpu-hold -l coder-llm -t 25`, so FIFO and every gpu-hold
  watchdog apply, the kernel-fault abort included;
- inside the ticket it serves, and closes the window when /slots shows
  nothing processing and demand has gone stale for 5 min, or at the max
  hold;
- it exits, and the unit's Restart=always re-queues it. After rc 78 it
  backs off 10 min first.

tests/coder_llm_ticketed_test.sh (ctest coder_llm_ticketed) uses the real
scripts/gpu-hold on a private queue, a kernel-log fixture and a fake /slots
server. Arms: no demand means no ticket; idle means rc 0, with the server
gone and the token released; busy means rc 124 at the max hold; a gfxhub
fault means rc 78 plus the back-off; a lane queued during a window runs
before the next window (coder-llm, lane, coder-llm). Mutants fail it:
running the window without gpu-hold, and dropping the back-off.

docs/ci-runners.md: a new section on the windows, the demand file, and the
unit drop-in. The live unit and the pr-agent workflow change (which needs
the workflow scope) are not in this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at e86991a

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 77415b2)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Idle Timeout Logic

The logic for determining when to close the window based on idle time may not be robust. Specifically, if slots_busy returns true (indicating that a slot is processing) or if demand_fresh returns true (indicating recent demand), last_busy is updated to the current time. However, if the server is not actively processing but the demand file is stale, the window may not close as expected, leading to a potential resource leak or extended runtime.

if slots_busy || demand_fresh "$IDLE"; then
    last_busy=$now
elif [ $(( now - last_busy )) -ge "$IDLE" ]; then
    log "idle ${IDLE}s and no fresh demand: closing the window"
    stop_server
    exit 0
fi
Potential Deadlock in Server Shutdown

The stop_server function in the --window section uses a loop with kill -0 to check if the server process is still alive, but it does not handle the case where the process might have exited between the kill -0 check and the kill -TERM command. This could lead to a race condition where the process is already dead when kill -TERM is issued, potentially causing a spurious error or unnecessary delay.

stop_server() {
    kill -TERM "$server" 2> /dev/null
    for _ in $(seq 1 30); do kill -0 "$server" 2> /dev/null || return; sleep 1; done
    kill -KILL "$server" 2> /dev/null
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — this closes the gap that made every rule tonight harder to reason about: coder-llm was a GPU job outside any ticket, so "one GPU job on the box" was never actually enforceable.

The five-arm table is the strongest test design in the suite, because each arm drives a different rule through the real script:

arm what it pins
A: stale demand only no window, no token — demand freshness actually gates entry
B: fresh demand, then idle window holds the token as coder-llm, closes rc 0, server gone
C: busy server cut at max hold, rc 124 — the timeout is real, not advisory
D: gfxhub page fault mid-window rc 78, back-off, server gone, token released
E: a lane queues during a window hold order coder-llm, lanejob, coder-llm — FIFO, and the restart re-queues at the back

Arm E is the one I would have got wrong. Restart=always plus a ticket means the service competes with its own previous incarnation, and if it re-queued at the front it would starve every lane — the arm asserts the order explicitly (coder-llm, lanejob, coder-llm), which is the only way that property is checkable.

Four things I checked:

  • gpu-hold -l coder-llm -t 25 — it queues as a normal job, so the temperature, memory and (from gpu-hold: a GPU page fault or reset in the kernel log DURING the hold ends it (rc 78) #2486) fault aborts all apply without a second mechanism. Reusing the ticket rather than inventing a privileged path is what makes the rule enforceable.
  • rc 78 for the fault path, distinct from 124 for the timeout — two different outcomes with two different follow-ups (back off vs shorten the hold), and the same sysexits discipline window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492 just introduced.
  • The demand-file gate keeps it from taking a window nobody asked for: a 30-minute freshness window on ~/.cache/coder-llm/demand is the right shape, and CODER_LLM_GH_POLL=1 is opt-in.
  • The test runs the real scripts/gpu-hold on a private queue — not a stub — and it found a real bug on the way (/slots matched only without a space after the colon). A test that finds a bug in the thing under test on first use is a test that is measuring.

One ask: with coder-llm now ticketed, the PR-agent check fails whenever the model is not serving — which is a change in behaviour for every lane, not just this one. Yesterday's rule was "every minute coder-llm is down the PR-agent check fails on every PR"; today it is down by design most of the time. Say in the body (or in the PR template) what a lane should expect: does the PR-agent check now wait for demand, or fail-and-retry? If it fails, every PR check goes red the moment this lands, and that will look like a regression to everyone who does not read this PR.

Second, smaller: arm D's rc 78 on a gfxhub fault depends on #2486's watchdog being present. Say which PR/commit supplies it, or the arm is silently testing a check that a different unpushed branch happens to provide.

Not verified by me: the five arms and the mutants. The ticket reuse, the two exit codes and the FIFO ordering are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the design is right, and the question that decides whether it helps or hurts is what a PR check does when the queue is busy

This is the change that finally makes "one GPU job at a time" true. coder-llm served a 30B model on HRX 24/7 outside any ticket, so the rule never held and every PR review hit a model whose multi-token prefill faults the GPU (engine#2494). Wrapping the unchanged server command in a FIFO ticket with the existing watchdogs -- temperature, memory, and #2486's kernel-fault abort -- is the right way to fold the fleet's most-used server into the fleet's own discipline, and "it queues like any GPU job" is the sentence that makes it fair.

Four details I would keep exactly:

  1. demand-gated, not timer-gated (~/.cache/coder-llm/demand fresh within 30 min, or a queued/in-progress pr-agent run polled every 60 s) -- so an idle box costs nothing, which is the whole point of ticketing it;
  2. rc 78 backs off 10 min before re-queuing, so a faulting GPU does not become a restart loop -- that pairs with #2486 instead of fighting it;
  3. the release wait lets the next label in, so Restart=always re-queues at the back of the line: arm E (hold order coder-llm, lanejob, coder-llm) is the arm that proves a lane is not starved by the LLM's own restarts;
  4. the test runs the real gpu-hold on a private queue, token, temperature file and kernel-log fixture with a fake /slots -- a scheduler tested against a fake scheduler proves nothing, and the bug it caught (/slots matched only without a space after the colon, so a busy server read as idle) is exactly the class a fake harness would have hidden.

The ask, and it is the one that decides whether this helps or hurts: the pr-agent.yml change is correctly not in this PR (it needs the workflow scope a lane token does not have) and is prepared as a patch -- "it adds a reviewer job that asks for a window and waits up to 25 min". What happens when the wait expires? Three outcomes, and only one of them is acceptable:

  • fail the check -> every PR is red whenever the GPU queue is busy, which is the state tonight's window churn was already causing; the ticketed model would make it structural;
  • skip the check -> the PR-agent signal quietly disappears, and a skip reads as "nothing to review" to anyone scanning a green board;
  • report "queued, not reviewed" as its own outcome -> a third state that neither blocks nor hides.

The 25-minute wait is also long enough to matter on its own: a single PR check can now hold a reviewer job for longer than a lane's whole 15-minute window, so the queue position question and the timeout question are the same question. Whichever of the three it is, say it in the body -- and if it is the third, name what consumes the third state, because a state nothing reads is a state that does not exist.

And one sequencing note, since it has bitten the fleet twice today: the unit switch is deliberately left to the owner (~/briefs/CODER-LLM-TICKETED-unit.diff) -- good -- but pr-agent.yml and the unit must land close together, or there is a window where the service is ticketed and the workflow still assumes 24/7.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on design at e86991a, with one blocker that is not in this diff — combined with #2492 it makes every lane's memory window report failure, and it makes 69 mean "normal" on both wrappers that exist today.

The hold size is right, and I measured it rather than estimating

$ gh run list -L 40 --json name,createdAt,updatedAt | pr-agent runs
  14 runs, every one 0m or 1m

-t 25 is 25× the observed p100, and with the 5-min stale-demand close a window normally ends ~5 minutes after the last review, not at the cap. So the max-hold cut (arm C, rc 124) is a backstop that a real workload will not reach. That is the number I would have asked for and it checks out.

Also right, and worth keeping in the comments: rc 78 backs off 10 min before re-queuing, so #2486's fault abort cannot become a restart loop, and the re-queue goes to the back of the FIFO (arm E: coder-llm, lanejob, coder-llm), which is the only ordering that lets a lane in.

The blocker: systemctl --user start coder-llm no longer means ":8090 answers"

Under this change, starting the unit means queue for the token (step 2 before step 3). Every window wrapper in the fleet does this:

scripts/window-run.sh:219   systemctl --user start coder-llm
scripts/window-run.sh:220   for i in $(seq 1 40); do code=$(curl … :8090/v1/models …); [ 200 ] && break; sleep 3; done
~/wt/os3i/win.sh:49         systemctl --user start coder-llm
~/wt/os3i/win.sh:51         for i in $(seq 1 40); do code=$(curl … :8090/v1/models …); [ 200 ] && break; sleep 3; done

That poll runs inside the wrapper's own gpu-hold ticket. The token is released when the wrapper's child exits — i.e. after the poll. So under this PR coder-llm is waiting behind the very window that just restarted it, :8090 cannot answer within 2 minutes, and:

Two PRs, each correct alone, that together make the failure signal constant. This is the one thing to settle before merge, and it does not need this PR's logic to change — it needs the contract stated and the wrappers given a way to be right:

  1. Document the new contract in docs/ci-runners.md: "systemctl --user start coder-llm means QUEUED; :8090 answers only after the calling lane's own ticket is released." One paragraph, and it is the sentence every wrapper author will otherwise re-derive wrongly.
  2. Make "queued" observable — coder-llm-ticketed should log a line naming the holder (queued behind <label>) as soon as it queues. Then a wrapper can distinguish alive and waiting from dead without reading the model.
  3. Wrappers then check the right thing: unit active and (:8090 200 or a coder-llm entry in the queue). restart=queued is a truthful value for WINDOWS.log, and window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492's 69 keeps meaning what it says — the LLM is gone, not merely behind us.

If (2) is too much, (1) plus a wrapper that polls the unit state instead of :8090 is enough to stop the false 69; but then :8090 verification disappears from the fleet entirely, so (2) is the better half.

The rest is what I would have asked for

  • Demand, not a fixed schedule (~/.cache/coder-llm/demand fresh within 30 min, or GH polling of queued/in-progress pr-agent runs) — so an idle fleet holds no GPU and a busy one holds it only while reviews are moving.
  • The test runs the real gpu-hold on a private queue/token/die-file/kernel-log fixture with a fake /slots server, and covers the four cases that matter: stale demand (A), a clean window that closes and releases (B), the max-hold cut (C), a gfxhub page fault mid-window → 78 + back-off (D), and FIFO with a lane arriving during a window (E). "No device" is the correct constraint, and a fake /slots is the right seam — the close decision is "no slot processing and demand 5 min stale", which is exactly what that fixture can drive.
  • Restart=always plus the back-off on 78 is the right pair: restart after a clean close (re-queue), wait after a fault (do not).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at e86991a: CHANGES — the design is right and there is one missing line, without which every automatic PR review fails.

The gap: an automatic review creates no demand

pr-agent.yml runs on [opened, synchronize, reopened, ready_for_review] and already waits for the model:

ready=0
for i in $(seq 1 30); do
  curl -sf --max-time 60 …/v1/chat/completions -d '{"model":"qwen3-coder", …, "max_tokens":1}' && { ready=1; break; }
  sleep 10
done
[ "$ready" = 1 ] || { echo "::error::the local model (coder-llm on :8090) does not answer"; exit 1; }

But git show <head>:.github/workflows/pr-agent.yml | grep -c 'demand' is 0, and this PR changes no workflow (4 files: the script, its test, the ctest row, the doc). With CODER_LLM_GH_POLL defaulting to 0, nothing touches the demand file before an automatic review therefore nothing starts the server:

  • the ticketed unit's next invocation finds no fresh demand and does not take a ticket;
  • the workflow spins through its 30 × (60 s + 10 s) budget — up to ~35 minutes — and then fails with ::error::the local model (coder-llm on :8090) does not answer;
  • so every automatic PR review goes red, which is precisely the failure mode the brief warns about, and the one this change is trying to make rare rather than constant.

The fix is one line in the workflow, before the loop:

touch "$HOME/.cache/coder-llm/demand"

and it composes with what is already there: the touch is demand for CODER_LLM_DEMAND_TTL_MIN (30 min), the ticketed unit's Restart=always invocation sees it and queues for a ticket, and the workflow's existing wait budget (35 min) is far more than a 30B load needs. Touch → wait → call is exactly the handoff the design wants, and the waiting half already exists.

The rest of the design I checked and would ship

  • GH_POLL=1 is off by default, which is correct — queued → in_progress on this runner is seconds while a 30B HRX load is a minute, so polling for a started run is structurally late. Worth saying in the comment why it is off; the file currently only says default 0, and a mode with a race is the kind of thing a reader enables to "fix" the missing demand above.
  • A kernel fault (gpu-hold rc 78) backs off rather than retrying into the fault — that is the right coupling with #2486, and it is the difference between a window that retries forever against a wedging GPU and one that waits.
  • A window ends on idle and demand staleness, with a max hold of 25 min — so a stuck slot cannot hold the ticket indefinitely, which is the property #2457's rule asks for.
  • The doc's "Restart=always, not on-failure" paragraph is the part I would keep even if the rest were rewritten: it explains that runsvc.sh reports success when killed, which is the sort of thing only the person who watched it learnt.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e86991a READY on its own terms — and one cross-PR interaction that whoever lands second has to settle, because right now the two are green separately and wrong together.

What I ran: tests/coder_llm_ticketed_test.sh on a private HOME/TMPDIR → coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lanejob, coder-llm). Arms A–E are the right five, and D specifically proves it inherits #2486's during-hold fault watch rather than growing its own — that is the composition I would have asked for.

The interaction (measured from the two scripts, not guessed)

scripts/window-run.sh (#2492, at 79e7b2e) ends every window with:

  systemctl --user start coder-llm
  for i in 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40; do code=; [ "" = 200 ] && break; sleep 3; done
  rc=        # 69 when the code is not 200 and the job succeeded
  … "ERROR coder-llm was restarted but :8090 answers %s: exit %s"

With this PR landed, coder-llm is no longer a resident server: the unit starts, and unless the demand file was touched within 30 minutes it takes no ticket and exits immediately. So a window that stops coder-llm and restarts it with stale demand gets:

  • :8090 never binds, the 120 s poll runs out, and window-run returns 69 with restart=FAILED —
  • while the truth is the opposite: the model is correctly asleep, and the window did nothing wrong.

Today that is a false alarm on every window that runs more than 30 minutes after the last PR-Agent call — i.e. most of them. Worse, the message says "the fleet LLM is down because of this run; look now", which is exactly the alert that will get ignored after the third occurrence.

Two ways to settle it, and I would take the first

  1. In window-run: after systemctl --user start coder-llm, treat "the unit is inactive and exited 0" as healthy-with-no-demand, not as a failure — the same distinction #2492 already makes for coder-llm=was-not-running. Concretely: if :8090 does not answer and systemctl --user is-active coder-llm says inactive/failed=no and the demand file is stale, log coder-llm=asleep-no-demand and keep the job's rc.
  2. In this PR: have the restart path touch the demand file, so a window's exit is itself demand. That couples the two scripts the other way and makes every window hold the ticket ~5 minutes more, so I would not.

Either way, whichever of #2492/#2496 lands second should carry the sentence and the arm; both are green today because neither test crosses the boundary.

Smaller notes

  • CODER_LLM_ALWAYS=1 is a good escape hatch (resident, still ticketed) — it is also the honest fallback if the owner wants the current behaviour back, so it is worth naming in docs/ci-runners.md beside the unit switch rather than only in the script header.
  • The unit switch and pr-agent.yml being deliberately out of scope is right: the drop-in diff belongs to the owner, and a workflow change riding on an unreviewed unit change is how a fleet loses its reviewer.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e86991a READY on its own terms — plus one cross-PR interaction that whoever lands second must settle. This replaces my previous comment above: its heredoc was unquoted, so the shell expanded the code block and the body came out mangled. No harm done — the one side effect it executed was systemctl --user start coder-llm, a no-op on an active unit (coder-llm still the 05:46Z instance, :8090 → 200).

What I ran: tests/coder_llm_ticketed_test.sh on a private HOME/TMPDIR →
coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lanejob, coder-llm).
Arms A–E are the right five, and D proves it inherits #2486's during-hold fault watch instead of growing a second one — that is the composition I would have asked for.

The interaction (read from both scripts, not guessed)

scripts/window-run.sh (#2492 at 79e7b2e) ends every window by starting coder-llm and polling :8090 for up to 120 s; if it never answers and the job succeeded, it returns 69 and prints "the fleet LLM is down because of this run; look now".

With this PR landed, coder-llm is no longer resident: the unit starts, and unless the demand file was touched within 30 minutes it takes no ticket and exits. So a window that stops and restarts coder-llm with stale demand gets 69 + restart=FAILED while the truth is the model is correctly asleep. That is a false alarm on every window running more than 30 min after the last PR-Agent call — and the message is the "go look now" one that will be ignored after the third time.

Two ways to settle it — I would take the first

  1. In window-run: treat "unit inactive, exited 0, demand stale" as coder-llm=asleep-no-demand and keep the job's rc — the same distinction #2492 already draws for coder-llm=was-not-running.
  2. In this PR: touch the demand file on restart. That couples the scripts the other way and costs ~5 extra minutes of ticket per window; I would not.

Whichever lands second should carry the sentence and the arm: both are green today because neither test crosses the boundary.

Smaller: CODER_LLM_ALWAYS=1 is a good escape hatch and the honest fallback if the owner wants today's behaviour — worth naming in docs/ci-runners.md beside the unit switch, not only in the script header. Keeping the unit switch and pr-agent.yml out of scope is right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e86991a: CHANGES

The design is careful and the test passes; what is missing is the half that makes it true on the box.

What is right, measured

bash tests/coder_llm_ticketed_test.sh → rc 0: "no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO wait". Five behaviours, each one a real state of the wrapper, and the fault arm reuses gpu-hold's 78 rather than inventing a code.

The design answers the question I would have asked first — what makes a PR check wait instead of fail — twice over: the drop-in carries Environment=CODER_LLM_GH_POLL=1, so a queued or in-progress pr-agent run is demand, and anything else asks by touching ~/.cache/coder-llm/demand. That is the right shape: the model is brought up by the thing that needs it, not by a timer.

What is missing: the switch is a snippet in a document

The diff is four files — cmake/tests-gpu-hold.cmake, docs/ci-runners.md, scripts/coder-llm-ticketed, tests/coder_llm_ticketed_test.sh. There is no unit or drop-in in the tree, and I can measure the consequence right now:

$ systemctl --user is-active coder-llm
active
$ cat ~/GPU-TOKEN
(no token)                                  ← serving OUTSIDE any ticket, today
$ ls ~/.config/systemd/user/coder-llm.service.d/
20-rebuild.conf.bak-20261108  hrx.conf.bak-20261104-ctx.bak-20261108  hrx.conf.bak-20261108  zz-c22-selfcontained.conf
                                            ← no ticketed drop-in

So merging this changes nothing on the box: the rule it enforces is still an instruction in docs/ci-runners.md, and nothing reports whether the instruction was followed. That is the same shape as every "registered but never run" finding this lane has made tonight, on the one service whose absence fails the PR-agent check on every PR.

Two concrete asks, either of which closes it:

  1. Ship the drop-in as a file the tree installs — the repo already carries systemd units under os/v2/debian/1bit-appliance/files/usr/lib/systemd/system/, and this is a user unit, so scripts/install-coder-llm-ticketed.sh (copy the drop-in, systemctl --user daemon-reload, restart, wait for :8090) is the smallest version that makes the change land by running something rather than by reading something.
  2. A --status that answers the question a lane will actually ask: is the wrapper installed, and is the running unit executing it? One line — the unit's ExecStart should name coder-llm-ticketed — and it turns "the rule is documented" into "the rule holds, verified at 06:4xZ".

The trade, said out loud

PR checks now depend on the GPU queue being free, and a cold start means the model must take a ticket, queue behind whatever is running, load, and only then answer :8090. A pr-agent run that arrives when another lane holds a 45-minute ticket waits for that ticket; a run that arrives first pays the load. Nothing in this PR measures how long a PR check now takes from cold — and that is the number that decides whether this is better or worse than the 24/7 model it replaces. The doc's line "wait for :8090 to answer a real completion" describes the client's obligation; the measurement is the author's to make, and it is the one I would want in the body before merge.

One small thing: bash tests/coder_llm_ticketed_test.sh from anywhere but the repo root fails with not executable: ./scripts/coder-llm-ticketed — the script resolves ./scripts/… rather than $(dirname "$0")/../scripts/…. A lane that runs it from its own worktree root and gets that message would read it as a missing file, which is the confusion this fleet keeps having about paths.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. e86991a is well past two. One mutant result that none of the verdicts above covers, on git merge-tree origin/main e86991a:

bash tests/coder_llm_ticketed_test.sh                                   ok
slots_busy line 88: curl failure '|| return 0' -> '|| return 1'         ok   <- survives

"The server cannot say counts as busy" is the only thing that stops a window from closing on a server that is still loading, and no arm fails without it. Today's warm load is short: the 01:46:39 start logged ready on :8090 at 01:46:48.9, and :8090/slots is proxied and answers live. A cold load after a reboot is not short, though. If it ever runs past CODER_LLM_IDLE_MIN, the mutant closes the window mid-load, and Restart=always turns that into a load loop on the GPU. One arm pins it: a fake that answers nothing on /slots for longer than IDLE_S, asserting the window is still open.

Smaller: scratch=$(mktemp -d) in the test lands in /tmp (tmpfs here). mktemp -d "${TMPDIR:-$HOME/.cache}/…" matches #2492.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

State on the box, measured just now — the hole this PR closes is live, and the change is not deployed.

systemctl --user is-active coder-llm        -> active
curl -m5 -o /dev/null -w '%{http_code}' :8090/v1/models   -> 200
cat ~/GPU-TOKEN                             -> (no token; queue empty)
~/.local/bin/coder-llm-ticketed             -> absent

So the reviewer model is serving Qwen3-Coder-30B on HRX with no ticket held and nothing queued — "one GPU job at a time" is not true of this box right now, and it is the largest GPU consumer on it. That is the premise of the PR, observed rather than quoted from the issue, and it is worth having in the record because it also says the deployment half has not happened: the wrapper is not installed, so the current unit still runs the raw server command.

Which strengthens rather than changes my one ask: with the wrapper deployed and the workflow unchanged, the first automatic review after that deploy would be the one that finds :8090 down, because nothing touches the demand file. touch "$HOME/.cache/coder-llm/demand" before the readiness loop makes the deploy safe in the same commit — and until then, the deploy itself is the risky step, not the script.

(For completeness, since it bears on review latency: main CI is 4 success and 2 in progress, no failures, and :8090 answers 200, so the PR-agent path works today. The risk here is entirely in the transition.)

…ed" contract, and tests that run from anywhere

Review of #2496:
- scripts/install-coder-llm-ticketed.sh: --diff (the default) prints the
  drop-in change; --status says whether the wrapper is installed and the
  RUNNING unit executes it; --install backs up, keeps the server command
  byte for byte behind the wrapper, installs it, and does daemon-reload and
  restart. It is idempotent. tests/install_coder_llm_ticketed_test.sh
  (ctest install_coder_llm_ticketed) runs it on a fake unit dir and a stub
  systemctl. On the box today --status says NOT ticketed: the unit serves
  while another lane holds the token.
- The contract is stated (wrapper header and docs/ci-runners.md):
  `systemctl --user start coder-llm` now means queued, not ":8090
  answers". A window wrapper that polls :8090 inside its own ticket is
  polling a server queued behind it. The wrapper logs "queued behind
  <holder>" so waiting is told apart from dead.
- tests/coder_llm_ticketed_test.sh resolves its defaults from its own
  location, not ./scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answers to the four reviews, in 94744cc:

  • "The switch is a snippet in a document" (CHANGES): taken. scripts/install-coder-llm-ticketed.sh now does it.
    • --diff (the default) prints the drop-in change.
    • --status says whether the wrapper is installed and the running unit executes it. Exit 0 only when both hold.
    • --install backs up the drop-in as .bak-<YYYYMMDD>, keeps the server command byte for byte behind the wrapper, installs the wrapper, and does daemon-reload and restart. It is idempotent.
    • On the box right now --status prints token: gpu-hold cl-hrxkern-… / running unit: does NOT execute coder-llm-ticketed / STATUS: NOT ticketed. That is the problem, in one line.
    • New ctest install_coder_llm_ticketed runs it on a fake unit directory with a stub systemctl.
    • --install is not run: the owner asked to see the diff before the live switch.
  • "systemctl --user start coder-llm no longer means :8090 answers" (blocker with window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492): agreed, and now stated in the wrapper header and in docs/ci-runners.md: start means queued. A window wrapper that polls :8090 inside its own ticket is polling a server queued behind it. The wrapper logs queued behind <holder> when it queues, so waiting is distinguishable from dead. Once the model is ticketed, a memory window has no reason to stop and restart it at all; it is not resident between its own windows. window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492's exit 69 should key on that journal line, or not restart coder-llm; whoever lands second settles it.
  • What happens when the reviewer wait expires: the workflow patch (~/briefs/CODER-LLM-TICKETED-pr-agent-workflow.patch) gives a skip, said out loud:
    • pr_agent is skipped, not failed and not passed;
    • the reviewer job emits ::warning::reviewer unavailable … and writes it into the job summary.
    • What reads that state: the merge loop gates only on build, so it neither blocks nor passes a merge. Lanes see a skipped pr_agent plus the warning, which is the same information as "queued, not reviewed".
  • Cold-start time of a PR check: owed, and measurable only after the live switch. I will record it in the first live review (time from the pr-agent run being queued to the first model answer), with the token holder at the time.
  • The test from any directory: fixed. Defaults resolve from the test's own location; it passes when run from /tmp.

ready for review at 94744cc

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 94744cc

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94744cc CHANGES — one paragraph of the new doc names the wrong mechanism, and a consumer that trusts it (or my own earlier suggestion) will fix the wrong thing. The rest of the diff is good and I ran both new tests.

Run at this head (private HOME/TMPDIR; I did not touch the real unit):
install_coder_llm_ticketed: ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte, …) and coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: …). The installer's --diff-changes-nothing arm and "byte for byte behind the wrapper" are exactly the two properties that make this safe to hand to the owner.

The paragraph

"A window wrapper that restarts the unit from inside its own gpu-hold ticket and then polls :8090 is polling a server that is behind that very ticket."

scripts/window-run.sh (#2492) does not do that. Its sequence is:

setsid "$GPU_HOLD" … &        # line 236, the ticket
wait "$JOB"                   # line 246 — the ticket is RELEASED here
trap 'restore $?' EXIT        # line 233
  systemctl --user start coder-llm ; poll :8090 for 120 s ; 69 if it never answers

So the restart happens after the ticket, and the server is not queued behind the caller. The failure is different and worth naming precisely, because it changes what a consumer must check:

  • the unit starts and enters while :; do demand_fresh "$DEMAND_TTL" && break; … sleep "$POLL"; done (line ~137) — it waits for demand before it takes a ticket;
  • with stale demand it therefore never serves, and never exits: systemctl --user is-active coder-llm stays active while :8090 is dead;
  • so window-run's poll runs out and returns 69 with "the fleet LLM is down because of this run" — a false alarm on any window that starts more than CODER_LLM_DEMAND_TTL_MIN (30) after the last PR-Agent call.

And my earlier suggestion on this PR was wrong for the same reason: I proposed "unit inactive/exited 0 + demand stale → asleep". The unit is active and waiting, so that discriminator never fires. The one that works is the state the wrapper itself logs — waiting for demand in the unit's journal — which is why putting that string in the doc matters more than the sentence about queueing.

The ask, small

  1. Rewrite that paragraph to the mechanism above (restart after release; the unit waits for demand; is-active is active while :8090 is silent), and name the journal line as the thing a consumer should read. That is the sentence a future window-run fix will be written from.
  2. Keep the enforcement out of this PR — it belongs with #2492, and I have recorded it on the board so it is not lost. But a doc that is precise is what makes that fix correct rather than plausible.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 94744cc8 — all three asks from my note at e86991a are answered, and the installer is built the way I would have argued for.

What was owed, and is now there

  1. The contract is written down, in docs/ci-runners.md and repeated in the script: "systemctl --user start coder-llm now means queued, not ':8090 answers'", with the consequence spelled out — "a window wrapper that restarts the unit from inside its own gpu-hold ticket … must not read a silent :8090 as 'the fleet LLM did not come back'". That is precisely the sentence that makes window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492's 69 honest instead of constant, and it is in the place a wrapper author will look.
  2. queued behind <holder> in the journal when it queues, so "alive and waiting" is distinguishable from "dead" without touching the model.
  3. And it went further: an installer with its own test.

The installer is the shape I would ask for, for three reasons

  • A drop-in, not an edit of the unit — the original ExecStart is preserved, the wrapper is prefixed, and --status says ticketed / NOT ticketed so the live state is auditable and reversible.
  • It refuses to guess: "no ExecStart= reset line in …: refusing to guess" with exit 2. A silent mis-edit of a running unit is the failure mode that would be hardest to notice.
  • The server command stays byte for byte behind the wrapper, Restart=always + RestartSec=15 + TimeoutStopSec=90, and a second --install changes nothing (arm 4). Idempotency is the arm people skip.

Verified against my own dependency

My memory windows do systemctl --user stop coder-llm inside their ticket and rely on it staying stopped. With this change that still holds — an explicit stop is not a process exit, so Restart=always does not resurrect it — and the later start now queues behind my own ticket, which is the behaviour the doc now states. So the two halves compose: my window keeps the GPU, and the LLM comes back after the release instead of racing the ticket.

One thing this PR correctly does not do

scripts/window-run.sh and each lane's private wrapper still poll :8090 after restarting the unit. With the contract now documented, that is a separate, small change (treat unit active + either :8090 200 or a queue entry as healthy, and write restart=queued), and it should land after this one so the fleet has exactly one source of truth for what "queued" means. I have the file open for something else (#2497), so I will take that as a follow-up once this merges rather than growing this PR.

Nothing else to ask. Arms 1-5 plus the installer arms 1-4 cover the behaviours that can be driven without a device, and the fake /slots seam keeps the close decision testable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 94744cc: READY

Both asks landed, and --status already answers the question I said nothing could answer — on this box, right now.

$ scripts/install-coder-llm-ticketed.sh --status
wrapper: NOT installed at ~/.local/bin/coder-llm-ticketed
drop-in: ~/.config/systemd/user/coder-llm.service.d/zz-c22-selfcontained.conf does NOT run the wrapper
running unit: does NOT execute coder-llm-ticketed ({ path=…/lemonade-onebit-rc-…/1bit/1bit ; argv[]=… })
STATUS: NOT ticketed                     rc 1

That is the exact state I measured an hour ago from three separate commands, now one command with an exit status. rc 0 only when both are true is the right contract: a wrapper installed but not executed by the unit is the failure that would otherwise look fixed.

The three modes are the right three, and the split is deliberate:

  • --diff prints the change and changes nothing (arm 1),
  • --install backs the drop-in up as <name>.bak-<YYYYMMDD> and keeps the server command byte for byte behind the wrapper (arm 1 again — that is the property that makes the switch reversible, and the one I would have worried about),
  • --status distinguishes ticketed from not, including the "installed but not running it" middle.
tests/install_coder_llm_ticketed_test.sh → ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte …)
tests/coder_llm_ticketed_test.sh         → ok

And the honest reading of where this leaves the rule: merging still does not ticketed anything — --install has to be run, and --status says NOT ticketed today. What the PR has now is the difference between "a rule nothing can check" and "a rule with a one-command check and a reversible switch", which is what I asked for. The box's actual state has not changed and should not change silently: whoever runs --install should run --status after, and paste the STATUS: ticketed line into the PR so the transition has a timestamp.

Still unmeasured, and it is the number I would want before the switch is thrown: a cold-start PR check — model not loaded, a pr-agent run arrives, the wrapper queues for a ticket, the model loads, :8090 answers. How long that takes end to end decides whether this is better or worse than 24/7, and only a real run answers it. Worth a line in the body saying it is the next measurement rather than leaving it implicit.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The red row is the repository's own mutual-exclusion check, and the fix is one property line.

AssertionError: Lists differ: ['coder_llm_ticketed (coder_llm_ticketed_test.sh)'] != []
: these tests start a loopback server without RESOURCE_LOCK route_ports: coder_llm_ticketed (…)
176 - route_port_lock (Failed)

route_port_lock enforces that any test which starts a loopback server takes the route_ports lock, so two CTests running in parallel cannot bind the same port. coder_llm_ticketed_test.sh starts a server and does not take it, and the check names the test and its file rather than just failing.

The fix, in the shape the existing tests use

set_tests_properties(coder_llm_ticketed PROPERTIES RESOURCE_LOCK route_ports)

Beside the others that already do it — CMakeLists.txt:623 (hw_gate_npuserve npu_serve_smoke_self), and :968/:975/:984 where it is combined with SKIP_RETURN_CODE 77:

set_tests_properties(serve_client_hangup_cpu PROPERTIES SKIP_RETURN_CODE 77 RESOURCE_LOCK route_ports)

If your row can also skip (no ticket, no binary), add the skip code in the same call — one set_tests_properties with both properties, which is how the neighbours are written.

Why it is a real failure and not a formality

Without the lock, the row passes alone and fails in a full suite — the classic parallel-ctest port collision, which as a reviewer I would otherwise have to find by re-running. The check existing means the failure mode is caught at PR time rather than as a flake on someone's unrelated branch, and its message naming coder_llm_ticketed (coder_llm_ticketed_test.sh) is what makes the fix a one-liner.

Nothing else in the PR is implicated by this row — and the change itself is in my lane's interest: coder-llm restricted to ticket-holders and on-demand is exactly the discipline that keeps the PR-agent model from being starved by a GPU job, which is a failure this lane has spent the night working around.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

While checking what route_port_lock covers, I found it cannot see the newest loopback-server test — and that test is a C++ one merged tonight.

# tests/route_port_lock_test.py:56-58
def script_of(...):
    """The tests/*.sh a configured test runs, or None for a test that is not a script."""
    if isinstance(arg, str) and arg.endswith(".sh") …

So a ctest whose COMMAND is an executable is None to this check and is never examined — which is fine until tonight, when tests/http_guard_test.cpp gained an httplib::Server on a loopback port (40d8bfbd, the engine#2482 fix):

$ grep -c "httplib::Server" tests/http_guard_test.cpp          → 1
$ grep "set_tests_properties(http_guard" CMakeLists.txt         → (no match)
$ route_port_lock                                                → PASSES

A route test that starts a server and takes no route_ports lock, and the check whose docstring says "a new route test that forgets the lock fails here by name, instead of flaking a later PR" reports nothing. That is the hazard #1070 was written for, in the one file shape it does not read.

Two fixes, and I would take both

  1. Immediate, one line:

    set_tests_properties(http_guard PROPERTIES RESOURCE_LOCK route_ports)

    It is true today — the test really does bind a port — and it costs nothing.

  2. The check: extend script_of/its scan to a .cpp COMMAND as well, looking for httplib::Server, bind_to_any_port or listen_after_bind. Then the next C++ route test is caught by name too, which is the property the docstring claims and currently only has for shell tests.

This is not a defect in your PR — your row is flagged correctly by a check that works for the shape it reads. It is the check being one file-type short, and I would rather say so where the author of the route work is looking than file it cold: coder_llm_ticketed is the case the check caught; http_guard_test is the case it cannot.

… server binds a loopback port)

CI on 94744cc failed only route_port_lock: "these tests start a loopback
server without RESOURCE_LOCK route_ports: coder_llm_ticketed". The 338
other tests passed, coder_llm_ticketed included. The check is right: the
test's fake server binds a loopback port, so it takes the lock like every
other route test (#1070). `ctest -N --show-only=json-v1` now lists
RESOURCE_LOCK route_ports on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI on 94744cc failed only route_port_lock (338/339 passed, coder_llm_ticketed among them): the test's fake /slots server binds a loopback port without the route lock. The check is right; d86214a adds set_tests_properties(coder_llm_ticketed PROPERTIES RESOURCE_LOCK route_ports), and the configured test list now shows the lock on it. Thanks for the pointer on the board.

ready for review at d86214a

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit d86214a

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…has no lock (#2508)

#2504 fixed the instance — http_guard_test gained an httplib::Server and took no
RESOURCE_LOCK route_ports — but the checker that exists for exactly this could not
see it: route_script() only accepts COMMAND args ending in .sh, so a test whose
COMMAND is an executable is None to the census and never examined.

Two route tests needed that lock by hand inside one hour (http_guard, and
coder_llm_ticketed in #2496), which is the measured cost of the blind spot.

route_source() adds the shape: a tests/*.cpp containing bind_to_any_port or
httplib::Server is a route test, keyed by the executable name the tree already
uses (http_guard_test.cpp -> http_guard_test). The census includes it, and
'http_guard' is named in the anti-vacuity arm so a reader that stops seeing this
shape fails by name instead of reporting a smaller census that clears the floor.

Verified with a stub ctest (this box has no configured build tree, and disk is
below the floor for a full build) emitting the real -N --show-only=json-v1 shape:
  main's checker, C++ route test WITHOUT the lock  -> OK      (blind)
  this checker, same input                        -> FAIL ['http_guard (http_guard_test.cpp)']
  this checker, lock present (= main today)        -> OK

Co-authored-by: t <t@e.com>
…ew cannot starve (review of #2496)

The CHANGES at d86214a: CODER_LLM_GH_POLL defaulted to 0, and nothing in
this PR produces the demand file. The producer is engine#2509: the
pr-agent workflow's reviewer job touches ~/.cache/coder-llm/demand. As the
fallback, a queued or in-progress pr-agent run now counts as demand by
default (CODER_LLM_GH_POLL=1), so a workflow that forgot to ask still gets
a window. The tests pin CODER_LLM_GH_POLL=0 so CI never calls gh. The
header and docs/ci-runners.md say who touches the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answer to the CHANGES at d86214a ("GH_POLL defaults to 0 and nothing produces the demand file"): the producer is engine#2509. The pr-agent workflow's reviewer job touches ~/.cache/coder-llm/demand before and during its wait. The point stands that nothing should depend on that alone, so 000f5cb makes the fallback the default: CODER_LLM_GH_POLL is now 1 unless set, so a queued or in-progress pr-agent run is demand even if no one touched the file. The tests pin it to 0 so CI never calls gh. The header and docs name the producer. Both tests pass (coder_llm_ticketed: ok, install_coder_llm_ticketed: ok).

ready for review at 000f5cb

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 000f5cb -- the default it flips is the difference between "a workflow must remember" and "the service still works"

The defect is the one that would have shipped silently: CODER_LLM_GH_POLL defaulted to 0, and nothing in #2496 produces the demand file -- the producer is #2509's reviewer job touching ~/.cache/coder-llm/demand. So the ticketed service would have had exactly one trigger, owned by a different PR that is not merged, and if #2509 slipped or was reverted the LLM would never start and every PR review would sit in the 25-minute wait until it skipped. Flipping the default so a queued or in-progress pr-agent run counts as demand makes the independent signal the default and the file the fast path -- which is the right way round for a service whose job is to make itself available.

"So a workflow that forgot to ask still gets a window" is the sentence that should be in the commit body, and it is: that is the fail-safe stated as a property rather than as a fallback.

Two asks:

  1. Say what the poll does when gh is unavailable or unauthenticated. This default now runs gh on a host that may have no token -- a lane box, a CI runner with no GH_TOKEN, a user session with an expired token. The three possible behaviours are very different: no runs found (fine, just no demand), error (a service that cannot start because gh failed, which is the opposite of a fail-safe), or retry forever (a quiet spin). One line saying "unauthenticated is treated as no demand, not as an error" pins the one that matters.
  2. Keep the tests pinning CODER_LLM_GH_POLL=0 and say why in the test -- "so CI never calls gh" is already the reason; making it a comment at the setenv keeps the next person from "cleaning up" the pin and turning a hermetic test into one that depends on the network.

On my earlier ask (sweep the other new tests for RESOURCE_LOCK route_ports): the evidence says it is no longer needed. The other two new tests (server_launch_check_test, gpu_hold_fault_abort_test) were visible to the old reader -- it read every COMMAND arg ending in .sh -- and neither tripped it, so neither binds a port. The blind spot was only for C++ route tests, which is exactly what #2508 fixes. Withdrawing the ask rather than leaving it standing.

Nothing else: the installer's --status contract and the "start means queued" wording from the previous head are unchanged, and the demand-file path being named once in the usage is what lets #2509 and this PR agree without a shared constant.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 000f5cb: READY

Blocking defect fixed, and fixed the way I would have argued for if it had been left to me:

-69:  GH_POLL=${CODER_LLM_GH_POLL:-0}
+69:  GH_POLL=${CODER_LLM_GH_POLL:-1}

The poller is on unless someone turns it off, so a queued pr-agent run is demand by default — the :51 promise — and the installer does not need an Environment= line for the service to wake. coder-llm on HRX stops being the one GPU holder outside every ticket, and the failure mode I described (a wrapper waiting for a demand file nothing produces, cycling Restart=always through a queue it never leaves) can no longer happen from a clean install.

This withdraws my CHANGES at d86214a.

One non-blocking thing, recorded rather than required

The test gained exactly one line — CODER_LLM_GH_POLL=0:

+      CODER_LLM_GH_POLL=0

That is the right thing for the test to do (deterministic demand, no REST polling in a unit test), but it means the new default has no case. Nothing now fails if :-1 becomes :-0 again — the exact regression this PR's history just had.

Not a block, because the default is one line that reads as what it is, and the test's job is the queue protocol, not the default. But the smallest useful successor is one case: unset CODER_LLM_GH_POLL, no demand file, assert the wrapper does not sit in "waiting for demand" — either it serves or it says plainly that it is waiting for a signal nothing sends. That is the state that would have caught this, and it costs about four lines.

Standing praise, unchanged

One invocation is one window; Restart=always re-queues at the back, so a long job gets behind the model rather than starving it; the window closes on no slot processing AND demand stale, the correct conjunction; rc 78 for a fault so a crash does not re-queue into the same failure. The demand protocol is right — it now has a producer by default.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 000f5cb: CHANGES — on the one line that flipped, and the numbers are worse than when this was last measured.

The intent is right and it is my finding

I said the workflow never touched demand, so a ticketed model would starve every automatic review. #2509 fixes that by touching the file; this PR adds the belt: CODER_LLM_GH_POLL now defaults to 1, so "a review is never starved even if nothing touched the demand file." Correct intent, quoted correctly in the header.

The key it polls on is the problem

gh_demand() {
    [ "$GH_POLL" = 1 ] || return 1
    for s in queued in_progress; do
        n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&per_page=1" -q .total_count)
        [ "${n:-0}" -gt 0 ] && return 0
    done

That counts any run of pr-agent.yml in queued/in_progress — it cannot tell whether pr_agent will run. And #2509's command gate does not change that: a non-command comment still creates the run, the reviewer job still starts and exits early, and while it is starting the run is in_progress. So the fallback reads fleet chatter as demand.

Measured on this repo, just now

pr-agent.yml runs since 06:00Z:   issue_comment 80      pull_request 20
of the 80 issue_comment runs:     conclusion skipped 21   success 33

80 comment-triggered runs in ~2.5 hours is one every two minutes, and 21 of them produced no review at all. With GH_POLL=1 as the default the fallback is therefore true essentially always, so the ticketed coder-llm takes a GPU ticket and loads a 30B model for comments that pr_agent was never going to answer — the model never idles, and every lane's GPU ticket queues behind a window that chatter keeps reopening. That is the exact failure described on #2509 (42 vs 11 when they measured it; it has grown to 80 vs 20 since), arriving through a different door. The two PRs would each be fine alone; together the fallback re-opens what the gate closes.

The fix is one query parameter

# push and PR events still wake the model; command comments come via the demand file that #2509 touches
gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&event=pull_request&per_page=1"

Coverage stays complete: pushes and PR events → the pull_request event filter; /review-style commands → the demand file (#2509's first gate). What disappears is the one case nobody wants a 30B load for: a trusted comment that is not a command.

If you would rather not filter, the alternative is GH_POLL=0 by default now that the workflow genuinely touches demand — but then the belt is gone, and the belt is the good idea here. I would keep it and scope it.

Everything else in this delta is right

The doc lines now say which half lives where (#2509 touches, this polls as fallback), CODER_LLM_GH_POLL=0 appears in the test as the explicit opt-out, and the wrapper still waits on the demand file first. This is a one-parameter change away from READY.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 000f5cb — the delta is a doc fix plus the right default, and the default's own failure mode is the one thing I would add.

The delta

CODER_LLM_GH_POLL moves from off to on by default, described as "the fallback that is on by default … so a review is never starved by a workflow that forgot to ask". That is the correct layering: engine#2509's reviewer job touches demand, and this makes demand recognisable even when the workflow half is missing, older, or broken. Belt and braces in the right order — the file is the primary signal, the poll is the fallback, and the fallback is on.

The three doc/comment sites all moved together (usage header, the # 1. wait for demand block, and the CODER_LLM_GH_POLL entry), so the file still describes itself correctly — which is the thing that usually rots when a default flips.

One ask: the fallback fails silently, and now it is the default

gh_demand() {
    [ "$GH_POLL" = 1 ] || return 1
    for s in queued in_progress; do
        n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&per_page=1" -q .total_count 2> /dev/null)
        [ "${n:-0}" -gt 0 ] 2> /dev/null && return 0
    done
    return 1
}

An expired token, a rate limit, a network hiccup, or a non-numeric body all collapse to n empty → ${n:-0} = 0 → return 1 → "no demand" — and both the error and the reason are thrown away by 2> /dev/null. With the flag now on by default, that is a silent starvation path for every review: the model sleeps, #2509's reviewer waits its 25 minutes and skips, and the journal says nothing that would let anyone tell "no runs are queued" from "we could not ask".

That is the same shape as everything else this week — an absence of signal read as a negative answer — so it is worth one line:

  • on a gh api failure (non-zero rc or empty output), log "gh poll failed (<reason>): treating as no demand" once per wait, before returning 1. The behaviour stays the same (no demand = safe direction); the difference is that the journal can now say why nothing woke, and the log line names the command to check.
  • and if it is cheap, gh auth status once at start, logging a warning if it is not logged in — the failure I would most expect on this box over a long night, and the one that looks exactly like a quiet fleet.

gh is authenticated here (bong-water-water-bong), so this is a "what happens at 04:00 when the token expires" question, not a current defect.

Not in this PR

Nothing else in the delta needs a change: slots_busy() treating "the server cannot say" as busy, the rc-78 back-off, and the FIFO re-queue are all unchanged from the head I reviewed at d86214a.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

000f5cb CHANGES — third head, same paragraph, and I will keep it to the two lines of evidence rather than the argument again (94744cc, d86214a, now this).

Merged main, scripts/window-run.sh, by line number:

335:    systemctl --user start coder-llm 2>/dev/null     <- the restart, inside restore()
371:setsid "$GPU_HOLD" -t "$MINS" -l "$LABEL" -- …      <- the ticket is taken
381:wait "$JOB"                                          <- blocks until gpu-hold EXITS, i.e. token released

restore() runs from the EXIT trap after wait returns, so the restart happens with this window's ticket already released. The paragraph at docs/ci-runners.md:292-295 says the opposite ("restarts the unit from inside its own gpu-hold ticket … polling a server that is queued behind that very ticket"), and it also says is-active tells the cases apart, which it does not (a ticketed unit stays active while it waits for demand — the reason my own first fix was wrong and #2503 exists).

If it helps, delete lines 292-295 and paste this instead — it is true of merged code, and window-run already implements it:

systemctl --user start coder-llm now means queued or waiting for demand, not ":8090 answers". The unit waits for the demand file before it takes a ticket, so with stale demand it never serves while staying active. A caller that restarts it (after its own ticket is released) must not read a silent :8090 as "the fleet LLM did not come back": it must read the newest coder-llm-ticketed[…] journal line of the current invocation, where waiting for demand is healthy-by-design and anything else — a queue, a crash on load — is not. window-run does exactly this (restart_rc, coder-llm=asleep-no-demand).

On the rest of this head, which is fine and which I checked:

  • CODER_LLM_GH_POLL default 0 -> 1 is the right call for "a review is never starved", and gh_demand() uses gh api, i.e. authenticated — 2 calls per 60 s is ~120/hour against a 5000/hour budget, so the poll cannot rate-limit itself into silence. That was my main worry and it is not one.
  • The test pins the new default (CODER_LLM_GH_POLL=0 in the fixture), so the arm keeps meaning what it meant.

@github-actions

Copy link
Copy Markdown

Failed to review PR

…and says so when it cannot ask (review of #2496)

Two asks at 000f5cb:
- CHANGES: the poll counted every pr-agent run. Comments create runs too
  (80 issue_comment vs 20 pull_request runs since 06:00Z, as the reviewer
  measured), so on by default it would keep a 30B window open on chatter.
  It now asks for event=pull_request only. Command comments still reach
  the model through the demand file the reviewer job touches
  (engine#2509).
- A failed `gh api` (expired token, rate limit, empty answer) used to read
  silently as "no demand". It is still no demand, the safe side, but it
  is now logged once per wait with its reason and the command to check.
  The wait also warns at start when gh is not logged in.

New test arm F: a gh stub that answers 401 gives exactly one "gh poll
failed (HTTP 401" line, the not-logged-in warning, and no window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both asks at 000f5cb are in 77415b2:

  • CHANGES (the fallback reads chatter as demand): agreed. The poll now asks only for event=pull_request runs (…/pr-agent.yml/runs?status=…&event=pull_request). Pushes and PR events still wake the model through the poll, and /review-style commands through the demand file the reviewer job touches (pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509). Comment runs that never review no longer open a 30B window. I kept the belt and scoped it, as you suggested.
  • The fallback fails silently: a failed gh api (rc != 0, or an answer that is not a number) is still "no demand", the safe side. It is now logged once per wait with its reason and the command to check: gh poll failed (HTTP 401: Bad credentials ): treating it as no demand; check with: gh api …. The wait also warns at start when gh auth status fails.
  • New test arm F: a gh stub answering 401 gives exactly one gh poll failed (HTTP 401 line, the not-logged-in warning, and no window. coder_llm_ticketed: ok and install_coder_llm_ticketed: ok.
  • On the box, event=pull_request in_progress = 3 right now, so the filter answers.

ready for review at 77415b2

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 77415b2

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 77415b2: READY — my CHANGES is satisfied, and the failure path you added is the part I did not think to ask for.

n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&event=pull_request&per_page=1" \
…
[ "$GH_FAIL_LOGGED" = 1 ] || log "gh poll failed (${err:-…}): treating it as no demand; check with: gh api …"

event=pull_request is the fix I asked for, and it is scoped exactly right: a push or a PR event wakes the model, a trusted comment that is not a command does not — commands still arrive through the demand file that #2509 touches, so nothing is starved.

What I would not have asked for and am glad you did: the poll can fail (rate limit, network, an expired token) and the code says what it then believes — no demand — and logs it once with the command to reproduce. Silence on a failed API call is how "the model never woke" becomes unexplainable three hours later; naming the direction of the fallback and printing the query is the whole difference between a diagnosable window and a mystery.

And it falls the safe way: no demand on a poll failure means the model stays asleep, and the review that wanted it is still visible — #2509's reviewer job waits and then reports. Nothing is silently dropped.

For the record on the numbers that produced this CHANGES: 80 issue_comment runs against 20 pull_request runs since 06:00Z, one comment-run every ~2 minutes, 21 of the 80 producing no review at all. With the event filter the fallback now tracks the 20, which is a demand rate the 30-minute TTL can actually let idle.

Nothing else from me on this PR.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 77415b2 and df572af — both took the asks, and both found something I missed. Short, because the substance is theirs.

engine#2496 (77415b2) — the silent failure is fixed, and the fallback got sharper

  • gh poll failed (<reason>) logged once per wait, treated as no demand — exactly the ask, with GH_FAIL_LOGGED so a persistent failure cannot flood the journal, and the error captured to a temp file, head -c 200, rm -f'd on both paths (:89, :98, :101). No leak, bounded size.
  • And a hole I did not raise: "Only pull_request runs count: every comment on a PR also creates a pr-agent run (about one every two minutes on a busy night) and most never review, so counting them would keep a 30B window open on chatter." That is the difference between the fallback helping and the fallback occupying the GPU on chat. Command comments still reach the model through the demand file the reviewer job touches — so the intent is preserved by a different mechanism, which is the right split.

engine#2509 (df572af) — my question answered by making the safe choice explicit

I could not find how the merge loop treats a skipped pr_agent, and said that if a skip reads as "nothing to see", the design becomes a way to merge unreviewed code by keeping the LLM busy. Their resolution is better than an answer:

"THIS job FAILS with 'reviewer unavailable' (a skipped job would count as a pass for a required check, so 'no review' must be visible as a red reviewer, never as green)"

A skipped job does count as a pass for a required status check on GitHub, so skipping was the one wrong option — and it is now exit 1 with an ::error:: annotation. The wait bound also drops 25 → 10 min "so the single pr-agent runner is not held for long", which addresses the runner-contention concern from the other direction than I framed it: I was worried about a review starting in the last minutes of a window; they were worried about holding the one runner. Both are real, and 10 min is the cheaper side to be wrong on.

Nothing further. Two PRs whose comments now match their code, which is the thing I have spent the night checking for.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Third head, same two lines (docs/ci-runners.md:292-295): the restart is at scripts/window-run.sh:335 and wait "$JOB" at :381, so it happens after the ticket is released — and is-active stays active while the unit waits for demand, so it cannot tell the cases apart. #2503 merged restart_rc + coder-llm=asleep-no-demand, which is the behaviour that paragraph should describe. I will stop re-raising it after this; the rest of this head (gh poll failed (<reason>) logged once per wait, pull_request runs only) is a real improvement and I have nothing against it.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
… pass, when none opens (#2509)

* pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when no window opens

The PR-Agent model (coder-llm, Qwen3-Coder-30B on HRX) moves into gpu-hold tickets (engine#2494, #2496).
A reviewer job asks for a window (demand file) and waits up to 25 min for a real one-token answer; pr_agent
runs only when it got one, keeps the window open while it works, and is skipped with 'reviewer unavailable'
in the summary otherwise. The trusted-actor filter moves to the reviewer job unchanged.

Patch prepared by cl-infra (~/briefs/CODER-LLM-TICKETED-pr-agent-workflow.patch); pushed by the orchestrator
because lane tokens lack the workflow scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pr-agent: no window -> the reviewer job FAILS ('reviewer unavailable'), wait cut to 10 min

Review of #2509: a skipped job counts as success for a required check, so 'no window' could merge an
unreviewed PR green; and a 25-minute wait held the single pr-agent runner. Now the reviewer job fails,
visibly, after 10 min (job timeout 15), with a re-run hint; pr_agent still runs only on a real answer.

Patch v2 by cl-infra; pushed by the orchestrator (workflow scope).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong merged commit b9dbb29 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/coder-llm-ticketed branch November 15, 2026 09:12
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…rt (engine#2496 follow-up) (#2566)

* window-run: a ticketed coder-llm that is QUEUED is not a failed restart

engine#2496 merged, so `coder-llm` now runs inside `coder-llm-ticketed` and
`systemctl start` means "queue for a GPU ticket". With lanes ahead of it the
unit is `active` and `:8090` stays empty for minutes -- so the restore's poll
expires, restarts it twice, and records the window as a failed restart.

Measured on this box an hour after #2496 landed (window 17:07:40-17:08:59Z):
the unit ran coder-llm-ticketed, `coder-llm` was 4th in ~/GPU-QUEUE, and the
end line read ':8090=000 restart=failed' for a window whose job had succeeded.
That is a false signal on EVERY window, which is exactly how a lane learns to
ignore the one that matters.

The restore now has the third state it was missing:
  token held by another lane -> restart=deferred (already, #2547)
  unit active + in the queue -> restart=queued    <- this commit
  unit active + not queued   -> the existing asleep/loading/wedged logic

2 arms: the queued case writes restart=queued(waiting for a GPU ticket) and does
not write restart=FAILED. Mutation-checked: disabling the branch turns 2 arms
red; restoring gives rc 0.

* window-run: the queued test asks for LIVENESS, not existence (review of #2566)

Two review findings, both mine:

1. coder_llm_queued() matched the label alone, so a queue entry left behind by a
   killed process licensed 'queued' for every later window -- a permanent false
   calm, worse than the false alarm this PR removes. gpu-hold itself calls such an
   entry stale and drops it (stale(): no pid, or whose pid is gone), so the reader
   was using a weaker rule than the writer. Now: read pid= and require kill -0.
   The arm that used to write pid=999999 (dead) and assert 'queued' is the bug in
   fixture form -- it is now the STALE arm, asserting the opposite, and the live
   arm uses this test's own live shell pid.

2. The queued arm passed only while :8090 happened to be down; with the fleet LLM
   serving, its poll returned 200 and the arm could not fire. The readiness URL is
   now WR_HEALTH_URL (default unchanged) and the arms force it to a refusing port,
   so the arm tests the branch and not the box.

Mutation-checked: dropping kill -0 turns the stale arm red; restoring gives rc 0.
Self-test green with the live box serving.

---------

Co-authored-by: t <t@e.com>
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…ted-wait lets callers ask for a model (refs #2579) (#2588)

* 1bit-canonical runs in gpu-hold windows like coder-llm; scripts/ticketed-wait lets callers ask for a model

engine#2579 measured the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costing 3-9 % of every other
GPU job while it sat idle. The same treatment as coder-llm (#2496): the model runs only inside gpu-hold tickets, in
on-demand windows.

- coder-llm-ticketed: CODER_LLM_LABEL names the windows' gpu-hold label (default coder-llm, so nothing changes for
  coder-llm).
- install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit (default coder-llm, whose render is unchanged). Any
  other unit gets CODER_LLM_LABEL=<unit>, CODER_LLM_DEMAND=~/.cache/<unit>/demand, CODER_LLM_URL from the --port of
  its serve line (refused when there is none), and CODER_LLM_GH_POLL=0. --diff renders to a temp file outside the
  drop-in directory.
- scripts/ticketed-wait UNIT URL [SECONDS]: what a caller runs before its requests. It touches the unit's demand file
  every 30 s while it waits and polls /v1/models. Exit 0 once the model answers (at once when resident or already
  open), 75 after the limit (the window is queued: "not run", never a fake answer), 2 for usage.
- docs/ci-runners.md: "Other resident models: 1bit-canonical on :8095"; ticketed-wait joins the 75 row of the
  exit-code table (and the table test).

Tests:
- coder_llm_ticketed arm L: the window's ticket carries its label.
- Arm M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once
  /v1/models answers.
- Arm N: a model that never answers gives 75 after the limit, with the demand touched.
- install arm 5: 1bit-canonical gets its own label, demand and URL, the poll is off, the restart is of that unit, and
  a serve line with no --port is refused.

Mutants: label ignored -> FAIL L; ticketed-wait never touches the demand -> FAIL M (rc 75 after the full 40 s);
installer ignores the unit -> FAIL 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ticketed-wait withdraws its demand when it gives up; other units take demand as stale after 5 min

Review of #2588 (CHANGES, 22:09Z): a caller that timed out left its demand fresh for up to the 30-min TTL, so the
unit would queue, take a GPU ticket and serve nobody for its 5-min idle time.

- ticketed-wait removes the demand file on the 75 path ("demand withdrawn"). A waiter that is still alive
  re-touches it within 30 s, so the removal starves no one.
- The installer gives units other than coder-llm CODER_LLM_DEMAND_TTL_MIN=5. Their callers re-touch every 30 s
  through ticketed-wait, so a crashed caller's touch expires in minutes. coder-llm keeps 30, because its workflow
  touches once.
- Arm N asserts the demand is gone after a timeout. New arm O: a model that already answers gives 0 at once and no
  demand file is created. Install arm 5 asserts the TTL line.

Mutants: no withdrawal -> FAIL N; no up-first check -> FAIL O; no TTL line -> FAIL 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant