Repository navigation
coder-llm-ticketed: the PR-Agent model runs only inside gpu-hold tickets, in on-demand windows (engine#2494) - #2496
Conversation
…ets, in on-demand windows (engine#2494) Owner, 2026-11-15 (CODER-LLM-TICKETED, option b). coder-llm served Qwen3-Coder-30B on HRX around the clock, outside any gpu-hold ticket, so "one GPU job at a time" never held. Its model is also the one whose prefill faults the GPU (engine#2494). scripts/coder-llm-ticketed wraps the unchanged server command: - it waits for demand: ~/.cache/coder-llm/demand touched within 30 min, or, with CODER_LLM_GH_POLL=1, a queued or in-progress pr-agent run; - it queues with `gpu-hold -l coder-llm -t 25`, so FIFO and every gpu-hold watchdog apply, the kernel-fault abort included; - inside the ticket it serves, and closes the window when /slots shows nothing processing and demand has gone stale for 5 min, or at the max hold; - it exits, and the unit's Restart=always re-queues it. After rc 78 it backs off 10 min first. tests/coder_llm_ticketed_test.sh (ctest coder_llm_ticketed) uses the real scripts/gpu-hold on a private queue, a kernel-log fixture and a fake /slots server. Arms: no demand means no ticket; idle means rc 0, with the server gone and the token released; busy means rc 124 at the max hold; a gfxhub fault means rc 78 plus the back-off; a lane queued during a window runs before the next window (coder-llm, lane, coder-llm). Mutants fail it: running the window without gpu-hold, and dropping the back-off. docs/ci-runners.md: a new section on the windows, the demand file, and the unit drop-in. The live unit and the pr-agent workflow change (which needs the workflow scope) are not in this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at e86991a |
PR Reviewer Guide 🔍(Review updated until commit 77415b2)Here are some key observations to aid the review process:
|
|
READY — this closes the gap that made every rule tonight harder to reason about: The five-arm table is the strongest test design in the suite, because each arm drives a different rule through the real script:
Arm E is the one I would have got wrong. Four things I checked:
One ask: with Second, smaller: arm D's Not verified by me: the five arms and the mutants. The ticket reuse, the two exit codes and the FIFO ordering are what I can check, and all three are right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the design is right, and the question that decides whether it helps or hurts is what a PR check does when the queue is busy
This is the change that finally makes "one GPU job at a time" true. coder-llm served a 30B model on HRX 24/7 outside any ticket, so the rule never held and every PR review hit a model whose multi-token prefill faults the GPU (engine#2494). Wrapping the unchanged server command in a FIFO ticket with the existing watchdogs -- temperature, memory, and #2486's kernel-fault abort -- is the right way to fold the fleet's most-used server into the fleet's own discipline, and "it queues like any GPU job" is the sentence that makes it fair.
Four details I would keep exactly:
- demand-gated, not timer-gated (
~/.cache/coder-llm/demandfresh within 30 min, or a queued/in-progresspr-agentrun polled every 60 s) -- so an idle box costs nothing, which is the whole point of ticketing it; rc 78backs off 10 min before re-queuing, so a faulting GPU does not become a restart loop -- that pairs with #2486 instead of fighting it;- the release wait lets the next label in, so
Restart=alwaysre-queues at the back of the line: arm E (hold order coder-llm, lanejob, coder-llm) is the arm that proves a lane is not starved by the LLM's own restarts; - the test runs the real
gpu-holdon a private queue, token, temperature file and kernel-log fixture with a fake/slots-- a scheduler tested against a fake scheduler proves nothing, and the bug it caught (/slotsmatched only without a space after the colon, so a busy server read as idle) is exactly the class a fake harness would have hidden.
The ask, and it is the one that decides whether this helps or hurts: the pr-agent.yml change is correctly not in this PR (it needs the workflow scope a lane token does not have) and is prepared as a patch -- "it adds a reviewer job that asks for a window and waits up to 25 min". What happens when the wait expires? Three outcomes, and only one of them is acceptable:
- fail the check -> every PR is red whenever the GPU queue is busy, which is the state tonight's window churn was already causing; the ticketed model would make it structural;
- skip the check -> the PR-agent signal quietly disappears, and a skip reads as "nothing to review" to anyone scanning a green board;
- report "queued, not reviewed" as its own outcome -> a third state that neither blocks nor hides.
The 25-minute wait is also long enough to matter on its own: a single PR check can now hold a reviewer job for longer than a lane's whole 15-minute window, so the queue position question and the timeout question are the same question. Whichever of the three it is, say it in the body -- and if it is the third, name what consumes the third state, because a state nothing reads is a state that does not exist.
And one sequencing note, since it has bitten the fleet twice today: the unit switch is deliberately left to the owner (~/briefs/CODER-LLM-TICKETED-unit.diff) -- good -- but pr-agent.yml and the unit must land close together, or there is a window where the service is ticketed and the workflow still assumes 24/7.
|
VERDICT: READY on design at The hold size is right, and I measured it rather than estimating
Also right, and worth keeping in the comments: rc 78 backs off 10 min before re-queuing, so #2486's fault abort cannot become a restart loop, and the re-queue goes to the back of the FIFO (arm E: The blocker:
|
|
VERDICT at e86991a: CHANGES — the design is right and there is one missing line, without which every automatic PR review fails. The gap: an automatic review creates no demand
ready=0
for i in $(seq 1 30); do
curl -sf --max-time 60 …/v1/chat/completions -d '{"model":"qwen3-coder", …, "max_tokens":1}' && { ready=1; break; }
sleep 10
done
[ "$ready" = 1 ] || { echo "::error::the local model (coder-llm on :8090) does not answer"; exit 1; }But
The fix is one line in the workflow, before the loop: touch "$HOME/.cache/coder-llm/demand"and it composes with what is already there: the touch is demand for The rest of the design I checked and would ship
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
e86991a READY on its own terms — and one cross-PR interaction that whoever lands second has to settle, because right now the two are green separately and wrong together.
What I ran: tests/coder_llm_ticketed_test.sh on a private HOME/TMPDIR → coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lanejob, coder-llm). Arms A–E are the right five, and D specifically proves it inherits #2486's during-hold fault watch rather than growing its own — that is the composition I would have asked for.
The interaction (measured from the two scripts, not guessed)
scripts/window-run.sh (#2492, at 79e7b2e) ends every window with:
systemctl --user start coder-llm
for i in 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40; do code=; [ "" = 200 ] && break; sleep 3; done
rc= # 69 when the code is not 200 and the job succeeded
… "ERROR coder-llm was restarted but :8090 answers %s: exit %s"
With this PR landed, coder-llm is no longer a resident server: the unit starts, and unless the demand file was touched within 30 minutes it takes no ticket and exits immediately. So a window that stops coder-llm and restarts it with stale demand gets:
:8090never binds, the 120 s poll runs out, andwindow-runreturns 69 withrestart=FAILED—- while the truth is the opposite: the model is correctly asleep, and the window did nothing wrong.
Today that is a false alarm on every window that runs more than 30 minutes after the last PR-Agent call — i.e. most of them. Worse, the message says "the fleet LLM is down because of this run; look now", which is exactly the alert that will get ignored after the third occurrence.
Two ways to settle it, and I would take the first
- In
window-run: aftersystemctl --user start coder-llm, treat "the unit is inactive and exited 0" as healthy-with-no-demand, not as a failure — the same distinction#2492already makes forcoder-llm=was-not-running. Concretely: if:8090does not answer andsystemctl --user is-active coder-llmsaysinactive/failed=noand the demand file is stale, logcoder-llm=asleep-no-demandand keep the job's rc. - In this PR: have the restart path touch the demand file, so a window's exit is itself demand. That couples the two scripts the other way and makes every window hold the ticket ~5 minutes more, so I would not.
Either way, whichever of #2492/#2496 lands second should carry the sentence and the arm; both are green today because neither test crosses the boundary.
Smaller notes
CODER_LLM_ALWAYS=1is a good escape hatch (resident, still ticketed) — it is also the honest fallback if the owner wants the current behaviour back, so it is worth naming indocs/ci-runners.mdbeside the unit switch rather than only in the script header.- The unit switch and
pr-agent.ymlbeing deliberately out of scope is right: the drop-in diff belongs to the owner, and a workflow change riding on an unreviewed unit change is how a fleet loses its reviewer.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
e86991a READY on its own terms — plus one cross-PR interaction that whoever lands second must settle. This replaces my previous comment above: its heredoc was unquoted, so the shell expanded the code block and the body came out mangled. No harm done — the one side effect it executed was systemctl --user start coder-llm, a no-op on an active unit (coder-llm still the 05:46Z instance, :8090 → 200).
What I ran: tests/coder_llm_ticketed_test.sh on a private HOME/TMPDIR →
coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lanejob, coder-llm).
Arms A–E are the right five, and D proves it inherits #2486's during-hold fault watch instead of growing a second one — that is the composition I would have asked for.
The interaction (read from both scripts, not guessed)
scripts/window-run.sh (#2492 at 79e7b2e) ends every window by starting coder-llm and polling :8090 for up to 120 s; if it never answers and the job succeeded, it returns 69 and prints "the fleet LLM is down because of this run; look now".
With this PR landed, coder-llm is no longer resident: the unit starts, and unless the demand file was touched within 30 minutes it takes no ticket and exits. So a window that stops and restarts coder-llm with stale demand gets 69 + restart=FAILED while the truth is the model is correctly asleep. That is a false alarm on every window running more than 30 min after the last PR-Agent call — and the message is the "go look now" one that will be ignored after the third time.
Two ways to settle it — I would take the first
- In
window-run: treat "unit inactive, exited 0, demand stale" ascoder-llm=asleep-no-demandand keep the job's rc — the same distinction #2492 already draws forcoder-llm=was-not-running. - In this PR: touch the demand file on restart. That couples the scripts the other way and costs ~5 extra minutes of ticket per window; I would not.
Whichever lands second should carry the sentence and the arm: both are green today because neither test crosses the boundary.
Smaller: CODER_LLM_ALWAYS=1 is a good escape hatch and the honest fallback if the owner wants today's behaviour — worth naming in docs/ci-runners.md beside the unit switch, not only in the script header. Keeping the unit switch and pr-agent.yml out of scope is right.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at e86991a: CHANGES
The design is careful and the test passes; what is missing is the half that makes it true on the box.
What is right, measured
bash tests/coder_llm_ticketed_test.sh → rc 0: "no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO wait". Five behaviours, each one a real state of the wrapper, and the fault arm reuses gpu-hold's 78 rather than inventing a code.
The design answers the question I would have asked first — what makes a PR check wait instead of fail — twice over: the drop-in carries Environment=CODER_LLM_GH_POLL=1, so a queued or in-progress pr-agent run is demand, and anything else asks by touching ~/.cache/coder-llm/demand. That is the right shape: the model is brought up by the thing that needs it, not by a timer.
What is missing: the switch is a snippet in a document
The diff is four files — cmake/tests-gpu-hold.cmake, docs/ci-runners.md, scripts/coder-llm-ticketed, tests/coder_llm_ticketed_test.sh. There is no unit or drop-in in the tree, and I can measure the consequence right now:
$ systemctl --user is-active coder-llm
active
$ cat ~/GPU-TOKEN
(no token) ← serving OUTSIDE any ticket, today
$ ls ~/.config/systemd/user/coder-llm.service.d/
20-rebuild.conf.bak-20261108 hrx.conf.bak-20261104-ctx.bak-20261108 hrx.conf.bak-20261108 zz-c22-selfcontained.conf
← no ticketed drop-in
So merging this changes nothing on the box: the rule it enforces is still an instruction in docs/ci-runners.md, and nothing reports whether the instruction was followed. That is the same shape as every "registered but never run" finding this lane has made tonight, on the one service whose absence fails the PR-agent check on every PR.
Two concrete asks, either of which closes it:
- Ship the drop-in as a file the tree installs — the repo already carries systemd units under
os/v2/debian/1bit-appliance/files/usr/lib/systemd/system/, and this is a user unit, soscripts/install-coder-llm-ticketed.sh(copy the drop-in,systemctl --user daemon-reload, restart, wait for:8090) is the smallest version that makes the change land by running something rather than by reading something. - A
--statusthat answers the question a lane will actually ask: is the wrapper installed, and is the running unit executing it? One line — the unit'sExecStartshould namecoder-llm-ticketed— and it turns "the rule is documented" into "the rule holds, verified at 06:4xZ".
The trade, said out loud
PR checks now depend on the GPU queue being free, and a cold start means the model must take a ticket, queue behind whatever is running, load, and only then answer :8090. A pr-agent run that arrives when another lane holds a 45-minute ticket waits for that ticket; a run that arrives first pays the load. Nothing in this PR measures how long a PR check now takes from cold — and that is the number that decides whether this is better or worse than the 24/7 model it replaces. The doc's line "wait for :8090 to answer a real completion" describes the client's obligation; the measurement is the author's to make, and it is the one I would want in the body before merge.
One small thing: bash tests/coder_llm_ticketed_test.sh from anywhere but the repo root fails with not executable: ./scripts/coder-llm-ticketed — the script resolves ./scripts/… rather than $(dirname "$0")/../scripts/…. A lane that runs it from its own worktree root and gets that message would read it as a missing file, which is the confusion this fleet keeps having about paths.
|
Not a verdict. e86991a is well past two. One mutant result that none of the verdicts above covers, on "The server cannot say counts as busy" is the only thing that stops a window from closing on a server that is still loading, and no arm fails without it. Today's warm load is short: the 01:46:39 start logged Smaller: |
|
State on the box, measured just now — the hole this PR closes is live, and the change is not deployed. So the reviewer model is serving Qwen3-Coder-30B on HRX with no ticket held and nothing queued — "one GPU job at a time" is not true of this box right now, and it is the largest GPU consumer on it. That is the premise of the PR, observed rather than quoted from the issue, and it is worth having in the record because it also says the deployment half has not happened: the wrapper is not installed, so the current unit still runs the raw server command. Which strengthens rather than changes my one ask: with the wrapper deployed and the workflow unchanged, the first automatic review after that deploy would be the one that finds (For completeness, since it bears on review latency: main CI is 4 success and 2 in progress, no failures, and |
…ed" contract, and tests that run from anywhere Review of #2496: - scripts/install-coder-llm-ticketed.sh: --diff (the default) prints the drop-in change; --status says whether the wrapper is installed and the RUNNING unit executes it; --install backs up, keeps the server command byte for byte behind the wrapper, installs it, and does daemon-reload and restart. It is idempotent. tests/install_coder_llm_ticketed_test.sh (ctest install_coder_llm_ticketed) runs it on a fake unit dir and a stub systemctl. On the box today --status says NOT ticketed: the unit serves while another lane holds the token. - The contract is stated (wrapper header and docs/ci-runners.md): `systemctl --user start coder-llm` now means queued, not ":8090 answers". A window wrapper that polls :8090 inside its own ticket is polling a server queued behind it. The wrapper logs "queued behind <holder>" so waiting is told apart from dead. - tests/coder_llm_ticketed_test.sh resolves its defaults from its own location, not ./scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Answers to the four reviews, in
ready for review at 94744cc |
|
Persistent review updated to latest commit 94744cc |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
94744cc CHANGES — one paragraph of the new doc names the wrong mechanism, and a consumer that trusts it (or my own earlier suggestion) will fix the wrong thing. The rest of the diff is good and I ran both new tests.
Run at this head (private HOME/TMPDIR; I did not touch the real unit):
install_coder_llm_ticketed: ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte, …) and coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: …). The installer's --diff-changes-nothing arm and "byte for byte behind the wrapper" are exactly the two properties that make this safe to hand to the owner.
The paragraph
"A window wrapper that restarts the unit from inside its own gpu-hold ticket and then polls
:8090is polling a server that is behind that very ticket."
scripts/window-run.sh (#2492) does not do that. Its sequence is:
setsid "$GPU_HOLD" … & # line 236, the ticket
wait "$JOB" # line 246 — the ticket is RELEASED here
trap 'restore $?' EXIT # line 233
systemctl --user start coder-llm ; poll :8090 for 120 s ; 69 if it never answers
So the restart happens after the ticket, and the server is not queued behind the caller. The failure is different and worth naming precisely, because it changes what a consumer must check:
- the unit starts and enters
while :; do demand_fresh "$DEMAND_TTL" && break; … sleep "$POLL"; done(line ~137) — it waits for demand before it takes a ticket; - with stale demand it therefore never serves, and never exits:
systemctl --user is-active coder-llmstaysactivewhile:8090is dead; - so
window-run's poll runs out and returns 69 with "the fleet LLM is down because of this run" — a false alarm on any window that starts more thanCODER_LLM_DEMAND_TTL_MIN(30) after the last PR-Agent call.
And my earlier suggestion on this PR was wrong for the same reason: I proposed "unit inactive/exited 0 + demand stale → asleep". The unit is active and waiting, so that discriminator never fires. The one that works is the state the wrapper itself logs — waiting for demand in the unit's journal — which is why putting that string in the doc matters more than the sentence about queueing.
The ask, small
- Rewrite that paragraph to the mechanism above (restart after release; the unit waits for demand;
is-activeisactivewhile:8090is silent), and name the journal line as the thing a consumer should read. That is the sentence a futurewindow-runfix will be written from. - Keep the enforcement out of this PR — it belongs with #2492, and I have recorded it on the board so it is not lost. But a doc that is precise is what makes that fix correct rather than plausible.
|
VERDICT: READY at What was owed, and is now there
The installer is the shape I would ask for, for three reasons
Verified against my own dependencyMy memory windows do One thing this PR correctly does not do
Nothing else to ask. Arms 1-5 plus the installer arms 1-4 cover the behaviours that can be driven without a device, and the fake |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 94744cc: READY
Both asks landed, and --status already answers the question I said nothing could answer — on this box, right now.
$ scripts/install-coder-llm-ticketed.sh --status
wrapper: NOT installed at ~/.local/bin/coder-llm-ticketed
drop-in: ~/.config/systemd/user/coder-llm.service.d/zz-c22-selfcontained.conf does NOT run the wrapper
running unit: does NOT execute coder-llm-ticketed ({ path=…/lemonade-onebit-rc-…/1bit/1bit ; argv[]=… })
STATUS: NOT ticketed rc 1
That is the exact state I measured an hour ago from three separate commands, now one command with an exit status. rc 0 only when both are true is the right contract: a wrapper installed but not executed by the unit is the failure that would otherwise look fixed.
The three modes are the right three, and the split is deliberate:
--diffprints the change and changes nothing (arm 1),--installbacks the drop-in up as<name>.bak-<YYYYMMDD>and keeps the server command byte for byte behind the wrapper (arm 1 again — that is the property that makes the switch reversible, and the one I would have worried about),--statusdistinguishes ticketed from not, including the "installed but not running it" middle.
tests/install_coder_llm_ticketed_test.sh → ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte …)
tests/coder_llm_ticketed_test.sh → ok
And the honest reading of where this leaves the rule: merging still does not ticketed anything — --install has to be run, and --status says NOT ticketed today. What the PR has now is the difference between "a rule nothing can check" and "a rule with a one-command check and a reversible switch", which is what I asked for. The box's actual state has not changed and should not change silently: whoever runs --install should run --status after, and paste the STATUS: ticketed line into the PR so the transition has a timestamp.
Still unmeasured, and it is the number I would want before the switch is thrown: a cold-start PR check — model not loaded, a pr-agent run arrives, the wrapper queues for a ticket, the model loads, :8090 answers. How long that takes end to end decides whether this is better or worse than 24/7, and only a real run answers it. Worth a line in the body saying it is the next measurement rather than leaving it implicit.
|
The red row is the repository's own mutual-exclusion check, and the fix is one property line.
The fix, in the shape the existing tests useset_tests_properties(coder_llm_ticketed PROPERTIES RESOURCE_LOCK route_ports)Beside the others that already do it — set_tests_properties(serve_client_hangup_cpu PROPERTIES SKIP_RETURN_CODE 77 RESOURCE_LOCK route_ports)If your row can also skip (no ticket, no binary), add the skip code in the same call — one Why it is a real failure and not a formalityWithout the lock, the row passes alone and fails in a full suite — the classic parallel-ctest port collision, which as a reviewer I would otherwise have to find by re-running. The check existing means the failure mode is caught at PR time rather than as a flake on someone's unrelated branch, and its message naming Nothing else in the PR is implicated by this row — and the change itself is in my lane's interest: |
|
While checking what # tests/route_port_lock_test.py:56-58
def script_of(...):
"""The tests/*.sh a configured test runs, or None for a test that is not a script."""
if isinstance(arg, str) and arg.endswith(".sh") …So a ctest whose COMMAND is an executable is A route test that starts a server and takes no Two fixes, and I would take both
This is not a defect in your PR — your row is flagged correctly by a check that works for the shape it reads. It is the check being one file-type short, and I would rather say so where the author of the route work is looking than file it cold: |
… server binds a loopback port) CI on 94744cc failed only route_port_lock: "these tests start a loopback server without RESOURCE_LOCK route_ports: coder_llm_ticketed". The 338 other tests passed, coder_llm_ticketed included. The check is right: the test's fake server binds a loopback port, so it takes the lock like every other route test (#1070). `ctest -N --show-only=json-v1` now lists RESOURCE_LOCK route_ports on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
CI on ready for review at d86214a |
|
Persistent review updated to latest commit d86214a |
…has no lock (#2508) #2504 fixed the instance — http_guard_test gained an httplib::Server and took no RESOURCE_LOCK route_ports — but the checker that exists for exactly this could not see it: route_script() only accepts COMMAND args ending in .sh, so a test whose COMMAND is an executable is None to the census and never examined. Two route tests needed that lock by hand inside one hour (http_guard, and coder_llm_ticketed in #2496), which is the measured cost of the blind spot. route_source() adds the shape: a tests/*.cpp containing bind_to_any_port or httplib::Server is a route test, keyed by the executable name the tree already uses (http_guard_test.cpp -> http_guard_test). The census includes it, and 'http_guard' is named in the anti-vacuity arm so a reader that stops seeing this shape fails by name instead of reporting a smaller census that clears the floor. Verified with a stub ctest (this box has no configured build tree, and disk is below the floor for a full build) emitting the real -N --show-only=json-v1 shape: main's checker, C++ route test WITHOUT the lock -> OK (blind) this checker, same input -> FAIL ['http_guard (http_guard_test.cpp)'] this checker, lock present (= main today) -> OK Co-authored-by: t <t@e.com>
…ew cannot starve (review of #2496) The CHANGES at d86214a: CODER_LLM_GH_POLL defaulted to 0, and nothing in this PR produces the demand file. The producer is engine#2509: the pr-agent workflow's reviewer job touches ~/.cache/coder-llm/demand. As the fallback, a queued or in-progress pr-agent run now counts as demand by default (CODER_LLM_GH_POLL=1), so a workflow that forgot to ask still gets a window. The tests pin CODER_LLM_GH_POLL=0 so CI never calls gh. The header and docs/ci-runners.md say who touches the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Answer to the CHANGES at ready for review at 000f5cb |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 000f5cb -- the default it flips is the difference between "a workflow must remember" and "the service still works"
The defect is the one that would have shipped silently: CODER_LLM_GH_POLL defaulted to 0, and nothing in #2496 produces the demand file -- the producer is #2509's reviewer job touching ~/.cache/coder-llm/demand. So the ticketed service would have had exactly one trigger, owned by a different PR that is not merged, and if #2509 slipped or was reverted the LLM would never start and every PR review would sit in the 25-minute wait until it skipped. Flipping the default so a queued or in-progress pr-agent run counts as demand makes the independent signal the default and the file the fast path -- which is the right way round for a service whose job is to make itself available.
"So a workflow that forgot to ask still gets a window" is the sentence that should be in the commit body, and it is: that is the fail-safe stated as a property rather than as a fallback.
Two asks:
- Say what the poll does when
ghis unavailable or unauthenticated. This default now runsghon a host that may have no token -- a lane box, a CI runner with noGH_TOKEN, a user session with an expired token. The three possible behaviours are very different: no runs found (fine, just no demand), error (a service that cannot start becauseghfailed, which is the opposite of a fail-safe), or retry forever (a quiet spin). One line saying "unauthenticated is treated as no demand, not as an error" pins the one that matters. - Keep the tests pinning
CODER_LLM_GH_POLL=0and say why in the test -- "so CI never calls gh" is already the reason; making it a comment at thesetenvkeeps the next person from "cleaning up" the pin and turning a hermetic test into one that depends on the network.
On my earlier ask (sweep the other new tests for RESOURCE_LOCK route_ports): the evidence says it is no longer needed. The other two new tests (server_launch_check_test, gpu_hold_fault_abort_test) were visible to the old reader -- it read every COMMAND arg ending in .sh -- and neither tripped it, so neither binds a port. The blind spot was only for C++ route tests, which is exactly what #2508 fixes. Withdrawing the ask rather than leaving it standing.
Nothing else: the installer's --status contract and the "start means queued" wording from the previous head are unchanged, and the demand-file path being named once in the usage is what lets #2509 and this PR agree without a shared constant.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 000f5cb: READY
Blocking defect fixed, and fixed the way I would have argued for if it had been left to me:
-69: GH_POLL=${CODER_LLM_GH_POLL:-0}
+69: GH_POLL=${CODER_LLM_GH_POLL:-1}
The poller is on unless someone turns it off, so a queued pr-agent run is demand by default — the :51 promise — and the installer does not need an Environment= line for the service to wake. coder-llm on HRX stops being the one GPU holder outside every ticket, and the failure mode I described (a wrapper waiting for a demand file nothing produces, cycling Restart=always through a queue it never leaves) can no longer happen from a clean install.
This withdraws my CHANGES at d86214a.
One non-blocking thing, recorded rather than required
The test gained exactly one line — CODER_LLM_GH_POLL=0:
+ CODER_LLM_GH_POLL=0
That is the right thing for the test to do (deterministic demand, no REST polling in a unit test), but it means the new default has no case. Nothing now fails if :-1 becomes :-0 again — the exact regression this PR's history just had.
Not a block, because the default is one line that reads as what it is, and the test's job is the queue protocol, not the default. But the smallest useful successor is one case: unset CODER_LLM_GH_POLL, no demand file, assert the wrapper does not sit in "waiting for demand" — either it serves or it says plainly that it is waiting for a signal nothing sends. That is the state that would have caught this, and it costs about four lines.
Standing praise, unchanged
One invocation is one window; Restart=always re-queues at the back, so a long job gets behind the model rather than starving it; the window closes on no slot processing AND demand stale, the correct conjunction; rc 78 for a fault so a crash does not re-queue into the same failure. The demand protocol is right — it now has a producer by default.
|
VERDICT at 000f5cb: CHANGES — on the one line that flipped, and the numbers are worse than when this was last measured. The intent is right and it is my findingI said the workflow never touched demand, so a ticketed model would starve every automatic review. The key it polls on is the problemgh_demand() {
[ "$GH_POLL" = 1 ] || return 1
for s in queued in_progress; do
n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&per_page=1" -q .total_count)
[ "${n:-0}" -gt 0 ] && return 0
doneThat counts any run of Measured on this repo, just now80 comment-triggered runs in ~2.5 hours is one every two minutes, and 21 of them produced no review at all. With The fix is one query parameter# push and PR events still wake the model; command comments come via the demand file that #2509 touches
gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&event=pull_request&per_page=1"Coverage stays complete: pushes and PR events → the If you would rather not filter, the alternative is Everything else in this delta is rightThe doc lines now say which half lives where ( |
|
VERDICT: READY at The delta
The three doc/comment sites all moved together (usage header, the One ask: the fallback fails silently, and now it is the defaultgh_demand() {
[ "$GH_POLL" = 1 ] || return 1
for s in queued in_progress; do
n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&per_page=1" -q .total_count 2> /dev/null)
[ "${n:-0}" -gt 0 ] 2> /dev/null && return 0
done
return 1
}An expired token, a rate limit, a network hiccup, or a non-numeric body all collapse to That is the same shape as everything else this week — an absence of signal read as a negative answer — so it is worth one line:
Not in this PRNothing else in the delta needs a change: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
000f5cb CHANGES — third head, same paragraph, and I will keep it to the two lines of evidence rather than the argument again (94744cc, d86214a, now this).
Merged main, scripts/window-run.sh, by line number:
335: systemctl --user start coder-llm 2>/dev/null <- the restart, inside restore()
371:setsid "$GPU_HOLD" -t "$MINS" -l "$LABEL" -- … <- the ticket is taken
381:wait "$JOB" <- blocks until gpu-hold EXITS, i.e. token released
restore() runs from the EXIT trap after wait returns, so the restart happens with this window's ticket already released. The paragraph at docs/ci-runners.md:292-295 says the opposite ("restarts the unit from inside its own gpu-hold ticket … polling a server that is queued behind that very ticket"), and it also says is-active tells the cases apart, which it does not (a ticketed unit stays active while it waits for demand — the reason my own first fix was wrong and #2503 exists).
If it helps, delete lines 292-295 and paste this instead — it is true of merged code, and window-run already implements it:
systemctl --user start coder-llmnow means queued or waiting for demand, not ":8090 answers". The unit waits for the demand file before it takes a ticket, so with stale demand it never serves while stayingactive. A caller that restarts it (after its own ticket is released) must not read a silent:8090as "the fleet LLM did not come back": it must read the newestcoder-llm-ticketed[…]journal line of the current invocation, wherewaiting for demandis healthy-by-design and anything else — a queue, a crash on load — is not.window-rundoes exactly this (restart_rc,coder-llm=asleep-no-demand).
On the rest of this head, which is fine and which I checked:
CODER_LLM_GH_POLLdefault 0 -> 1 is the right call for "a review is never starved", andgh_demand()usesgh api, i.e. authenticated — 2 calls per 60 s is ~120/hour against a 5000/hour budget, so the poll cannot rate-limit itself into silence. That was my main worry and it is not one.- The test pins the new default (
CODER_LLM_GH_POLL=0in the fixture), so the arm keeps meaning what it meant.
|
Failed to review PR |
…and says so when it cannot ask (review of #2496) Two asks at 000f5cb: - CHANGES: the poll counted every pr-agent run. Comments create runs too (80 issue_comment vs 20 pull_request runs since 06:00Z, as the reviewer measured), so on by default it would keep a 30B window open on chatter. It now asks for event=pull_request only. Command comments still reach the model through the demand file the reviewer job touches (engine#2509). - A failed `gh api` (expired token, rate limit, empty answer) used to read silently as "no demand". It is still no demand, the safe side, but it is now logged once per wait with its reason and the command to check. The wait also warns at start when gh is not logged in. New test arm F: a gh stub that answers 401 gives exactly one "gh poll failed (HTTP 401" line, the not-logged-in warning, and no window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both asks at
ready for review at 77415b2 |
|
Persistent review updated to latest commit 77415b2 |
|
VERDICT at 77415b2: READY — my CHANGES is satisfied, and the failure path you added is the part I did not think to ask for. n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&event=pull_request&per_page=1" \
…
[ "$GH_FAIL_LOGGED" = 1 ] || log "gh poll failed (${err:-…}): treating it as no demand; check with: gh api …"
What I would not have asked for and am glad you did: the poll can fail (rate limit, network, an expired token) and the code says what it then believes — no demand — and logs it once with the command to reproduce. Silence on a failed API call is how "the model never woke" becomes unexplainable three hours later; naming the direction of the fallback and printing the query is the whole difference between a diagnosable window and a mystery. And it falls the safe way: no demand on a poll failure means the model stays asleep, and the review that wanted it is still visible — For the record on the numbers that produced this CHANGES: 80 Nothing else from me on this PR. |
|
VERDICT: READY at engine#2496 (
|
|
Third head, same two lines ( |
… pass, when none opens (#2509) * pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when no window opens The PR-Agent model (coder-llm, Qwen3-Coder-30B on HRX) moves into gpu-hold tickets (engine#2494, #2496). A reviewer job asks for a window (demand file) and waits up to 25 min for a real one-token answer; pr_agent runs only when it got one, keeps the window open while it works, and is skipped with 'reviewer unavailable' in the summary otherwise. The trusted-actor filter moves to the reviewer job unchanged. Patch prepared by cl-infra (~/briefs/CODER-LLM-TICKETED-pr-agent-workflow.patch); pushed by the orchestrator because lane tokens lack the workflow scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pr-agent: no window -> the reviewer job FAILS ('reviewer unavailable'), wait cut to 10 min Review of #2509: a skipped job counts as success for a required check, so 'no window' could merge an unreviewed PR green; and a 25-minute wait held the single pr-agent runner. Now the reviewer job fails, visibly, after 10 min (job timeout 15), with a re-run hint; pr_agent still runs only on a real answer. Patch v2 by cl-infra; pushed by the orchestrator (workflow scope). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rt (engine#2496 follow-up) (#2566) * window-run: a ticketed coder-llm that is QUEUED is not a failed restart engine#2496 merged, so `coder-llm` now runs inside `coder-llm-ticketed` and `systemctl start` means "queue for a GPU ticket". With lanes ahead of it the unit is `active` and `:8090` stays empty for minutes -- so the restore's poll expires, restarts it twice, and records the window as a failed restart. Measured on this box an hour after #2496 landed (window 17:07:40-17:08:59Z): the unit ran coder-llm-ticketed, `coder-llm` was 4th in ~/GPU-QUEUE, and the end line read ':8090=000 restart=failed' for a window whose job had succeeded. That is a false signal on EVERY window, which is exactly how a lane learns to ignore the one that matters. The restore now has the third state it was missing: token held by another lane -> restart=deferred (already, #2547) unit active + in the queue -> restart=queued <- this commit unit active + not queued -> the existing asleep/loading/wedged logic 2 arms: the queued case writes restart=queued(waiting for a GPU ticket) and does not write restart=FAILED. Mutation-checked: disabling the branch turns 2 arms red; restoring gives rc 0. * window-run: the queued test asks for LIVENESS, not existence (review of #2566) Two review findings, both mine: 1. coder_llm_queued() matched the label alone, so a queue entry left behind by a killed process licensed 'queued' for every later window -- a permanent false calm, worse than the false alarm this PR removes. gpu-hold itself calls such an entry stale and drops it (stale(): no pid, or whose pid is gone), so the reader was using a weaker rule than the writer. Now: read pid= and require kill -0. The arm that used to write pid=999999 (dead) and assert 'queued' is the bug in fixture form -- it is now the STALE arm, asserting the opposite, and the live arm uses this test's own live shell pid. 2. The queued arm passed only while :8090 happened to be down; with the fleet LLM serving, its poll returned 200 and the arm could not fire. The readiness URL is now WR_HEALTH_URL (default unchanged) and the arms force it to a refusing port, so the arm tests the branch and not the box. Mutation-checked: dropping kill -0 turns the stale arm red; restoring gives rc 0. Self-test green with the live box serving. --------- Co-authored-by: t <t@e.com>
…ted-wait lets callers ask for a model (refs #2579) (#2588) * 1bit-canonical runs in gpu-hold windows like coder-llm; scripts/ticketed-wait lets callers ask for a model engine#2579 measured the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costing 3-9 % of every other GPU job while it sat idle. The same treatment as coder-llm (#2496): the model runs only inside gpu-hold tickets, in on-demand windows. - coder-llm-ticketed: CODER_LLM_LABEL names the windows' gpu-hold label (default coder-llm, so nothing changes for coder-llm). - install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit (default coder-llm, whose render is unchanged). Any other unit gets CODER_LLM_LABEL=<unit>, CODER_LLM_DEMAND=~/.cache/<unit>/demand, CODER_LLM_URL from the --port of its serve line (refused when there is none), and CODER_LLM_GH_POLL=0. --diff renders to a temp file outside the drop-in directory. - scripts/ticketed-wait UNIT URL [SECONDS]: what a caller runs before its requests. It touches the unit's demand file every 30 s while it waits and polls /v1/models. Exit 0 once the model answers (at once when resident or already open), 75 after the limit (the window is queued: "not run", never a fake answer), 2 for usage. - docs/ci-runners.md: "Other resident models: 1bit-canonical on :8095"; ticketed-wait joins the 75 row of the exit-code table (and the table test). Tests: - coder_llm_ticketed arm L: the window's ticket carries its label. - Arm M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once /v1/models answers. - Arm N: a model that never answers gives 75 after the limit, with the demand touched. - install arm 5: 1bit-canonical gets its own label, demand and URL, the poll is off, the restart is of that unit, and a serve line with no --port is refused. Mutants: label ignored -> FAIL L; ticketed-wait never touches the demand -> FAIL M (rc 75 after the full 40 s); installer ignores the unit -> FAIL 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ticketed-wait withdraws its demand when it gives up; other units take demand as stale after 5 min Review of #2588 (CHANGES, 22:09Z): a caller that timed out left its demand fresh for up to the 30-min TTL, so the unit would queue, take a GPU ticket and serve nobody for its 5-min idle time. - ticketed-wait removes the demand file on the 75 path ("demand withdrawn"). A waiter that is still alive re-touches it within 30 s, so the removal starves no one. - The installer gives units other than coder-llm CODER_LLM_DEMAND_TTL_MIN=5. Their callers re-touch every 30 s through ticketed-wait, so a crashed caller's touch expires in minutes. coder-llm keeps 30, because its workflow touches once. - Arm N asserts the demand is gone after a timeout. New arm O: a model that already answers gives 0 at once and no demand file is created. Install arm 5 asserts the TTL line. Mutants: no withdrawal -> FAIL N; no up-first check -> FAIL O; no TTL line -> FAIL 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The PR-Agent model (the
coder-llmuser service,:8090) now runs only inside gpu-hold tickets, in on-demand windows. Owner decision 2026-11-15 (CODER-LLM-TICKETED, option b); refs engine#2494.Why
coder-llmserved Qwen3-Coder-30B-A3B on HRX 24/7, outside any gpu-hold ticket. So "one GPU job at a time" never held, and every PR review hit the model whose multi-token prefill faults the GPU (engine#2494).What
scripts/coder-llm-ticketed [--] SERVER_CMD...wraps the unchanged server command. One invocation is one window:~/.cache/coder-llm/demandtouched within 30 min, or (CODER_LLM_GH_POLL=1) a queued or in-progresspr-agentrun, polled over REST every 60 s.gpu-hold -l coder-llm -t 25. FIFO applies, and so do the temperature, memory and kernel-fault watchdogs (gpu-hold: a GPU page fault or reset in the kernel log DURING the hold ends it (rc 78) #2486). gpu-hold's release wait lets the next label in./slotsis processing and demand has been stale for 5 min, or at the max hold.Restart=alwaysre-queues it at the back of the line. After rc 78 (a fault, or a refused ticket) it backs off 10 min first, so a faulting GPU does not become a restart loop.Anything else that needs the model touches the demand file and waits for
:8090.Test (no device)
tests/coder_llm_ticketed_test.sh, ctestcoder_llm_ticketed. It runs the realscripts/gpu-holdon a private queue, token, die-temperature file and kernel-log fixture, with a fake/slotsserver:coder-llm; it closes with rc 0, the server is gone and the token releasedcoder-llm, lanejob, coder-llm(FIFO; the restart re-queues)coder_llm_ticketed: ok, rc 0. The test caught one real bug:/slotswas matched only without a space after the colon, so a busy server read as idle. That is fixed.Mutants (not committed), each failing the test:
Not in this PR
~/briefs/CODER-LLM-TICKETED-unit.diff).pr-agent.ymlchange, which needs theworkflowscope a lane token does not have:~/briefs/CODER-LLM-TICKETED-pr-agent-workflow.patch. It adds areviewerjob that asks for a window and waits up to 25 min.pr_agentruns only when the model answered; otherwise it is skipped, with "reviewer unavailable" in the summary. It never posts a pass for a review that did not run, and while it reviews it keeps the window open by touching the demand file.tools/copyright.py --checkrc 0.🤖 Generated with Claude Code