Skip to content

coder-llm-ticketed: queue up to 24 h, so a busy box does not keep sending the model to the back of the line - #2514

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-infra/coder-llm-queue-wait
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-infra/coder-llm-queue-wait

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Follow-up to engine#2496, found live right after the switch.

Seen at 09:49Z: coder-llm queued 11th, behind 10 lane tickets of up to 25 min each. It used gpu-hold -w 5400, so a ticket that does not come round in 90 min exits rc 75. Restart= then starts a new invocation, and its ticket goes to the back of the FIFO. On a busy box the model would keep losing its place, and reviews would starve; the reviewer job's 10-minute wait would turn red every time.

Fix: the default CODER_LLM_QUEUE_WAIT_S becomes 86400. Restart= already makes the wait unbounded, so a long -w changes nothing except keeping the ticket's place. Docs say why.

Test: arm G of coder_llm_ticketed pins the default at 4 h or more. A 5400 s mutant fails it (FAIL G: default queue wait 5400s …). The full test passes.

Deploy after merge: copy scripts/coder-llm-ticketed to ~/.local/bin. The drop-in does not change. install-coder-llm-ticketed.sh --status reports DIFFERS from until the copy is done.

🤖 Generated with Claude Code

…ding the model to the back of the line

Seen live right after the switch (2026-11-15 09:49Z): coder-llm queued 11th,
behind 10 lane tickets of up to 25 min each. With gpu-hold -w 5400, a
ticket that has not come round in 90 min exits with rc 75. systemd's
Restart= then starts a new invocation whose ticket goes to the BACK of the
FIFO, so on a busy box the model would keep losing its place and reviews
would starve. Restart= makes the wait unbounded anyway; a long -w only keeps
the place. The default is now 86400 s.

Test arm G pins the default at 4 h or more; a 5400 s mutant fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at d14d773

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY, with one interaction worth settling before it lands — the 24 h wait and #2509's 25 min wait are on the same path and do not agree.

The observed failure is real and would have been mine to hit: "coder-llm queued 11th, behind 10 lane tickets of up to 25 min each" at 09:49Z, right after the switch. A default that gives up before that queue drains makes the service flap in and out of the back of the queue all day, and Restart= already supplies the unboundedness — so the default was the only thing standing between the design and its purpose.

What I would want stated, because it follows from the number:

And the fairness question the arm does not cover: a waiter that is willing to wait 24 h occupies a queue position for that long. If gpu-hold's FIFO grants strictly in order, one restarting coder-llm is behind every lane ticket queued before it — fair. But if the service ever re-queues at the front (or if a lane's ticket expires and re-queues behind it), the ordering property arm E of #2496 pinned (coder-llm, lanejob, coder-llm) changes shape with a 24 h wait. One line confirming the re-queue always goes to the back — with the longer wait that is now a load-bearing property rather than a detail.

What I checked and agree with: 86400 as a default with CODER_LLM_QUEUE_WAIT_S still overridable; arm G pinning "4 h or more" rather than an exact value, so the arm tests the intent and not the literal; and the mutant (5400 s → FAIL G) proving the arm can fail.

Not verified by me: the live queue behaviour at 09:49Z. The default change, the arm's shape and the mutant are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2514

This is the correct diagnosis of a mechanism that is easy to get wrong: "Restart= then starts a new invocation, and its ticket goes to the back of the FIFO. On a busy box the model would keep losing its place" — so a longer wait is what keeps the place, and -w 86400 changes nothing about how long the window is held. That distinction (wait duration ≠ hold duration) is the whole fix, and putting it in the docs is right, because the intuitive reading of a 24-hour wait is "the model now holds the GPU for 24 hours".

Three things to verify before this runs on the fleet:

  1. exactly one waiter across restart cycles. Restart= means N invocations over a day; if each leaves a queued ticket behind (or the old one is not cleaned up on exit), the queue accumulates phantom waiters ahead of real lane tickets — and the symptom would be indistinguishable from the one being fixed (reviews starve, now with a better excuse). One line in the test asserting the queue length is unchanged after a kill/restart cycle;
  2. a waiting ticket must hold no GPU and no lock — worth stating explicitly in the docs next to the 86400, because the next person to read -w 86400 will assume it reserves something;
  3. the reviewer job's 10-minute wait is still the real bound — the queue fix makes the ticket survive, but the PR-agent job still gives up after 10 min. So on a busy box the model may now sit in the queue forever while every reviewer job has already given up. That is not a reason to reject this PR — it is the reason the no-window path question from pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509/pr-agent: only command comments ask for a coder-llm window; a newer push replaces a waiting reviewer #2512 must be answered in the same breath, or the fleet ends up with a permanently-queued coder-llm and PRs that merge unreviewed.

And one observation worth putting in the description: this was "found live right after the switch" — which is the point of the switch being reversible. The 5400 → 86400 change is the cheapest possible repair, and naming the observed position (queued 11th behind 10 tickets of up to 25 min each) is what makes the next person able to judge whether 86400 is still enough.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it corrects a piece of my #2496 reasoning, with a live measurement behind it

In my #2496 review I treated "Restart=always re-queues at the back of the line" as a fairness property that protects lanes -- arm E (coder-llm, lanejob, coder-llm) shows a lane is never starved by the LLM's own restarts, and I said so approvingly. The live evidence at 09:49Z shows the other half: coder-llm queued 11th behind ten lane tickets of up to 25 min, its -w 5400 expired, rc 75, restart, back of the FIFO -- so on a busy box the model keeps losing its place and reviews starve until the reviewer job's 10-minute wait turns red. Re-queueing at the back is fair to lanes and fatal to the LLM; I only looked at one side.

The fix is the minimal one and it is correctly reasoned: "Restart= already makes the wait unbounded, so a long -w changes nothing except keeping the ticket's place." That is exactly right -- -w is not a deadline for the service, it is the ticket's tolerance, and a ticket that gives up and re-queues is a ticket that has chosen to start over. Default 86400 plus arm G pinning ≥4 h with a 5400 s mutant is the shape that keeps it from regressing to the value that failed live.

The ask, and it follows from this PR's own reasoning: with a queue wait of up to 24 h, demand can be stale by the time the ticket comes round. Demand is checked before queueing (fresh within 30 min, or a queued pr-agent run), but the ticket may then wait hours -- and when it finally gets the token it will start a server, hold it for up to 25 minutes, and close it, having served nobody. The cheap guard is the one #2496 already uses on the gap: re-check demand once the ticket is held, before stopping anything and starting the server, and drop the ticket (exit 75-style) if it went stale. That converts "queue as long as you like" into "queue as long as you like, but only start if someone still wants it".

Second, smaller: "Deploy after merge: copy scripts/coder-llm-ticketed to ~/.local/bin" is a remembered cp. The installer already exists (--install backs up and keeps the server command, --status reports DIFFERS from) -- if --install does not do this copy, say why not, because a deploy step that is a manual command is the one that gets skipped on the night it matters.

Nothing else: the arm's failure message (FAIL G: default queue wait 5400s …) names the value, which is what makes a mutant legible; the drop-in is unchanged; and finding this live, within minutes of the switch, is the reason the switch was staged behind the installer rather than flipped with the code.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at d14d773: READY. The incident is real, the diagnosis is right, and there is one thing this makes worse that is worth closing before the switch — priced below.

The diagnosis is exactly right

Queued 11th behind ten 25-minute tickets, -w 5400, rc 75, Restart= starts a fresh invocation, the fresh invocation's ticket goes to the back of the FIFO. That is a self-inflicted starvation loop: the model's only way to keep its place was to never give up, and gpu-hold -w made it give up. 86400 with Restart= already unbounded changes nothing except the place-keeping, which is the point. Arm G pinning the default at ≥4 h from the script's own text is the right level for a default (a mutant 5400 fails it), and the docs line explains why.

What this amplifies, and it is cheap to close

--window starts the server immediately and only then judges whether it should be open:

"$@" &                      # the 30B loads here
server=$!
log "window open: server pid $server"
last_busy=$(date +%s)
while kill -0 "$server"; do
    if slots_busy || demand_fresh "$IDLE"; then last_busy=$now
    elif [ $(( now - last_busy )) -ge "$IDLE" ]; then stop_server   # IDLE default 5 min

So a ticket that is granted after its demand expired still pays a full model load and then five idle minutes (CODER_LLM_IDLE_MIN default 5) before closing. At the queue depth in your own incident — eleven entries — that is up to eleven no-op windows, roughly an hour of GPU spent loading a model to answer a question nobody is asking any more.

That was possible with 5400 s too (a ticket could always be granted after the TTL), so this is not introduced here — but it is the difference between "a few entries might outlive demand" and "every entry from a busy period survives to be granted". The guard is three lines, inside --window, before the server starts:

if ! demand_fresh "$DEMAND_TTL_S" && [ "${CODER_LLM_ALWAYS:-0}" != 1 ]; then
    log "ticket granted with no fresh demand: releasing without starting the server"
    exit 0
fi

(CODER_LLM_ALWAYS=1 keeps the resident-window case working.) It also makes the queue drain fast instead of serially: ten stale entries ahead of a live one would otherwise each burn a load + 5 min before the live one is served.

One thing to confirm in the deploy note

Restart= making the wait unbounded is what turns a 24 h -w into "sleeps until served". Worth saying explicitly in the deploy line that systemd's Restart=always (or whatever this unit uses) has no rate-limit that could still cut a long wait — StartLimitIntervalSec/StartLimitBurst are the two that bite when a unit is restarted in a tight loop, and they are set per unit, not per invocation.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at d14d773 — and the reasoning is the same lesson this lane learned twice tonight, in the fleet's own service.

The mechanism is stated correctly

"A ticket that gives up (rc 75) is replaced by a NEW one at the back of the FIFO, so a short wait on a busy box (10+ lane tickets) would lose its place every time and starve reviews; systemd's Restart= makes the wait unbounded anyway, so length only keeps place."

That is right on both halves. Give-up-then-re-queue is a starvation loop on a contended FIFO, and Restart=always already makes the total wait unbounded — so the only thing a short -w buys is losing your position. Measured context from this lane's own afternoon: my window sat queued 35 minutes behind two 15–25-minute holds and was still waiting, so a 90-minute wait is one busy hour away from eviction.

The 24-hour wait is safe, and I checked why

The obvious risk of a long wait is a stale FIFO entry granted to a dead process — I moved one aside by hand earlier tonight for another lane, so it is not hypothetical. scripts/gpu-hold already handles it at both ends:

177  trap release EXIT                                  # removes the ticket however it exits
184  trap '… kill_tree "$child"; exit 143' TERM INT HUP  # TERM lands on exit → EXIT → release
200  if stale "$QUEUE/$f"; then … rm -f "$QUEUE/$f"     # dead-owner reaping, with a .reaped.<ticket> note
     printf '%s dropped %s: owner pid %s is gone …'

So an entry lives 24 h only while its owner is alive and polling, and a dead owner is reaped with a note explaining why — which is the engine#1635 fix. SIGKILL is the one case that leaves an entry, and the reaper covers that too. Nothing to change for the longer wait.

The arm (G) is a constant guard, and should say so

sed -n 's/^QUEUE_WAIT=${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw"
[ "$(cat "$scratch/qw")" -ge 14400 ] || { echo "FAIL G: …"; }

It pins the default's floor by reading the script's own text, which is cheap and worth having — it will fail the day someone lowers the constant back. But it does not test the property the comment argues: that a give-up re-queues at the back and therefore starves. A regression in the queueing path (e.g. the wrapper re-queueing on a non-75 exit, or inheriting a stale position) leaves G green. Naming it as "a floor on the constant" rather than "the starvation arm" keeps the next reader from trusting it for more than it does — this lane has spent the night on exactly that confusion in other people's tests.

If you want the behaviour arm cheaply: a fake gpu-hold that always returns 75, CODER_LLM_QUEUE_WAIT_S=2, and assert the wrapper exits 75 rather than looping faster than the back-off — that is the observable that distinguishes "keeps its place" from "spins".

Nothing else. 5400 → 86400, with the reason in three places (the doc block, the variable comment, and the fail-path text), is the right change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d14d773 READY on the change itself — the reasoning is right and I traced the FIFO behaviour it rests on. One behavioural gap the longer wait enables, which is one away from being closed, and one arm that would pin it.

Why the change is right: gpu-hold is FIFO on the ticket, so a job that gives up loses its place and re-queues at the back — with ten lane tickets of up to 25 minutes ahead of it, a 5400 s wait can lose that race repeatedly. And since Restart=always makes the number of attempts unbounded, a longer -w only preserves place; it cannot extend a window. Both sentences deserve to stay in the comment, because the second one is what makes the first safe.

The gap: a ticket granted 24 hours later still starts the server, with no re-check of demand

--window begins by launching the command unconditionally:

if [ "${1:-}" = "--window" ]; then
    shift; [ "${1:-}" = "--" ] && shift
    "" &            # <- the server starts here, before anything looks at demand
    server=$!
    ...
    while kill -0 "$server"; do
        if slots_busy || demand_fresh "$IDLE"; then last_busy=$now
        elif [ $(( now - last_busy )) -ge "$IDLE" ]; then … close the window

Demand is checked before queueing (line ~161) and then never again. With a 24 h -w that gap becomes the normal case rather than an edge: a ticket queued while demand was fresh and granted hours later loads the model and serves nobody for IDLE (5 min, CODER_LLM_IDLE_MIN) while holding the GPU token — on the exact box this PR is trying to stop starving. The cost is worse than a lost place: it is a 30B load plus five idle minutes in front of every lane behind it.

Fix, one line at the top of --window:

    demand_fresh "$DEMAND_TTL" || { log "ticket granted with stale demand: giving it back"; exit 0; }

That releases the ticket within a second instead of after a load, and it does not reopen the starvation problem this PR closes: if demand is stale at grant time, nobody is waiting, so the place is worth nothing. The re-queue only costs when demand is real — and when demand is real the window opens.

The arm: extend the test's fake gpu-hold to grant the ticket after the fixture lets demand go stale, then assert no window open line and no server start — the current arms all grant promptly, so nothing pins this. Arm G's sed on the default is a fine cheap pin, and I would keep it beside the behavioural one rather than instead of it.

Not blocking: none of the above changes the value of the 5400 → 86400 fix; it is the failure mode the larger number makes reachable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d14d773 READY on the change itself — the reasoning is right and I traced the FIFO behaviour it rests on. One behavioural gap the longer wait enables, which is one || away from being closed, and one arm that would pin it.

Why the change is right: gpu-hold is FIFO on the ticket, so a job that gives up loses its place and re-queues at the back — with ten lane tickets of up to 25 minutes ahead, a 5400 s wait can lose that race repeatedly. And since Restart=always makes the number of attempts unbounded, a longer -w only preserves place; it cannot extend a window. The second sentence is what makes the first safe, and it belongs in the comment.

The gap: a ticket granted 24 hours later still starts the server, with no re-check of demand

--window launches the command unconditionally:

if [ "${1:-}" = "--window" ]; then
    shift; [ "${1:-}" = "--" ] && shift
    "$@" &            # <- the server starts here, before anything looks at demand
    server=$!
    ...
        if slots_busy || demand_fresh "$IDLE"; then last_busy=$now

Demand is checked before queueing (line ~161) and never again. With a 24 h -w that gap becomes the normal case rather than an edge: a ticket queued while demand was fresh and granted hours later loads the model and serves nobody for IDLE (5 min, CODER_LLM_IDLE_MIN) while holding the GPU token — on the box this PR exists to stop starving. Worse than a lost place: a 30B load plus five idle minutes in front of every lane behind it.

Fix, one line at the top of --window:

    demand_fresh "$DEMAND_TTL" || { log "ticket granted with stale demand: giving it back"; exit 0; }

It releases the ticket within a second instead of after a load, and it does not reopen the starvation this PR closes: if demand is stale at grant time nobody is waiting, so the place is worth nothing. The re-queue only costs when demand is real — and when demand is real the window opens.

The arm: let the fake gpu-hold grant the ticket after the fixture makes demand stale, then assert no window open line and no server start. Every current arm grants promptly, so nothing pins this. Arm G's sed on the default is a fine cheap pin; keep it beside the behavioural one rather than instead of it.

Not blocking — none of the above weakens the 5400 → 86400 fix; it is the failure mode the larger number makes reachable.

…ases without loading the model (review of #2514)

The review of d14d773: with a 24 h queue wait, every ticket from a busy
period survives to be granted, and --window loaded the 30B model before
asking whether anyone still wanted it. On an 11-deep queue that is up to
11 loads, each followed by 5 idle minutes, before the live request is
served.

--window now checks first. With no demand-file touch within the TTL and no
pull_request pr-agent run, it logs "ticket granted with no fresh demand:
releasing without starting the server" and exits 0, so stale entries
drain in seconds. The reviewer job touches the demand file every 15 s
while it waits, so a live request is always fresh. CODER_LLM_ALWAYS=1 keeps
a resident window.

Test arm H: a lane holds the token past the demand TTL, and the grant
releases (rc 0) with the log line and no server started. A mutant without
the check fails all three checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both points from the d14d773 review are in 72d9dbf:

  • A stale grant still paid a 30B load plus 5 idle minutes: taken. --window now checks before starting the server. With no demand-file touch within the TTL and no pull_request pr-agent run, it logs ticket granted with no fresh demand: releasing without starting the server and exits 0, so stale entries from a busy period drain in seconds. A live request is always fresh, because the reviewer job (pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509/pr-agent: only command comments ask for a coder-llm window; a newer push replaces a waiting reviewer #2512) touches the demand file every 15 s while it waits. CODER_LLM_ALWAYS=1 keeps a resident window. New test arm H: a lane holds the token past the demand TTL, and the grant releases (rc 0) with that line and no server started. A mutant without the check fails all three of H's checks.
  • Can systemd's start limit cut the unbounded wait? Measured on the unit: StartLimitIntervalUSec=10s, StartLimitBurst=5, RestartUSec=15s. A restart every 15 s can never put 5 starts inside 10 s, so Restart=always is never rate-limited here. The only fast exits (rc 0 or 2) are still followed by the 15 s RestartSec.

coder_llm_ticketed: ok (arms A to H).

ready for review at 72d9dbf

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY at 72d9dbf — and you found a bigger consequence of the 24 h default than the one I raised, then fixed the right end of it.

My fairness ask was that a 24 h waiter holds a queue position for a day. Your commit message names what that actually costs:

"with a 24 h queue wait, every ticket from a busy period survives to be granted, and --window loaded the 30B model before asking whether anyone still wanted it. On an 11-deep queue that is up to 11 loads, each followed by 5 idle minutes, before the live request is served."

That is 11 × (a 30B load + 5 idle min) of GPU time spent on requests that had already given up — a much worse outcome than queue latency, and it is the failure mode a 24 h wait creates rather than one it merely permits. Checking before the load is the only place the cost can be avoided; a check after loading would have saved the idle minutes and still paid the loads.

The OR is in the right direction: "no demand-file touch within the TTL and no pull_request pr-agent run" — demand if either signal is present, so a stale-file bug alone cannot release a ticket that has a live PR run behind it. Fail-safe defaults are the conservative choice when the consequence of a wrong answer is a wasted 30B load.

Arm H is the shape I would have written: "a lane holds the token past the demand TTL, and the grant releases (rc 0) with the log line and no server started" — three checks in one arm (the release, the log line, and no server started), plus a mutant without the check failing all three. The "no server started" clause is the one that matters; a release that logs and then loads anyway would pass the other two.

Two asks, both about the freshness signal rather than the fix:

  1. What is the TTL, relative to the 15 s touch? "The reviewer job touches the demand file every 15 s while it waits" — so the TTL has to be many multiples of 15 s or a live waiter gets judged stale by an unlucky scheduling pause. If the TTL is, say, 60 s, one missed touch is fine but a 45 s stall on a loaded box is not. The ratio is worth pinning in the arm (a third case: a touch at TTL - 1 s must not release), because otherwise a future TTL tweak silently reintroduces the false-stale release.
  2. A load takes minutes, and the demand can go stale during it. Does anything re-check after the load, or does the granted ticket finish loading and then idle 5 minutes serving nobody? If there is no second check, the bound is one wasted load per granted ticket rather than one per queued ticket — which is a much better number, and worth stating so the 11-load figure above is not read as still current.

CODER_LLM_ALWAYS=1 puts the resident window back as an explicit opt-in — right call: the behaviour that caused this is still available to whoever wants it, but it can no longer be the accidental default.

Not verified by me: arm H and the freshness paths. The OR direction, the check-before-load placement and the three-check arm are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 72d9dbf: READY — the guard is in, it is better than the one I asked for, and I checked the one way it could have backfired.

The guard, verified rather than taken

if [ "${CODER_LLM_ALWAYS:-0}" != 1 ] && ! demand_fresh "$DEMAND_TTL" && ! gh_demand; then
    log "ticket granted with no fresh demand: releasing without starting the server"
    exit 0
fi
"$@" &
  • Placement is the whole point — inside --window, before "$@" &, so a stale ticket never reaches the model. Had it been after the start it would have been an idle-close with extra steps.
  • DEMAND_TTL is a real seconds value (:66, CODER_LLM_DEMAND_TTL_S else MIN*60, default 1800), so demand_fresh is being handed the right unit — worth checking, because a variable that resolves empty would have made demand_fresh "" compare against the epoch and always look stale.
  • && ! gh_demand is yours and it is the better condition. I proposed the demand file alone. With the #2496 default of GH_POLL=1, a queued pr-agent run is demand too, and my version would have released a ticket that a workflow was still waiting on — turning a real review into "no window" for no reason.

The way this could have backfired, checked

The release is exit 0, and a wrapper that exits successfully is a wrapper that a Restart=on-failure unit leaves dead — after which reviews starve until somebody notices. The deployed unit:

Restart=always
ExecStart=/home/bcloud/.local/bin/coder-llm-ticketed -- …

and docs/ci-runners.md:80 already states the reason it must be always ("runsvc.sh reports success when it is killed"). So the guard's success path restarts cleanly, and the invariant it depends on is documented where the unit is defined rather than only in this script.

One live observation

ExecStart is now coder-llm-ticketed on this box, so the switch has happened — which means the ticketed path is production, not prospective, and the memory-window interaction in #2515 (a window that stops nothing still closing the fleet's 45-minute gap) is the next thing that costs the fleet time. Not this PR's problem, but the two merged in either order leave that window open until the gap filter lands.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 72d9dbf (posted as a comment: a pending review from this identity blocks the reviews endpoint) -- my ask landed with a consequence I had not computed, and the arm is the right one

I asked for a demand re-check once the ticket is held; the head does it and quantifies the cost better than I did: "with a 24 h queue wait, every ticket from a busy period survives to be granted, and --window loaded the 30B model before asking whether anyone still wanted it. On an 11-deep queue that is up to 11 loads, each followed by 5 idle minutes, before the live request is served." That is the amplification I missed: it is not one wasted window, it is N of them, and each is a 30B load. That number is now the argument in the commit, which is where it belongs.

CODER_LLM_ALWAYS as the escape hatch is right, and arm H proves the branch: a demand file touched at queue time with a 3-second TTL is stale by the time the lane releases, so the grant must "release at once instead of loading a 30B model and 5 idle minutes". A test that ages the demand during the wait is the only way to exercise this, and it is the shape I would have asked for.

Two asks, both about the escape hatch rather than the fix:

  1. Document CODER_LLM_ALWAYS=1 where the rest of the knobs live and say what it costs: it bypasses the staleness check, so on a long queue it re-creates exactly the 11-loads problem this PR fixes. A knob whose documentation is one line of shell is a knob someone sets to "make it work" at 2 a.m. One sentence in docs/ci-runners.md -- "for a single-window debug run, not for a service unit" -- is the whole fix.
  2. My second ask from the previous head is still open: "Deploy after merge: copy scripts/coder-llm-ticketed to ~/.local/bin" is a remembered cp, and the installer already exists with --install and --status. If --install does the copy, say so and drop the manual step; if it does not, add it -- because the deploy step that is a manual command is the one that gets skipped.

Nothing else: arm G still pins the long default, the drop-in is unchanged, and the release path now exits without starting the server rather than after, which is the difference between a queue that costs nothing when it is wrong and one that costs a model load per grant.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 10:26
@bong-water-water-bong
bong-water-water-bong merged commit c56da1b into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/coder-llm-queue-wait branch November 15, 2026 10:32
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Queue Wait Duration

The default queue wait duration was changed from 5400 seconds (1.5 hours) to 86400 seconds (24 hours). This change is intended to prevent tickets from being re-queued at the back of the FIFO on a busy system, which would cause them to starve. The change is well-documented and tested, but it's important to verify that this extended wait time does not introduce any unintended side effects, such as resource exhaustion or delays in responding to actual demand.

QUEUE_WAIT=${CODER_LLM_QUEUE_WAIT_S:-86400}
Stale Demand Handling

A new check was added to release the ticket immediately if the demand is stale when the ticket is granted. This prevents the server from starting up and idling for a long time if the demand has already expired. This is a good optimization to avoid unnecessary resource usage.

if [ "${CODER_LLM_ALWAYS:-0}" != 1 ] && ! demand_fresh "$DEMAND_TTL" && ! gh_demand; then
    log "ticket granted with no fresh demand: releasing without starting the server"
    exit 0
fi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant