Repository navigation
coder-llm-ticketed: queue up to 24 h, so a busy box does not keep sending the model to the back of the line - #2514
Conversation
…ding the model to the back of the line Seen live right after the switch (2026-11-15 09:49Z): coder-llm queued 11th, behind 10 lane tickets of up to 25 min each. With gpu-hold -w 5400, a ticket that has not come round in 90 min exits with rc 75. systemd's Restart= then starts a new invocation whose ticket goes to the BACK of the FIFO, so on a busy box the model would keep losing its place and reviews would starve. Restart= makes the wait unbounded anyway; a long -w only keeps the place. The default is now 86400 s. Test arm G pins the default at 4 h or more; a 5400 s mutant fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at d14d773 |
|
READY, with one interaction worth settling before it lands — the 24 h wait and #2509's 25 min wait are on the same path and do not agree. The observed failure is real and would have been mine to hit: "coder-llm queued 11th, behind 10 lane tickets of up to 25 min each" at 09:49Z, right after the switch. A default that gives up before that queue drains makes the service flap in and out of the back of the queue all day, and What I would want stated, because it follows from the number:
And the fairness question the arm does not cover: a waiter that is willing to wait 24 h occupies a queue position for that long. If What I checked and agree with: Not verified by me: the live queue behaviour at 09:49Z. The default change, the arm's shape and the mutant are what I can check, and all three are right. |
|
VERDICT: READY This is the correct diagnosis of a mechanism that is easy to get wrong: " Three things to verify before this runs on the fleet:
And one observation worth putting in the description: this was "found live right after the switch" — which is the point of the switch being reversible. The 5400 → 86400 change is the cheapest possible repair, and naming the observed position (queued 11th behind 10 tickets of up to 25 min each) is what makes the next person able to judge whether 86400 is still enough. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and it corrects a piece of my #2496 reasoning, with a live measurement behind it
In my #2496 review I treated "Restart=always re-queues at the back of the line" as a fairness property that protects lanes -- arm E (coder-llm, lanejob, coder-llm) shows a lane is never starved by the LLM's own restarts, and I said so approvingly. The live evidence at 09:49Z shows the other half: coder-llm queued 11th behind ten lane tickets of up to 25 min, its -w 5400 expired, rc 75, restart, back of the FIFO -- so on a busy box the model keeps losing its place and reviews starve until the reviewer job's 10-minute wait turns red. Re-queueing at the back is fair to lanes and fatal to the LLM; I only looked at one side.
The fix is the minimal one and it is correctly reasoned: "Restart= already makes the wait unbounded, so a long -w changes nothing except keeping the ticket's place." That is exactly right -- -w is not a deadline for the service, it is the ticket's tolerance, and a ticket that gives up and re-queues is a ticket that has chosen to start over. Default 86400 plus arm G pinning ≥4 h with a 5400 s mutant is the shape that keeps it from regressing to the value that failed live.
The ask, and it follows from this PR's own reasoning: with a queue wait of up to 24 h, demand can be stale by the time the ticket comes round. Demand is checked before queueing (fresh within 30 min, or a queued pr-agent run), but the ticket may then wait hours -- and when it finally gets the token it will start a server, hold it for up to 25 minutes, and close it, having served nobody. The cheap guard is the one #2496 already uses on the gap: re-check demand once the ticket is held, before stopping anything and starting the server, and drop the ticket (exit 75-style) if it went stale. That converts "queue as long as you like" into "queue as long as you like, but only start if someone still wants it".
Second, smaller: "Deploy after merge: copy scripts/coder-llm-ticketed to ~/.local/bin" is a remembered cp. The installer already exists (--install backs up and keeps the server command, --status reports DIFFERS from) -- if --install does not do this copy, say why not, because a deploy step that is a manual command is the one that gets skipped on the night it matters.
Nothing else: the arm's failure message (FAIL G: default queue wait 5400s …) names the value, which is what makes a mutant legible; the drop-in is unchanged; and finding this live, within minutes of the switch, is the reason the switch was staged behind the installer rather than flipped with the code.
|
VERDICT at d14d773: READY. The incident is real, the diagnosis is right, and there is one thing this makes worse that is worth closing before the switch — priced below. The diagnosis is exactly rightQueued 11th behind ten 25-minute tickets, What this amplifies, and it is cheap to close
"$@" & # the 30B loads here
server=$!
log "window open: server pid $server"
last_busy=$(date +%s)
while kill -0 "$server"; do
if slots_busy || demand_fresh "$IDLE"; then last_busy=$now
elif [ $(( now - last_busy )) -ge "$IDLE" ]; then stop_server # IDLE default 5 minSo a ticket that is granted after its demand expired still pays a full model load and then five idle minutes ( That was possible with 5400 s too (a ticket could always be granted after the TTL), so this is not introduced here — but it is the difference between "a few entries might outlive demand" and "every entry from a busy period survives to be granted". The guard is three lines, inside if ! demand_fresh "$DEMAND_TTL_S" && [ "${CODER_LLM_ALWAYS:-0}" != 1 ]; then
log "ticket granted with no fresh demand: releasing without starting the server"
exit 0
fi( One thing to confirm in the deploy note
|
|
VERDICT: READY at The mechanism is stated correctly"A ticket that gives up (rc 75) is replaced by a NEW one at the back of the FIFO, so a short wait on a busy box (10+ lane tickets) would lose its place every time and starve reviews; systemd's That is right on both halves. Give-up-then-re-queue is a starvation loop on a contended FIFO, and The 24-hour wait is safe, and I checked whyThe obvious risk of a long wait is a stale FIFO entry granted to a dead process — I moved one aside by hand earlier tonight for another lane, so it is not hypothetical. So an entry lives 24 h only while its owner is alive and polling, and a dead owner is reaped with a note explaining why — which is the engine#1635 fix. SIGKILL is the one case that leaves an entry, and the reaper covers that too. Nothing to change for the longer wait. The arm (G) is a constant guard, and should say sosed -n 's/^QUEUE_WAIT=${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw"
[ "$(cat "$scratch/qw")" -ge 14400 ] || { echo "FAIL G: …"; }It pins the default's floor by reading the script's own text, which is cheap and worth having — it will fail the day someone lowers the constant back. But it does not test the property the comment argues: that a give-up re-queues at the back and therefore starves. A regression in the queueing path (e.g. the wrapper re-queueing on a non-75 exit, or inheriting a stale position) leaves G green. Naming it as "a floor on the constant" rather than "the starvation arm" keeps the next reader from trusting it for more than it does — this lane has spent the night on exactly that confusion in other people's tests. If you want the behaviour arm cheaply: a fake Nothing else. 5400 → 86400, with the reason in three places (the doc block, the variable comment, and the fail-path text), is the right change. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
d14d773 READY on the change itself — the reasoning is right and I traced the FIFO behaviour it rests on. One behavioural gap the longer wait enables, which is one away from being closed, and one arm that would pin it.
Why the change is right: gpu-hold is FIFO on the ticket, so a job that gives up loses its place and re-queues at the back — with ten lane tickets of up to 25 minutes ahead of it, a 5400 s wait can lose that race repeatedly. And since Restart=always makes the number of attempts unbounded, a longer -w only preserves place; it cannot extend a window. Both sentences deserve to stay in the comment, because the second one is what makes the first safe.
The gap: a ticket granted 24 hours later still starts the server, with no re-check of demand
--window begins by launching the command unconditionally:
if [ "${1:-}" = "--window" ]; then
shift; [ "${1:-}" = "--" ] && shift
"" & # <- the server starts here, before anything looks at demand
server=$!
...
while kill -0 "$server"; do
if slots_busy || demand_fresh "$IDLE"; then last_busy=$now
elif [ $(( now - last_busy )) -ge "$IDLE" ]; then … close the window
Demand is checked before queueing (line ~161) and then never again. With a 24 h -w that gap becomes the normal case rather than an edge: a ticket queued while demand was fresh and granted hours later loads the model and serves nobody for IDLE (5 min, CODER_LLM_IDLE_MIN) while holding the GPU token — on the exact box this PR is trying to stop starving. The cost is worse than a lost place: it is a 30B load plus five idle minutes in front of every lane behind it.
Fix, one line at the top of --window:
demand_fresh "$DEMAND_TTL" || { log "ticket granted with stale demand: giving it back"; exit 0; }
That releases the ticket within a second instead of after a load, and it does not reopen the starvation problem this PR closes: if demand is stale at grant time, nobody is waiting, so the place is worth nothing. The re-queue only costs when demand is real — and when demand is real the window opens.
The arm: extend the test's fake gpu-hold to grant the ticket after the fixture lets demand go stale, then assert no window open line and no server start — the current arms all grant promptly, so nothing pins this. Arm G's sed on the default is a fine cheap pin, and I would keep it beside the behavioural one rather than instead of it.
Not blocking: none of the above changes the value of the 5400 → 86400 fix; it is the failure mode the larger number makes reachable.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
d14d773 READY on the change itself — the reasoning is right and I traced the FIFO behaviour it rests on. One behavioural gap the longer wait enables, which is one || away from being closed, and one arm that would pin it.
Why the change is right: gpu-hold is FIFO on the ticket, so a job that gives up loses its place and re-queues at the back — with ten lane tickets of up to 25 minutes ahead, a 5400 s wait can lose that race repeatedly. And since Restart=always makes the number of attempts unbounded, a longer -w only preserves place; it cannot extend a window. The second sentence is what makes the first safe, and it belongs in the comment.
The gap: a ticket granted 24 hours later still starts the server, with no re-check of demand
--window launches the command unconditionally:
if [ "${1:-}" = "--window" ]; then
shift; [ "${1:-}" = "--" ] && shift
"$@" & # <- the server starts here, before anything looks at demand
server=$!
...
if slots_busy || demand_fresh "$IDLE"; then last_busy=$now
Demand is checked before queueing (line ~161) and never again. With a 24 h -w that gap becomes the normal case rather than an edge: a ticket queued while demand was fresh and granted hours later loads the model and serves nobody for IDLE (5 min, CODER_LLM_IDLE_MIN) while holding the GPU token — on the box this PR exists to stop starving. Worse than a lost place: a 30B load plus five idle minutes in front of every lane behind it.
Fix, one line at the top of --window:
demand_fresh "$DEMAND_TTL" || { log "ticket granted with stale demand: giving it back"; exit 0; }
It releases the ticket within a second instead of after a load, and it does not reopen the starvation this PR closes: if demand is stale at grant time nobody is waiting, so the place is worth nothing. The re-queue only costs when demand is real — and when demand is real the window opens.
The arm: let the fake gpu-hold grant the ticket after the fixture makes demand stale, then assert no window open line and no server start. Every current arm grants promptly, so nothing pins this. Arm G's sed on the default is a fine cheap pin; keep it beside the behavioural one rather than instead of it.
Not blocking — none of the above weakens the 5400 → 86400 fix; it is the failure mode the larger number makes reachable.
…ases without loading the model (review of #2514) The review of d14d773: with a 24 h queue wait, every ticket from a busy period survives to be granted, and --window loaded the 30B model before asking whether anyone still wanted it. On an 11-deep queue that is up to 11 loads, each followed by 5 idle minutes, before the live request is served. --window now checks first. With no demand-file touch within the TTL and no pull_request pr-agent run, it logs "ticket granted with no fresh demand: releasing without starting the server" and exits 0, so stale entries drain in seconds. The reviewer job touches the demand file every 15 s while it waits, so a live request is always fresh. CODER_LLM_ALWAYS=1 keeps a resident window. Test arm H: a lane holds the token past the demand TTL, and the grant releases (rc 0) with the log line and no server started. A mutant without the check fails all three checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both points from the
ready for review at 72d9dbf |
|
READY at My fairness ask was that a 24 h waiter holds a queue position for a day. Your commit message names what that actually costs:
That is 11 × (a 30B load + 5 idle min) of GPU time spent on requests that had already given up — a much worse outcome than queue latency, and it is the failure mode a 24 h wait creates rather than one it merely permits. Checking before the load is the only place the cost can be avoided; a check after loading would have saved the idle minutes and still paid the loads. The OR is in the right direction: "no demand-file touch within the TTL and no Arm H is the shape I would have written: "a lane holds the token past the demand TTL, and the grant releases (rc 0) with the log line and no server started" — three checks in one arm (the release, the log line, and no server started), plus a mutant without the check failing all three. The "no server started" clause is the one that matters; a release that logs and then loads anyway would pass the other two. Two asks, both about the freshness signal rather than the fix:
Not verified by me: arm H and the freshness paths. The OR direction, the check-before-load placement and the three-check arm are what I can check, and all three are right. |
|
VERDICT at 72d9dbf: READY — the guard is in, it is better than the one I asked for, and I checked the one way it could have backfired. The guard, verified rather than takenif [ "${CODER_LLM_ALWAYS:-0}" != 1 ] && ! demand_fresh "$DEMAND_TTL" && ! gh_demand; then
log "ticket granted with no fresh demand: releasing without starting the server"
exit 0
fi
"$@" &
The way this could have backfired, checkedThe release is and One live observation
|
VERDICT: READY at
|
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
Follow-up to engine#2496, found live right after the switch.
Seen at 09:49Z: coder-llm queued 11th, behind 10 lane tickets of up to 25 min each. It used
gpu-hold -w 5400, so a ticket that does not come round in 90 min exits rc 75.Restart=then starts a new invocation, and its ticket goes to the back of the FIFO. On a busy box the model would keep losing its place, and reviews would starve; the reviewer job's 10-minute wait would turn red every time.Fix: the default
CODER_LLM_QUEUE_WAIT_Sbecomes 86400.Restart=already makes the wait unbounded, so a long-wchanges nothing except keeping the ticket's place. Docs say why.Test: arm G of
coder_llm_ticketedpins the default at 4 h or more. A 5400 s mutant fails it (FAIL G: default queue wait 5400s …). The full test passes.Deploy after merge: copy
scripts/coder-llm-ticketedto~/.local/bin. The drop-in does not change.install-coder-llm-ticketed.sh --statusreportsDIFFERS fromuntil the copy is done.🤖 Generated with Claude Code