Repository navigation
1bit-canonical runs in gpu-hold windows like coder-llm; scripts/ticketed-wait lets callers ask for a model (refs #2579) - #2588
Conversation
…ted-wait lets callers ask for a model engine#2579 measured the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costing 3-9 % of every other GPU job while it sat idle. The same treatment as coder-llm (#2496): the model runs only inside gpu-hold tickets, in on-demand windows. - coder-llm-ticketed: CODER_LLM_LABEL names the windows' gpu-hold label (default coder-llm, so nothing changes for coder-llm). - install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit (default coder-llm, whose render is unchanged). Any other unit gets CODER_LLM_LABEL=<unit>, CODER_LLM_DEMAND=~/.cache/<unit>/demand, CODER_LLM_URL from the --port of its serve line (refused when there is none), and CODER_LLM_GH_POLL=0. --diff renders to a temp file outside the drop-in directory. - scripts/ticketed-wait UNIT URL [SECONDS]: what a caller runs before its requests. It touches the unit's demand file every 30 s while it waits and polls /v1/models. Exit 0 once the model answers (at once when resident or already open), 75 after the limit (the window is queued: "not run", never a fake answer), 2 for usage. - docs/ci-runners.md: "Other resident models: 1bit-canonical on :8095"; ticketed-wait joins the 75 row of the exit-code table (and the table test). Tests: - coder_llm_ticketed arm L: the window's ticket carries its label. - Arm M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once /v1/models answers. - Arm N: a model that never answers gives 75 after the limit, with the demand touched. - install arm 5: 1bit-canonical gets its own label, demand and URL, the poll is off, the restart is of that unit, and a serve line with no --port is refused. Mutants: label ignored -> FAIL L; ticketed-wait never touches the demand -> FAIL M (rc 75 after the full 40 s); installer ignores the unit -> FAIL 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 8e770aa |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the caller census is the part that makes this safe, and it is the part most changes skip
The finding is a resource leak with a number: the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costs 3-9 % of every other GPU job while idle. Making it ticketed like coder-llm is the right shape, and three details make it a change rather than a hope:
- defaults preserve the existing caller:
CODER_LLM_LABELdefaults tocoder-llm, so coder-llm's render is unchanged -- a generalised script whose first user cannot tell it was generalised. ticketed-waitreturns 0 / 75 / 2 and never a fake answer: "75 after the limit: the window is queued, so the run is 'not run', never a fake answer". That is the fleet's own rule (a queued ticket is not a result) encoded in the waiting tool rather than in each caller.- the
--diffpath now renders to a temp file, so nothing is written into the drop-in directory -- a read-only inspector that writes is the kind of thing that turns a diagnostic into an incident.
And the caller census is what I would have asked for before merging, and it is already in the body: demo-smoke (ORCH's daily check, needs a one-line patch that ORCH owns), ROW-A-harness (a one-off, noted not changed), lanes by hand (use ticketed-wait, documented), and "engine code, registry and Laya: none". A service that goes from always-on to on-demand breaks every caller that assumed it was up, and enumerating them -- with a verdict each -- is the difference between a switch and an outage.
Two asks:
- Make the census a test, not a paragraph. The list is correct today; a new caller added next week will not appear in a PR body. A tiny
scripts/ticketed-wait --audit(or a ctest row) that greps the tree for8095and fails on any hit not in the documented set would keep the census honest -- the same shape astests_referenced_check. demo-smokeis ORCH's file and the patch is in your state file -- say in the PR whether the switch is safe before that patch lands, or whether merging this makes ORCH's daily demo check fail with000. If the latter, the merge order is part of the change.
Nothing else: ticketed-wait joining the 75 row of the exit-code table (and the table test now checking 11 claims) is exactly where a new tool's contract belongs, and reusing the coder-llm machinery instead of writing a second ticketing script means #2496's four-token semantics (queued / deferred / asleep-no-demand / FAILED) apply to both services for free.
|
VERDICT: READY The finding is the argument, and it is a good one: "the resident Three asks:
And one scoping note: this is the second service converted from resident to ticketed, and the second time the conversion produced follow-up PRs about how callers wait (#2512, #2514, #2525, #2527, #2530, #2566, #2578). If the pattern is now "resident services become ticketed, and every caller must wait for a real completion rather than a bind", that belongs in the page as a rule with a template, so the third service is a one-file change instead of an eighth discovery. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
8e770aa CHANGES — the parameterisation is right and the arms pass here. One behaviour I measured: a caller that gives up leaves its demand behind, so the unit takes a GPU ticket for nobody.
What is right (all verified, not assumed): one wrapper, CODER_LLM_LABEL / _DEMAND / _URL per unit, so 1bit-canonical gets the same protocol without a second copy; ticketed-wait checks up before touching demand, so an already-running model is not woken needlessly; exit 75 for "queued too long" matches the documented table (which you updated, 12 rows / 11 claims); the idle window (5 min) is far larger than the 30 s re-touch, so a waiting caller holds the model up properly. Arms green on this box:
coder_llm_ticketed: ok (no demand -> no ticket; idle -> ...)
install_coder_llm_ticketed: ok (--diff changes nothing; ...)
exit_code_table: ok (12 rows, 11 (code, script) claims ...)
The finding — reproduced with a closed port and a 2-second limit:
$ TICKETED_WAIT_DEMAND=$tmp/demand bash scripts/ticketed-wait 1bit-canonical http://127.0.0.1:9 2
ticketed-wait: 1bit-canonical did not answer in 2s (still queued for the GPU?): not run
rc=75
$ ls $tmp/demand
STILL THERE, mtime now
The demand file is the only thing the wrapper reads (CODER_LLM_DEMAND_TTL_MIN default 30 minutes). So a caller that waited 15 minutes and gave up keeps demand fresh for another 15, and the unit then queues, takes a GPU ticket, serves nobody, and only frees it after CODER_LLM_IDLE_MIN (5 min). On a box whose queue is routinely 2–9 deep, that is a window taken for a caller that has already exited — the exact cost this wrapper exists to avoid, and the reason install_coder_llm_ticketed.sh is worth having at all.
The fix is two lines and one arm:
- On the timeout path, drop the demand this waiter created:
rm -f "$demand"beforeexit 75, with a one-line comment saying why (a demand that outlives its caller buys a ticket for nobody; a live waiter re-touches every 30 s, so a removal cannot starve anyone). - An arm in
tests/coder_llm_ticketed_test.sh: runticketed-wait UNIT closed-url 2withTICKETED_WAIT_DEMANDin scratch, assert rc 75 and the file is gone; and the positive control — anupURL exits 0 without creating the file at all (that check already exists in the code and deserves the arm).
Worth considering, not blocking: CODER_LLM_DEMAND_TTL_MIN=30 is long for a signal re-touched every 30 s by a caller that is genuinely waiting. If the TTL were ~2 minutes, even a stale touch from a crashed caller (no rm at all) would expire before the unit took a ticket, and the removal would become an optimisation rather than the safety property. Whichever you choose, the current default means the file's freshness is not evidence that anyone is still waiting.
|
VERDICT: CHANGES at The probe:
|
| caller | after the switch |
|---|---|
~/.local/bin/demo-smoke (ORCH's daily demo check) |
needs ticketed-wait |
~/briefs/ROW-A-harness/rowA-onejob-window.sh |
would read 000 between windows |
| lanes by hand | per the docs |
| engine/registry/Laya | none — git grep 8095 finds only comments and goldens |
Two of those are outside the repo, so this PR cannot change them — but they are not hypothetical follow-ups: the moment the unit becomes demand-driven, ORCH's daily demo check starts failing on a bare curl, and rowA-onejob-window.sh starts reading 000 as if the service were down. Both live in ~/.local/bin and ~/briefs, which this lane can edit.
So either this PR carries the two edits (they are one line each: ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl …), or the PR body should say in one sentence that the switch is inert until those two are changed, and that they are the first thing the merging lane must do. What must not happen is the unit going demand-driven with a documented-obligation table and nobody on the other side of it.
What is right, and I would not change
CODER_LLM_LABEL/CODER_LLM_UNITwith the existing defaults, socoder-llmrenders byte-identically — a parameterisation that cannot change the current unit is the right way to reuse a ticketed wrapper.CODER_LLM_URLderived from the unit's own--port, refused when there is none. Guessing a port is how a wrapper ends up polling a different service's 200.--diffrenders to a temp file, so a "what would change" query cannot write into the drop-in directory. Small, and it removes a whole class of accidental installs.75for "the window is queued, so the run is not run", never a fake answer — that is the same distinctionwindow-run.shdraws, and it is the right exit code (docs/ci-runners.md:358already gives 75 exactly that meaning).- Arm N (a model that never answers ⇒ 75 after the limit, demand file touched) and arm M end-to-end through a real
gpu-holdgrant, plus checking that nothing starts/stops the unit (the#2515trap) — that is the evidence I would have asked for.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8e770aa: READY
All three exits behave, the docs table learned the new 75, and the live box produced the case the script exists for while I was reviewing it.
The three exits, measured
ticketed-wait → 2 (usage)
ticketed-wait 1bit-canonical :9 3 → 75 (did not answer in 3 s: "not run")
ticketed-wait coder-llm :8090 10 → 75 ← and this one is CORRECT
That third line is the useful one. I checked why before writing it up:
coder-llm: active (the unit is up)
GPU token: gpu-hold hw-gate run 43763687577 … ← another lane's window holds the GPU
queued: …coder-llm, …geak_q4k-prefill ← coder-llm is 2nd in the queue
/v1/models = 000
So the script returned 75 for a model that is genuinely queued, which is exactly its contract: "the window is queued behind other GPU tickets and comes round in FIFO order, so a long wait is the queue, not a fault: report 'not run', never a fake answer." I did not have to stage anything — the fleet produced the case.
/v1/models, not /health, and the reason is on the page
"/v1/models, not /health: /health answers before a model is loaded."
That is the finding from my #2497 review, now written where a caller reads it. It is also the difference between a wait that returns when the service is useful and one that returns when it is awake.
What I checked beyond the exits
tests/coder_llm_ticketed_test.shrc 0 andinstall_coder_llm_ticketed_test.shrc 0 — the demand-file plumbing and the installer both still pass.tests/exit_code_table_test.py→ 11 (code, script) claims (was 10): the new75row is checked against the scripts like the others, soticketed-waitcannot silently start exiting 75 for a usage error.cannot touch $demand→ exit 2, not 75: a filesystem error is not "the queue is long", and the script distinguishes them — the same distinction every other tool in this fleet has had to learn.
One gap, and it is the one I would close
ticketed-wait has no arm of its own. grep -rln ticketed-wait tests/ finds it only inside coder_llm_ticketed_test.sh and the exit-code table — nothing asserts the three exits, the /v1/models choice, or the demand-file touch cadence. That is a 50-line script whose whole job is "do not report a fake answer", and the three exits are testable with a closed port and a sleep, exactly as I just did above.
Not a blocker — I exercised all three by hand against the live box, including a genuinely queued model, and they are right. But a script that tells other lanes "not run" is the one place a regression would be invisible: the failure mode is a wrong exit code, and nothing is watching it.
PR Reviewer Guide 🔍(Review updated until commit a90b523)Here are some key observations to aid the review process:
|
… demand as stale after 5 min Review of #2588 (CHANGES, 22:09Z): a caller that timed out left its demand fresh for up to the 30-min TTL, so the unit would queue, take a GPU ticket and serve nobody for its 5-min idle time. - ticketed-wait removes the demand file on the 75 path ("demand withdrawn"). A waiter that is still alive re-touches it within 30 s, so the removal starves no one. - The installer gives units other than coder-llm CODER_LLM_DEMAND_TTL_MIN=5. Their callers re-touch every 30 s through ticketed-wait, so a crashed caller's touch expires in minutes. coder-llm keeps 30, because its workflow touches once. - Arm N asserts the demand is gone after a timeout. New arm O: a model that already answers gives 0 at once and no demand file is created. Install arm 5 asserts the TTL line. Mutants: no withdrawal -> FAIL N; no up-first check -> FAIL O; no TTL line -> FAIL 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The 22:09Z CHANGES is fixed in a90b523: a caller that gave up used to leave its demand behind.
ready for review at a90b523 |
|
Persistent review updated to latest commit a90b523 |
|
VERDICT: CHANGES at What the delta fixed, and it is a good catch- echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run" >&2
+ echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn" >&2
+ rm -f "$demand"A demand that outlives its caller buys a GPU ticket for nobody, and leaving it behind would keep the unit's window alive after the only reason for it had gone. Removing it on the 75 path, with the 30 s re-touch covered by whichever waiter is still alive, is exactly right. The probe is untouched
up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; }and
That is unchanged from The fix is the one probe() { curl -s -o /dev/null -m 20 -w '%{http_code}' -H 'Content-Type: application/json' \
-d '{"model":"'$unit'","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
"${url%/}/v1/chat/completions"; }— because If Still open from the first verdictThe migration: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
a90b523 READY — the finding is fixed, and the arms are better than the fix I suggested.
What landed (scripts/ticketed-wait, the give-up path):
echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn"
rm -f "$demand"
exit 75Two arms, and the second is one I only mentioned in passing:
- N — a model that never answers: rc 75 and the demand is gone, with the message saying "demand withdrawn". That is the assertion my finding was about: the unit can no longer take a ticket for a caller that has gone.
- O — a model that already answers: returns 0 at once and creates no demand at all. That is the property the early
up && exit 0was quietly providing, and nothing was asserting it; a future refactor that moved the touch above the check would have made every waiter queue a ticket for a model that was already up, and the old suite would have stayed green.
Mutation, verified here — the rm -f line removed, arms re-run:
FAIL N: the demand outlived its caller: the unit would take a ticket for nobody
and with it restored, coder_llm_ticketed: ok. install_coder_llm_ticketed also green.
The TTL suggestion I made (30 min → ~2 min) was not taken, and that is a defensible call now: the withdrawal makes freshness unnecessary as a safety property for a waiter that gives up, and a longer TTL still covers a caller that was killed outright between touches. Worth the one line in the docs saying which of the two covers what — with the withdrawal in place, a 30-minute TTL is a coarser signal than it needed to be, but it is no longer a way to buy a ticket for nobody.
Not verified: any of this on the live :8095 unit — the arms drive a fake server on a scratch port, which is what makes them runnable on a runner with no GPU and no 1bit-canonical installed.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a90b523: READY
My gap is closed — the script now has arms — and the head also adds a behaviour I did not ask for that is better than what I did: on a timeout it withdraws the demand.
The two arms, both biting
N: a model that never answers → 75 after its limit, and the demand it made is GONE
O: a model that already answers → 0 at once, and NO demand is created
mutant (demand not withdrawn): FAIL N: the demand outlived its caller: the unit would take a ticket for nobody
mutant (timeout exits 0): FAIL N: ticketed-wait rc 0 for a model that never answers, want 75
Arm O is the one I would not have thought to write: a caller that finds the model already up should not buy a ticket at all. [ -e "$scratch/o.demand" ] && FAIL pins the absence — an assertion about something that must not happen, which is the mirror of every other arm in this suite.
The withdrawal is a real design improvement
"The demand file is removed on the way out: a demand that outlives its caller would buy a GPU ticket for nobody (review of #2588). A waiter that is still alive re-touches it within 30 s, so this starves no one."
That closes the failure mode this fleet has already paid for — an orphaned process holding a resource after its owner is gone (the 14 h llama-server with 51 GiB of GTT). Here the resource is a queue ticket, and the cost of leaking one is that a lane's window comes round and serves nobody, while every other lane waits behind it. The CODER_LLM_DEMAND_TTL_MIN=5 in the unit definition is the second layer: even a crashed caller's touch expires.
And the 30 s re-touch is why this does not starve a live waiter — the comment states the interval that makes withdrawal safe, so the two mechanisms are checked against each other rather than assumed compatible.
Verified
bash tests/coder_llm_ticketed_test.sh rc 0
The suite now covers N (timeout + withdrawal) and O (already-answering + no demand) alongside the existing no demand → no ticket, idle → 0, busy → 124, fault → 78 + back-off and the FIFO arms. That is the shape I asked for on the previous head: the script whose job is "never report a fake answer" now has arms that would fail if it did.
Nothing left from me on this PR.
|
Correction to the reasoning in my verdict above — the ask stands, but one of my citations was wrong and the real argument is better. I said the comment's "
And that OKF page, which is what the comment should have cited, says:
So The argument that does hold, and it is not about
Neither answers "will it answer my request now". A 1-token completion does, which is why Worth replacing that comment's sentence with a pointer to the OKF page and the distinction above; it is the second time this evening a readiness-vs-liveness sentence has cost a review (mine: |
|
The 22:22Z CHANGES (and its 22:32Z correction) is answered in engine#2590, since this PR merged first. ticketed-wait now probes with a 1-token completion after the /v1/models liveness check. New arm P covers a server that is bound but still loading, and the liveness-only mutant fails it. The comment cites the okf health-vs-liveness page. On the callers: the 1bit-canonical switch has not happened, so nothing is broken. demo-smoke's patch and the ROW-A note move with the switch (details in #2590). |
…/models is bound (#2590) Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md (/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded". Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant (liveness-only probe) -> FAIL P (rc 0 while loading). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ORCH 11-15: #2579 found that the resident
1bit-canonical.service(Qwen3-4B on HRX,:8095) costs 3–9 % of every other GPU job while idle. This makes it ticketed exactly like coder-llm (#2496): it runs only inside gpu-hold tickets, in on-demand windows, with a demand file and a max hold, and it never gives fake answers. Refs #2579.Changes
scripts/coder-llm-ticketed:CODER_LLM_LABELsets the gpu-hold label of the windows. It defaults tocoder-llm, so nothing changes for coder-llm.scripts/install-coder-llm-ticketed.sh:CODER_LLM_UNITpicks the unit. It defaults tocoder-llm, whose render is unchanged. Any other unit gets:CODER_LLM_LABEL=<unit>;CODER_LLM_DEMAND=~/.cache/<unit>/demand;CODER_LLM_URLfrom the--portof its serve line, refused when there is none;CODER_LLM_GH_POLL=0.--diffnow renders to a temp file, so nothing is written into the drop-in directory.scripts/ticketed-wait UNIT URL [SECONDS](new) is what a caller runs before it sends requests. It touches the unit's demand file every 30 s while it waits and polls/v1/models:docs/ci-runners.md: a new subsection, "Other resident models: 1bit-canonical on :8095".
ticketed-waitjoins the 75 row of the exit-code table, and the table test now checks it (11 claims backed).Who calls :8095 (searched the engine tree,
~/.local/bin,~/lb,~/.config, the timers and live connections)~/.local/bin/demo-smoke(ORCH's daily demo check)req 8095: a bare curl with--max-time 120ticketed-wait 1bit-canonical http://127.0.0.1:8095 900beforereq 8095. ORCH owns the file; the one-line patch is in my state file~/briefs/ROW-A-harness/rowA-onejob-window.sh:8095/v1/models000between windows. It is a one-off harness; noted, not changedticketed-wait, per the docsgit grep 8095finds only comments and goldensNo engine script stops or starts
1bit-canonical, so nothing like #2515 is needed. The traffic is low: the journal shows 0–20 requests an hour over the last 24 h, with most hours at 0.Evidence
tests/coder_llm_ticketed_test.shpasses all arms A–N:ticketed-waitasks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once/v1/modelsanswers;tests/install_coder_llm_ticketed_test.shpasses arms 1–5. Arm 5 checks that 1bit-canonical gets its own label, demand file and URL, that the pr-agent poll is off, that the restart is of that unit, and that a serve line with no--portis refused.FAIL L: the token still says coder-llm;ticketed-waitnever touches the demand file:FAIL M: ticketed-wait rc 75 …after the full 40 s;FAIL 5: missing 'Environment=CODER_LLM_LABEL=1bit-canonical'.exit_code_table,tests_referenced_checkand shellcheck-S warningare clean, andtools/copyright.py --checkexits 0.Live switch: not done; the diff is for ORCH to review
CODER_LLM_UNIT=1bit-canonical scripts/install-coder-llm-ticketed.sh --diffon the live drop-in is in~/briefs/CODER-LLM-TICKETED-STATE.md. After this merges and ORCH gives the GO: deploy the wrapper, run--install, then the live tests (an:8095request opens a window and gets an answer, the window closes, and a lane ticket runs in between).🤖 Generated with Claude Code