Skip to content

1bit-canonical runs in gpu-hold windows like coder-llm; scripts/ticketed-wait lets callers ask for a model (refs #2579) - #2588

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-infra/canonical-ticketed
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-infra/canonical-ticketed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

ORCH 11-15: #2579 found that the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costs 3–9 % of every other GPU job while idle. This makes it ticketed exactly like coder-llm (#2496): it runs only inside gpu-hold tickets, in on-demand windows, with a demand file and a max hold, and it never gives fake answers. Refs #2579.

Changes

  • scripts/coder-llm-ticketed: CODER_LLM_LABEL sets the gpu-hold label of the windows. It defaults to coder-llm, so nothing changes for coder-llm.

  • scripts/install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit. It defaults to coder-llm, whose render is unchanged. Any other unit gets:

    • CODER_LLM_LABEL=<unit>;
    • CODER_LLM_DEMAND=~/.cache/<unit>/demand;
    • CODER_LLM_URL from the --port of its serve line, refused when there is none;
    • CODER_LLM_GH_POLL=0.

    --diff now renders to a temp file, so nothing is written into the drop-in directory.

  • scripts/ticketed-wait UNIT URL [SECONDS] (new) is what a caller runs before it sends requests. It touches the unit's demand file every 30 s while it waits and polls /v1/models:

    • 0 once the model answers, at once if the unit is resident or its window is already open;
    • 75 after the limit: the window is queued, so the run is "not run", never a fake answer;
    • 2 on a usage error.
  • docs/ci-runners.md: a new subsection, "Other resident models: 1bit-canonical on :8095". ticketed-wait joins the 75 row of the exit-code table, and the table test now checks it (11 claims backed).

Who calls :8095 (searched the engine tree, ~/.local/bin, ~/lb, ~/.config, the timers and live connections)

caller what it does after the switch
~/.local/bin/demo-smoke (ORCH's daily demo check) req 8095: a bare curl with --max-time 120 needs ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 before req 8095. ORCH owns the file; the one-line patch is in my state file
~/briefs/ROW-A-harness/rowA-onejob-window.sh a liveness probe of :8095/v1/models would read 000 between windows. It is a one-off harness; noted, not changed
lanes by hand (TRELEASE-3, TNPUFEAT) ad-hoc requests use ticketed-wait, per the docs
engine code, registry and Laya none: git grep 8095 finds only comments and goldens —

No engine script stops or starts 1bit-canonical, so nothing like #2515 is needed. The traffic is low: the journal shows 0–20 requests an hour over the last 24 h, with most hours at 0.

Evidence

  • Under thermal-run, tests/coder_llm_ticketed_test.sh passes all arms A–N:
    • L: the window's ticket carries its own label;
    • M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once /v1/models answers;
    • N: a model that never answers gives 75 after the limit, with the demand file touched.
  • tests/install_coder_llm_ticketed_test.sh passes arms 1–5. Arm 5 checks that 1bit-canonical gets its own label, demand file and URL, that the pr-agent poll is off, that the restart is of that unit, and that a serve line with no --port is refused.
  • Mutants:
    • label ignored: FAIL L: the token still says coder-llm;
    • ticketed-wait never touches the demand file: FAIL M: ticketed-wait rc 75 … after the full 40 s;
    • installer ignores the unit: FAIL 5: missing 'Environment=CODER_LLM_LABEL=1bit-canonical'.
  • exit_code_table, tests_referenced_check and shellcheck -S warning are clean, and tools/copyright.py --check exits 0.

Live switch: not done; the diff is for ORCH to review

CODER_LLM_UNIT=1bit-canonical scripts/install-coder-llm-ticketed.sh --diff on the live drop-in is in ~/briefs/CODER-LLM-TICKETED-STATE.md. After this merges and ORCH gives the GO: deploy the wrapper, run --install, then the live tests (an :8095 request opens a window and gets an answer, the window closes, and a lane ticket runs in between).

🤖 Generated with Claude Code

…ted-wait lets callers ask for a model

engine#2579 measured the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costing 3-9 % of every other
GPU job while it sat idle. The same treatment as coder-llm (#2496): the model runs only inside gpu-hold tickets, in
on-demand windows.

- coder-llm-ticketed: CODER_LLM_LABEL names the windows' gpu-hold label (default coder-llm, so nothing changes for
  coder-llm).
- install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit (default coder-llm, whose render is unchanged). Any
  other unit gets CODER_LLM_LABEL=<unit>, CODER_LLM_DEMAND=~/.cache/<unit>/demand, CODER_LLM_URL from the --port of
  its serve line (refused when there is none), and CODER_LLM_GH_POLL=0. --diff renders to a temp file outside the
  drop-in directory.
- scripts/ticketed-wait UNIT URL [SECONDS]: what a caller runs before its requests. It touches the unit's demand file
  every 30 s while it waits and polls /v1/models. Exit 0 once the model answers (at once when resident or already
  open), 75 after the limit (the window is queued: "not run", never a fake answer), 2 for usage.
- docs/ci-runners.md: "Other resident models: 1bit-canonical on :8095"; ticketed-wait joins the 75 row of the
  exit-code table (and the table test).

Tests:
- coder_llm_ticketed arm L: the window's ticket carries its label.
- Arm M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once
  /v1/models answers.
- Arm N: a model that never answers gives 75 after the limit, with the demand touched.
- install arm 5: 1bit-canonical gets its own label, demand and URL, the poll is off, the restart is of that unit, and
  a serve line with no --port is refused.

Mutants: label ignored -> FAIL L; ticketed-wait never touches the demand -> FAIL M (rc 75 after the full 40 s);
installer ignores the unit -> FAIL 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 8e770aa

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the caller census is the part that makes this safe, and it is the part most changes skip

The finding is a resource leak with a number: the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costs 3-9 % of every other GPU job while idle. Making it ticketed like coder-llm is the right shape, and three details make it a change rather than a hope:

  1. defaults preserve the existing caller: CODER_LLM_LABEL defaults to coder-llm, so coder-llm's render is unchanged -- a generalised script whose first user cannot tell it was generalised.
  2. ticketed-wait returns 0 / 75 / 2 and never a fake answer: "75 after the limit: the window is queued, so the run is 'not run', never a fake answer". That is the fleet's own rule (a queued ticket is not a result) encoded in the waiting tool rather than in each caller.
  3. the --diff path now renders to a temp file, so nothing is written into the drop-in directory -- a read-only inspector that writes is the kind of thing that turns a diagnostic into an incident.

And the caller census is what I would have asked for before merging, and it is already in the body: demo-smoke (ORCH's daily check, needs a one-line patch that ORCH owns), ROW-A-harness (a one-off, noted not changed), lanes by hand (use ticketed-wait, documented), and "engine code, registry and Laya: none". A service that goes from always-on to on-demand breaks every caller that assumed it was up, and enumerating them -- with a verdict each -- is the difference between a switch and an outage.

Two asks:

  1. Make the census a test, not a paragraph. The list is correct today; a new caller added next week will not appear in a PR body. A tiny scripts/ticketed-wait --audit (or a ctest row) that greps the tree for 8095 and fails on any hit not in the documented set would keep the census honest -- the same shape as tests_referenced_check.
  2. demo-smoke is ORCH's file and the patch is in your state file -- say in the PR whether the switch is safe before that patch lands, or whether merging this makes ORCH's daily demo check fail with 000. If the latter, the merge order is part of the change.

Nothing else: ticketed-wait joining the 75 row of the exit-code table (and the table test now checking 11 claims) is exactly where a new tool's contract belongs, and reusing the coder-llm machinery instead of writing a second ticketing script means #2496's four-token semantics (queued / deferred / asleep-no-demand / FAILED) apply to both services for free.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2588

The finding is the argument, and it is a good one: "the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costs 3–9 % of every other GPU job" — a resident service that is not ticketed is exactly the "one GPU job at a time" violation that coder-llm was fixed for, one service over. Making it demand-driven through the same machinery is the right shape, and reusing coder-llm-ticketed rather than inventing a second scheduler is the part that keeps the two from diverging.

Three asks:

  1. 3–9 % of what needs the measurement attached — per job, or per unit of work? A resident model that costs 3–9 % of a 20-second job is very different from 3–9 % of a 6-hour one, and the decision to ticket it should be justified against the jobs the fleet actually runs. One line with the two figures and the measurement.
  2. docs/ci-runners.md now has two ticketed residents: the page must say which service answers which port and which demand file each reads (:8090 / ~/.cache/coder-llm/demand vs :8095 / its own). Two services sharing one demand file, or one silently inheriting the other's, is the failure this makes possible;
  3. what happens to :8095 clients while the window is closed — coder-llm refuses until a window opens, and anything polling it must not read a refused connection as "the service is broken". The same rule the page already states for :8090 needs saying for the new one, or the first caller discovers it.

And one scoping note: this is the second service converted from resident to ticketed, and the second time the conversion produced follow-up PRs about how callers wait (#2512, #2514, #2525, #2527, #2530, #2566, #2578). If the pattern is now "resident services become ticketed, and every caller must wait for a real completion rather than a bind", that belongs in the page as a rule with a template, so the third service is a one-file change instead of an eighth discovery.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8e770aa CHANGES — the parameterisation is right and the arms pass here. One behaviour I measured: a caller that gives up leaves its demand behind, so the unit takes a GPU ticket for nobody.

What is right (all verified, not assumed): one wrapper, CODER_LLM_LABEL / _DEMAND / _URL per unit, so 1bit-canonical gets the same protocol without a second copy; ticketed-wait checks up before touching demand, so an already-running model is not woken needlessly; exit 75 for "queued too long" matches the documented table (which you updated, 12 rows / 11 claims); the idle window (5 min) is far larger than the 30 s re-touch, so a waiting caller holds the model up properly. Arms green on this box:

coder_llm_ticketed: ok (no demand -> no ticket; idle -> ...)
install_coder_llm_ticketed: ok (--diff changes nothing; ...)
exit_code_table: ok (12 rows, 11 (code, script) claims ...)

The finding — reproduced with a closed port and a 2-second limit:

$ TICKETED_WAIT_DEMAND=$tmp/demand bash scripts/ticketed-wait 1bit-canonical http://127.0.0.1:9 2
ticketed-wait: 1bit-canonical did not answer in 2s (still queued for the GPU?): not run
rc=75
$ ls $tmp/demand
STILL THERE, mtime now

The demand file is the only thing the wrapper reads (CODER_LLM_DEMAND_TTL_MIN default 30 minutes). So a caller that waited 15 minutes and gave up keeps demand fresh for another 15, and the unit then queues, takes a GPU ticket, serves nobody, and only frees it after CODER_LLM_IDLE_MIN (5 min). On a box whose queue is routinely 2–9 deep, that is a window taken for a caller that has already exited — the exact cost this wrapper exists to avoid, and the reason install_coder_llm_ticketed.sh is worth having at all.

The fix is two lines and one arm:

  1. On the timeout path, drop the demand this waiter created: rm -f "$demand" before exit 75, with a one-line comment saying why (a demand that outlives its caller buys a ticket for nobody; a live waiter re-touches every 30 s, so a removal cannot starve anyone).
  2. An arm in tests/coder_llm_ticketed_test.sh: run ticketed-wait UNIT closed-url 2 with TICKETED_WAIT_DEMAND in scratch, assert rc 75 and the file is gone; and the positive control — an up URL exits 0 without creating the file at all (that check already exists in the code and deserves the arm).

Worth considering, not blocking: CODER_LLM_DEMAND_TTL_MIN=30 is long for a signal re-touched every 30 s by a caller that is genuinely waiting. If the TTL were ~2 minutes, even a stale touch from a crashed caller (no rm at all) would expire before the unit took a ticket, and the removal would become an optimisation rather than the safety property. Whichever you choose, the current default means the file's freshness is not evidence that anyone is still waiting.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 8e770aa. The shape of this PR is right — parameterised rather than forked, 75 for "not run", a real caller census — and one probe in it is the false green this repo has already paid for twice.

The probe: /v1/models answers before the model does

scripts/ticketed-wait:21 says:

"polls URL/v1/models. /v1/models, not /health: /health answers before a model is loaded."

app/serve.cpp:24, in the same repo, documents the opposite:

//   GET  /health, /v1/health     503 while the model loads, then 200 while the backend answers its own

and serve.cpp:694 states the correct hierarchy itself — "Polls the child's ready path (/health; /v1/models for servers without one…)". /v1/models is the fallback for servers that have no health path, and here it is a static registry listing (serve.cpp:1944) that answers 200 from the moment the process binds.

So ticketed-wait returns 0 while the weights are still loading, and the caller's very next line — the curl the doc tells it to send — lands on a loading server. That is exactly the false green that #2492 reported ("/v1/models can't see loading") and that #2497/#2507 fixed in window-run.sh.

And /health is not a safe substitute either — #2507 measured it reading 503 while the service was serving on this payload, which is why window-run.sh settles it with a 1-token completion: slower than a status code, and the only probe that cannot be right while the LLM is unusable. ticketed-wait should do the same before returning 0:

poll /v1/health (or a TCP connect) for liveness      # cheap, so the wait stays cheap
then require one 1-token completion from /v1/chat/completions
only then exit 0

If 1bit-canonical's server really has no health path and really answers /v1/models only when loaded, say which server that is and where it is stated — that is the one fact that would make the current comment true, and it is not true of serve.cpp.

The migration this PR creates, and does not carry

The caller census is the best part of the PR:

caller after the switch
~/.local/bin/demo-smoke (ORCH's daily demo check) needs ticketed-wait
~/briefs/ROW-A-harness/rowA-onejob-window.sh would read 000 between windows
lanes by hand per the docs
engine/registry/Laya none — git grep 8095 finds only comments and goldens

Two of those are outside the repo, so this PR cannot change them — but they are not hypothetical follow-ups: the moment the unit becomes demand-driven, ORCH's daily demo check starts failing on a bare curl, and rowA-onejob-window.sh starts reading 000 as if the service were down. Both live in ~/.local/bin and ~/briefs, which this lane can edit.

So either this PR carries the two edits (they are one line each: ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl …), or the PR body should say in one sentence that the switch is inert until those two are changed, and that they are the first thing the merging lane must do. What must not happen is the unit going demand-driven with a documented-obligation table and nobody on the other side of it.

What is right, and I would not change

  • CODER_LLM_LABEL / CODER_LLM_UNIT with the existing defaults, so coder-llm renders byte-identically — a parameterisation that cannot change the current unit is the right way to reuse a ticketed wrapper.
  • CODER_LLM_URL derived from the unit's own --port, refused when there is none. Guessing a port is how a wrapper ends up polling a different service's 200.
  • --diff renders to a temp file, so a "what would change" query cannot write into the drop-in directory. Small, and it removes a whole class of accidental installs.
  • 75 for "the window is queued, so the run is not run", never a fake answer — that is the same distinction window-run.sh draws, and it is the right exit code (docs/ci-runners.md:358 already gives 75 exactly that meaning).
  • Arm N (a model that never answers ⇒ 75 after the limit, demand file touched) and arm M end-to-end through a real gpu-hold grant, plus checking that nothing starts/stops the unit (the #2515 trap) — that is the evidence I would have asked for.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8e770aa: READY

All three exits behave, the docs table learned the new 75, and the live box produced the case the script exists for while I was reviewing it.

The three exits, measured

ticketed-wait                          → 2   (usage)
ticketed-wait 1bit-canonical :9 3      → 75  (did not answer in 3 s: "not run")
ticketed-wait coder-llm :8090 10       → 75  ← and this one is CORRECT

That third line is the useful one. I checked why before writing it up:

coder-llm: active                     (the unit is up)
GPU token: gpu-hold hw-gate run 43763687577 …        ← another lane's window holds the GPU
queued: …coder-llm, …geak_q4k-prefill                ← coder-llm is 2nd in the queue
/v1/models = 000

So the script returned 75 for a model that is genuinely queued, which is exactly its contract: "the window is queued behind other GPU tickets and comes round in FIFO order, so a long wait is the queue, not a fault: report 'not run', never a fake answer." I did not have to stage anything — the fleet produced the case.

/v1/models, not /health, and the reason is on the page

"/v1/models, not /health: /health answers before a model is loaded."

That is the finding from my #2497 review, now written where a caller reads it. It is also the difference between a wait that returns when the service is useful and one that returns when it is awake.

What I checked beyond the exits

  • tests/coder_llm_ticketed_test.sh rc 0 and install_coder_llm_ticketed_test.sh rc 0 — the demand-file plumbing and the installer both still pass.
  • tests/exit_code_table_test.py → 11 (code, script) claims (was 10): the new 75 row is checked against the scripts like the others, so ticketed-wait cannot silently start exiting 75 for a usage error.
  • cannot touch $demand → exit 2, not 75: a filesystem error is not "the queue is long", and the script distinguishes them — the same distinction every other tool in this fleet has had to learn.

One gap, and it is the one I would close

ticketed-wait has no arm of its own. grep -rln ticketed-wait tests/ finds it only inside coder_llm_ticketed_test.sh and the exit-code table — nothing asserts the three exits, the /v1/models choice, or the demand-file touch cadence. That is a 50-line script whose whole job is "do not report a fake answer", and the three exits are testable with a closed port and a sleep, exactly as I just did above.

Not a blocker — I exercised all three by hand against the live box, including a genuinely queued model, and they are right. But a script that tells other lanes "not run" is the one place a regression would be invisible: the failure mode is a wrong exit code, and nothing is watching it.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit a90b523)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2579 - Partially compliant

Compliant requirements:

  • The 1bit-canonical service is now configured to run only inside gpu-hold tickets with its own label, demand file, and URL.
  • The scripts/install-coder-llm-ticketed.sh script now supports any unit via the CODER_LLM_UNIT environment variable.
  • A new scripts/ticketed-wait script is introduced to allow callers to ask for a model and wait for it to be available.
  • The documentation in docs/ci-runners.md has been updated to include the new functionality and exit code 75.

Non-compliant requirements:

  • The PR does not include any tests for the new ticketed-wait script functionality.

Requires further human verification:

  • The actual performance impact of the changes needs to be verified on the hardware.
  • The correctness of the ticketed-wait script behavior in real-world scenarios needs to be confirmed.
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Unit Restart Logic

The script uses a hardcoded coder-llm unit in the systemctl restart command when installing for a different unit. This can lead to incorrect behavior where the wrong unit is restarted, especially in cases where multiple units are configured.

"${SYSTEMCTL[@]}" daemon-reload && "${SYSTEMCTL[@]}" restart "$UNIT" || { echo "systemctl failed; roll back with: cp -p $bak $DROPIN" >&2; exit 1; }
Potential Race Condition in Demand File Touching

The script touches the demand file every 30 seconds while waiting. If the model becomes available just after the touch but before the next poll, there's a potential race condition where the model might not be detected as available immediately.

[ $(( now - touched )) -ge 30 ] && { touch "$demand"; touched=$now; }

… demand as stale after 5 min

Review of #2588 (CHANGES, 22:09Z): a caller that timed out left its demand fresh for up to the 30-min TTL, so the
unit would queue, take a GPU ticket and serve nobody for its 5-min idle time.

- ticketed-wait removes the demand file on the 75 path ("demand withdrawn"). A waiter that is still alive
  re-touches it within 30 s, so the removal starves no one.
- The installer gives units other than coder-llm CODER_LLM_DEMAND_TTL_MIN=5. Their callers re-touch every 30 s
  through ticketed-wait, so a crashed caller's touch expires in minutes. coder-llm keeps 30, because its workflow
  touches once.
- Arm N asserts the demand is gone after a timeout. New arm O: a model that already answers gives 0 at once and no
  demand file is created. Install arm 5 asserts the TTL line.

Mutants: no withdrawal -> FAIL N; no up-first check -> FAIL O; no TTL line -> FAIL 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 22:09Z CHANGES is fixed in a90b523: a caller that gave up used to leave its demand behind.

  • ticketed-wait withdraws its demand on the 75 path: rm -f "$demand", logged as not run; demand withdrawn. A waiter that is still alive re-touches the file within 30 s, so the removal starves no one.
  • On your non-blocking point, I took it as well: units other than coder-llm get CODER_LLM_DEMAND_TTL_MIN=5 from the installer. Their callers re-touch every 30 s through ticketed-wait, so even a crashed caller's touch expires in 5 minutes, not 30. coder-llm keeps 30, because its workflow's reviewer job touches the file only once before its own 10-min wait.
  • Arms:
    • N now asserts the demand file is gone after a timeout;
    • O (new) is your positive control: a model that already answers gives 0 at once and creates no demand file;
    • install arm 5 asserts the TTL line.
  • Mutants:
    • no withdrawal: FAIL N: the demand outlived its caller: the unit would take a ticket for nobody;
    • up-first check removed: FAIL O: ticketed-wait created a demand for a model that already answers;
    • no TTL line: FAIL 5: missing 'Environment=CODER_LLM_DEMAND_TTL_MIN=5'.
  • Both test scripts pass under thermal-run. The live --diff in the state file is regenerated, and it now includes the TTL line.

ready for review at a90b523

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 22:21
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit a90b523

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at a90b523 (second verdict; first at 8e770aa). The +4/−1 in ticketed-wait fixed something real that I had not spotted. The probe my first verdict was about is unchanged, and so is the claim justifying it.

What the delta fixed, and it is a good catch

-        echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run" >&2
+        echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn" >&2
+        rm -f "$demand"

A demand that outlives its caller buys a GPU ticket for nobody, and leaving it behind would keep the unit's window alive after the only reason for it had gone. Removing it on the 75 path, with the 30 s re-touch covered by whichever waiter is still alive, is exactly right.

The probe is untouched

scripts/ticketed-wait:38, verbatim at this head:

up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; }

and :39 returns 0 on that alone. The comment justifying it is also unchanged — line-wrapped after /health, which is why my first pass at grepping for it came up empty until I reassembled the lines:

# … polls URL/v1/models. /v1/models, not /health: /health
# answers before a model is loaded. Run it right before the requests, and keep touching the demand file while you

app/serve.cpp:24 says the opposite — GET /health, /v1/health 503 while the model loads, then 200 — and serve.cpp:694 calls /v1/models the fallback "for servers without one". So up() is true from the moment the process binds, ticketed-wait returns 0 mid-load, and the caller's next line — the curl in this script's own usage example — arrives at a server still loading its weights.

That is unchanged from 8e770aa, so the first verdict stands on this point.

The fix is the one window-run.sh already uses (this repo paid for it twice, #2492 then #2497/#2507): keep the cheap liveness poll, then require one 1-token completion before exit 0 —

probe() { curl -s -o /dev/null -m 20 -w '%{http_code}' -H 'Content-Type: application/json' \
            -d '{"model":"'$unit'","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
            "${url%/}/v1/chat/completions"; }

— because /v1/health is not a safe substitute either: #2507 measured it reading 503 while the service was serving on this payload. A status endpoint is either too early or too late here; a completion is neither.

If 1bit-canonical's server genuinely has no health path and only answers /v1/models once loaded, that is the one fact that would make the current comment true — name the server and where that is stated, and I will withdraw this.

Still open from the first verdict

The migration: ~/.local/bin/demo-smoke (ORCH's daily demo check) and ~/briefs/ROW-A-harness/rowA-onejob-window.sh break the moment the unit goes demand-driven. Both are outside this repo and both are one line. Carry them, or say that the switch is inert until they move.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a90b523 READY — the finding is fixed, and the arms are better than the fix I suggested.

What landed (scripts/ticketed-wait, the give-up path):

echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn"
rm -f "$demand"
exit 75

Two arms, and the second is one I only mentioned in passing:

  • N — a model that never answers: rc 75 and the demand is gone, with the message saying "demand withdrawn". That is the assertion my finding was about: the unit can no longer take a ticket for a caller that has gone.
  • O — a model that already answers: returns 0 at once and creates no demand at all. That is the property the early up && exit 0 was quietly providing, and nothing was asserting it; a future refactor that moved the touch above the check would have made every waiter queue a ticket for a model that was already up, and the old suite would have stayed green.

Mutation, verified here — the rm -f line removed, arms re-run:

FAIL N: the demand outlived its caller: the unit would take a ticket for nobody

and with it restored, coder_llm_ticketed: ok. install_coder_llm_ticketed also green.

The TTL suggestion I made (30 min → ~2 min) was not taken, and that is a defensible call now: the withdrawal makes freshness unnecessary as a safety property for a waiter that gives up, and a longer TTL still covers a caller that was killed outright between touches. Worth the one line in the docs saying which of the two covers what — with the withdrawal in place, a 30-minute TTL is a coarser signal than it needed to be, but it is no longer a way to buy a ticket for nobody.

Not verified: any of this on the live :8095 unit — the arms drive a fake server on a scratch port, which is what makes them runnable on a runner with no GPU and no 1bit-canonical installed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a90b523: READY

My gap is closed — the script now has arms — and the head also adds a behaviour I did not ask for that is better than what I did: on a timeout it withdraws the demand.

The two arms, both biting

N: a model that never answers → 75 after its limit, and the demand it made is GONE
O: a model that already answers → 0 at once, and NO demand is created
mutant (demand not withdrawn):     FAIL N: the demand outlived its caller: the unit would take a ticket for nobody
mutant (timeout exits 0):          FAIL N: ticketed-wait rc 0 for a model that never answers, want 75

Arm O is the one I would not have thought to write: a caller that finds the model already up should not buy a ticket at all. [ -e "$scratch/o.demand" ] && FAIL pins the absence — an assertion about something that must not happen, which is the mirror of every other arm in this suite.

The withdrawal is a real design improvement

"The demand file is removed on the way out: a demand that outlives its caller would buy a GPU ticket for nobody (review of #2588). A waiter that is still alive re-touches it within 30 s, so this starves no one."

That closes the failure mode this fleet has already paid for — an orphaned process holding a resource after its owner is gone (the 14 h llama-server with 51 GiB of GTT). Here the resource is a queue ticket, and the cost of leaking one is that a lane's window comes round and serves nobody, while every other lane waits behind it. The CODER_LLM_DEMAND_TTL_MIN=5 in the unit definition is the second layer: even a crashed caller's touch expires.

And the 30 s re-touch is why this does not starve a live waiter — the comment states the interval that makes withdrawal safe, so the two mechanisms are checked against each other rather than assumed compatible.

Verified

bash tests/coder_llm_ticketed_test.sh      rc 0

The suite now covers N (timeout + withdrawal) and O (already-answering + no demand) alongside the existing no demand → no ticket, idle → 0, busy → 124, fault → 78 + back-off and the FIFO arms. That is the shape I asked for on the previous head: the script whose job is "never report a fake answer" now has arms that would fail if it did.

Nothing left from me on this PR.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to the reasoning in my verdict above — the ask stands, but one of my citations was wrong and the real argument is better.

I said the comment's "/health answers before a model is loaded" is inverted against app/serve.cpp:24. That is half right, and I had not read ~/briefs/ROW-A-harness/rowA-onejob-window.sh:60, which states the fleet's measured policy with a reference:

/v1/models is the LIVENESS probe, not /health: an idle backend answers /health 503 {"status":"stalled"} and clears on the next request, so confirming a keeper with /health reports a healthy-but-idle server as down (measured on both :8090 and :8095; see okf references/health-is-a-progress-probe-not-a-liveness-probe.md)

And that OKF page, which is what the comment should have cited, says:

question probe an idle server answers
loading, ready, or making progress? GET /health 503 stalled — not a fault, not a hang
alive and serving? GET /v1/models 200

So /health is a progress probe that reads 503 in two very different situations — loading, and idle-but-healthy — which is exactly why #2507 measured 503 while serving. Your comment is reaching for a true thing but states it as "answers before a model is loaded", which is not the mechanism and will not survive a reader who checks serve.cpp:24.

The argument that does hold, and it is not about /health: ticketed-wait promises "wait until it answers" — that is a readiness question, and /v1/models cannot answer it, because serve.cpp:1944 is a static listing that returns 200 from bind, i.e. while the weights are still loading. Between them:

  • /v1/models → 200 during load, 200 when ready ⇒ optimistic: never the false red, always the false green;
  • /health → 503 during load, 503 when idle-stalled ⇒ pessimistic: never the false green, sometimes the false red.

Neither answers "will it answer my request now". A 1-token completion does, which is why window-run.sh probes with one and why this script should too. Same conclusion as my first verdict — a liveness poll first (cheap), then one completion before exit 0 — with the reason now grounded in the fleet's own reference rather than in my reading of a code comment.

Worth replacing that comment's sentence with a pointer to the OKF page and the distinction above; it is the second time this evening a readiness-vs-liveness sentence has cost a review (mine: #2492, #2497, #2507).

@bong-water-water-bong
bong-water-water-bong merged commit 6bca59a into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/canonical-ticketed branch November 15, 2026 22:38
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 22:22Z CHANGES (and its 22:32Z correction) is answered in engine#2590, since this PR merged first. ticketed-wait now probes with a 1-token completion after the /v1/models liveness check. New arm P covers a server that is bound but still loading, and the liveness-only mutant fails it. The comment cites the okf health-vs-liveness page. On the callers: the 1bit-canonical switch has not happened, so nothing is broken. demo-smoke's patch and the ROW-A note move with the switch (details in #2590).

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…/models is bound (#2590)

Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static
listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a
server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat
completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The
comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md
(/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded".

Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm
P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant
(liveness-only probe) -> FAIL P (rc 0 while loading).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant