From 20d83e687b492f934ddb45dd225bac261f38471f Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 18:50:47 -0400 Subject: [PATCH] ticketed-wait: "answers" means a completion is answered, not that /v1/models is bound Review of #2588 (CHANGES, 22:22Z/22:32Z; merged first): ticketed-wait returned 0 on GET /v1/models, a static listing that answers 200 from bind while the weights are still loading. The caller's next request then reached a server that could not answer it. Each poll is now /v1/models first (cheap liveness), then one 1-token chat completion, and only a 200 from the completion counts: the probe window-run.sh uses after #2492/#2497/#2507. The comment now explains liveness vs readiness and cites okf references/health-is-a-progress-probe-not-a-liveness-probe.md (/health reads 503 while loading AND when idle-but-healthy), replacing the wrong "answers before a model is loaded". Test: the fake server's completions answer 503 while a "loading" file exists, while /v1/models stays 200. New arm P: ticketed-wait gives 75 while the model is only bound, and 0 once the completion is answered. Mutant (liveness-only probe) -> FAIL P (rc 0 while loading). Co-Authored-By: Claude Opus 5.5 --- docs/ci-runners.md | 5 ++++- scripts/ticketed-wait | 20 ++++++++++++++++---- tests/coder_llm_ticketed_test.sh | 24 +++++++++++++++++++++++- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/docs/ci-runners.md b/docs/ci-runners.md index a1688590a..a0f50b1eb 100644 --- a/docs/ci-runners.md +++ b/docs/ci-runners.md @@ -332,7 +332,10 @@ on HRX, `:8095`) costing 3-9 % of every other GPU job while it sat idle. Tickete ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 || { echo "not run: :8095 queued"; exit 75; } -The script touches the demand file every 30 s while it waits, and polls `/v1/models`. It returns 0 once the model +The script touches the demand file every 30 s while it waits. Each poll is a cheap `GET /v1/models` (liveness), then +one 1-token chat completion (readiness): `/v1/models` is a static listing that answers 200 from bind while the +weights are still loading, and `/health` reads 503 both while loading and when idle (okf +`references/health-is-a-progress-probe-not-a-liveness-probe.md`). It returns 0 once the model answers, at once if the unit is resident or its window is already open, and 75 after the limit. A 75 means the window is queued behind other tickets, so the run is **not run**, never a fake answer. On a 75 it also **removes the demand file**, so a caller that has gone does not buy a GPU ticket for nobody. A unit other than coder-llm also takes demand diff --git a/scripts/ticketed-wait b/scripts/ticketed-wait index 03eb3c29a..dc622058e 100755 --- a/scripts/ticketed-wait +++ b/scripts/ticketed-wait @@ -18,9 +18,15 @@ # # Ask for a model that runs in gpu-hold windows (scripts/coder-llm-ticketed: coder-llm on :8090, 1bit-canonical on # :8095; engine#2494, #2579) and wait until it answers, bounded. It touches ~/.cache/UNIT/demand (the file the -# unit's window waits for) every 30 s while it waits, and polls URL/v1/models. /v1/models, not /health: /health -# answers before a model is loaded. Run it right before the requests, and keep touching the demand file while you -# work if that takes longer than the unit's idle time (5 min), or the window closes under you. +# unit's window waits for) every 30 s while it waits. Run it right before the requests, and keep touching the demand +# file while you work if that takes longer than the unit's idle time (5 min), or the window closes under you. +# +# "Answers" is READINESS, and only a completion shows it (review of #2588; the same probe window-run.sh uses after +# #2492/#2497/#2507). Each poll is a cheap GET /v1/models first (liveness: is anything bound there?), then one 1-token +# chat completion, and only a 200 from the completion counts. /v1/models alone is a static listing that answers 200 +# from bind, while the weights are still loading. /health is a PROGRESS probe: it reads 503 both while loading and +# when idle-but-healthy (okf references/health-is-a-progress-probe-not-a-liveness-probe.md). Neither says whether +# YOUR request will be answered now. # # ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl ... http://127.0.0.1:8095/v1/chat/completions # @@ -35,7 +41,13 @@ unit=${1:-}; url=${2:-}; limit=${3:-900} [ -n "$unit" ] && [ -n "$url" ] && [ "$limit" -ge 0 ] 2> /dev/null || { echo "usage: ticketed-wait UNIT URL [SECONDS]" >&2; exit 2; } demand=${TICKETED_WAIT_DEMAND:-$HOME/.cache/$unit/demand} poll=${TICKETED_WAIT_POLL_S:-5} -up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; } +model=${TICKETED_WAIT_MODEL:-m} # 1bit serve answers any model name with its loaded model +up() { + [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ] || return 1 + [ "$(curl -s -o /dev/null -m 30 -w '%{http_code}' -H 'Content-Type: application/json' \ + -d '{"model":"'"$model"'","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \ + "${url%/}/v1/chat/completions" 2> /dev/null)" = 200 ] +} up && exit 0 mkdir -p "$(dirname "$demand")" && touch "$demand" || { echo "ticketed-wait: cannot touch $demand" >&2; exit 2; } start=$(date +%s); touched=$start diff --git a/tests/coder_llm_ticketed_test.sh b/tests/coder_llm_ticketed_test.sh index e62135396..3f1b92bbc 100755 --- a/tests/coder_llm_ticketed_test.sh +++ b/tests/coder_llm_ticketed_test.sh @@ -46,6 +46,9 @@ # N ticketed-wait on a model that never answers gives up with 75 after its limit and WITHDRAWS the demand it made, # so the unit does not take a ticket for a caller that has gone (review of #2588); # O ticketed-wait on a model that already answers returns 0 at once and creates no demand at all. +# P readiness, not liveness (review of #2588): a server that lists /v1/models (200 from bind) but cannot complete yet +# (503 while "loading") does not count as answering: ticketed-wait waits, and returns 0 only once a 1-token +# completion is answered. set -u root=$(cd "$(dirname "$0")/.." && pwd) # the defaults work from any directory, not only the repo root tick=${1:-$root/scripts/coder-llm-ticketed} @@ -71,6 +74,14 @@ class H(http.server.BaseHTTPRequestHandler): busy = open("$scratch/state").read().strip() == "busy" body = json.dumps([{"id": 0, "is_processing": busy}]).encode() self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(body) + def do_POST(self): + # a completion answers only once "loaded": a $scratch/loading file makes it 503 (weights still loading), while + # GET /v1/models above keeps answering 200 from bind, as 1bit serve's static listing does + self.rfile.read(int(self.headers.get("Content-Length", 0))) + if os.path.exists("$scratch/loading"): + self.send_response(503); self.end_headers(); return + body = json.dumps({"choices": [{"message": {"role": "assistant", "content": "ok"}}]}).encode() + self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(body) def log_message(self, *a): pass http.server.ThreadingHTTPServer(("127.0.0.1", $port), H).serve_forever() FAKE @@ -278,9 +289,20 @@ env TICKETED_WAIT_DEMAND="$scratch/o.demand" TICKETED_WAIT_POLL_S=1 "$waiter" te kill "$of" 2> /dev/null; wait "$of" 2> /dev/null [ $(( $(date +%s) - t0 )) -le 10 ] || { echo "FAIL N: ticketed-wait overran its 3 s limit"; fail=1; } +# ---- P: bound but still loading is not "answers" --------------------------------------------------------------------- +echo idle > "$scratch/state"; rm -f "$scratch/p.demand"; touch "$scratch/loading" +"$scratch/fake" > /dev/null 2>&1 & pf=$!; pids+=($pf) +wait_for 10 curl -sf -o /dev/null "http://127.0.0.1:$port/v1/models" || { echo "FAIL P: the fake did not come up"; fail=1; } +env TICKETED_WAIT_DEMAND="$scratch/p.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$port" 3 2> "$scratch/p.wait"; rc=$? +[ "$rc" = 75 ] || { echo "FAIL P: ticketed-wait rc $rc while the model was still loading (only /v1/models answered), want 75"; fail=1; } +( sleep 2; rm -f "$scratch/loading" ) & +env TICKETED_WAIT_DEMAND="$scratch/p.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$port" 20 2> "$scratch/p2.wait"; rc=$? +[ "$rc" = 0 ] && grep -q 'answers after' "$scratch/p2.wait" || { echo "FAIL P: rc $rc once loaded, want 0 after a wait ($(tail -1 "$scratch/p2.wait"))"; fail=1; } +kill "$pf" 2> /dev/null; wait "$pf" 2> /dev/null; rm -f "$scratch/loading" + # ---- G: the default queue wait keeps the ticket's place on a busy box (a give-up re-queues at the back) ------------ sed -n 's/^QUEUE_WAIT=\${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw" [ "$(cat "$scratch/qw")" -ge 14400 ] 2> /dev/null || { echo "FAIL G: default queue wait $(cat "$scratch/qw")s; under 4 h a busy queue makes it re-queue at the back and starve"; fail=1; } -[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits, gives up with 75 and withdraws its demand, and asks for nothing when the model is up)" +[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits, gives up with 75 and withdraws its demand, and asks for nothing when the model is up; bound-but-loading is not an answer)" exit "$fail"