Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/ci-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,10 @@ on HRX, `:8095`) costing 3-9 % of every other GPU job while it sat idle. Tickete

ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 || { echo "not run: :8095 queued"; exit 75; }

The script touches the demand file every 30 s while it waits, and polls `/v1/models`. It returns 0 once the model
The script touches the demand file every 30 s while it waits. Each poll is a cheap `GET /v1/models` (liveness), then
one 1-token chat completion (readiness): `/v1/models` is a static listing that answers 200 from bind while the
weights are still loading, and `/health` reads 503 both while loading and when idle (okf
`references/health-is-a-progress-probe-not-a-liveness-probe.md`). It returns 0 once the model
answers, at once if the unit is resident or its window is already open, and 75 after the limit. A 75 means the window
is queued behind other tickets, so the run is **not run**, never a fake answer. On a 75 it also **removes the demand
file**, so a caller that has gone does not buy a GPU ticket for nobody. A unit other than coder-llm also takes demand
Expand Down
20 changes: 16 additions & 4 deletions scripts/ticketed-wait
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,15 @@
#
# Ask for a model that runs in gpu-hold windows (scripts/coder-llm-ticketed: coder-llm on :8090, 1bit-canonical on
# :8095; engine#2494, #2579) and wait until it answers, bounded. It touches ~/.cache/UNIT/demand (the file the
# unit's window waits for) every 30 s while it waits, and polls URL/v1/models. /v1/models, not /health: /health
# answers before a model is loaded. Run it right before the requests, and keep touching the demand file while you
# work if that takes longer than the unit's idle time (5 min), or the window closes under you.
# unit's window waits for) every 30 s while it waits. Run it right before the requests, and keep touching the demand
# file while you work if that takes longer than the unit's idle time (5 min), or the window closes under you.
#
# "Answers" is READINESS, and only a completion shows it (review of #2588; the same probe window-run.sh uses after
# #2492/#2497/#2507). Each poll is a cheap GET /v1/models first (liveness: is anything bound there?), then one 1-token
# chat completion, and only a 200 from the completion counts. /v1/models alone is a static listing that answers 200
# from bind, while the weights are still loading. /health is a PROGRESS probe: it reads 503 both while loading and
# when idle-but-healthy (okf references/health-is-a-progress-probe-not-a-liveness-probe.md). Neither says whether
# YOUR request will be answered now.
#
# ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl ... http://127.0.0.1:8095/v1/chat/completions
#
Expand All @@ -35,7 +41,13 @@ unit=${1:-}; url=${2:-}; limit=${3:-900}
[ -n "$unit" ] && [ -n "$url" ] && [ "$limit" -ge 0 ] 2> /dev/null || { echo "usage: ticketed-wait UNIT URL [SECONDS]" >&2; exit 2; }
demand=${TICKETED_WAIT_DEMAND:-$HOME/.cache/$unit/demand}
poll=${TICKETED_WAIT_POLL_S:-5}
up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; }
model=${TICKETED_WAIT_MODEL:-m} # 1bit serve answers any model name with its loaded model
up() {
[ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ] || return 1
[ "$(curl -s -o /dev/null -m 30 -w '%{http_code}' -H 'Content-Type: application/json' \
-d '{"model":"'"$model"'","messages":[{"role":"user","content":"ok"}],"max_tokens":1,"temperature":0}' \
"${url%/}/v1/chat/completions" 2> /dev/null)" = 200 ]
}
up && exit 0
mkdir -p "$(dirname "$demand")" && touch "$demand" || { echo "ticketed-wait: cannot touch $demand" >&2; exit 2; }
start=$(date +%s); touched=$start
Expand Down
24 changes: 23 additions & 1 deletion tests/coder_llm_ticketed_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@
# N ticketed-wait on a model that never answers gives up with 75 after its limit and WITHDRAWS the demand it made,
# so the unit does not take a ticket for a caller that has gone (review of #2588);
# O ticketed-wait on a model that already answers returns 0 at once and creates no demand at all.
# P readiness, not liveness (review of #2588): a server that lists /v1/models (200 from bind) but cannot complete yet
# (503 while "loading") does not count as answering: ticketed-wait waits, and returns 0 only once a 1-token
# completion is answered.
set -u
root=$(cd "$(dirname "$0")/.." && pwd) # the defaults work from any directory, not only the repo root
tick=${1:-$root/scripts/coder-llm-ticketed}
Expand All @@ -71,6 +74,14 @@ class H(http.server.BaseHTTPRequestHandler):
busy = open("$scratch/state").read().strip() == "busy"
body = json.dumps([{"id": 0, "is_processing": busy}]).encode()
self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(body)
def do_POST(self):
# a completion answers only once "loaded": a $scratch/loading file makes it 503 (weights still loading), while
# GET /v1/models above keeps answering 200 from bind, as 1bit serve's static listing does
self.rfile.read(int(self.headers.get("Content-Length", 0)))
if os.path.exists("$scratch/loading"):
self.send_response(503); self.end_headers(); return
body = json.dumps({"choices": [{"message": {"role": "assistant", "content": "ok"}}]}).encode()
self.send_response(200); self.send_header("Content-Type", "application/json"); self.end_headers(); self.wfile.write(body)
def log_message(self, *a): pass
http.server.ThreadingHTTPServer(("127.0.0.1", $port), H).serve_forever()
FAKE
Expand Down Expand Up @@ -278,9 +289,20 @@ env TICKETED_WAIT_DEMAND="$scratch/o.demand" TICKETED_WAIT_POLL_S=1 "$waiter" te
kill "$of" 2> /dev/null; wait "$of" 2> /dev/null
[ $(( $(date +%s) - t0 )) -le 10 ] || { echo "FAIL N: ticketed-wait overran its 3 s limit"; fail=1; }

# ---- P: bound but still loading is not "answers" ---------------------------------------------------------------------
echo idle > "$scratch/state"; rm -f "$scratch/p.demand"; touch "$scratch/loading"
"$scratch/fake" > /dev/null 2>&1 & pf=$!; pids+=($pf)
wait_for 10 curl -sf -o /dev/null "http://127.0.0.1:$port/v1/models" || { echo "FAIL P: the fake did not come up"; fail=1; }
env TICKETED_WAIT_DEMAND="$scratch/p.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$port" 3 2> "$scratch/p.wait"; rc=$?
[ "$rc" = 75 ] || { echo "FAIL P: ticketed-wait rc $rc while the model was still loading (only /v1/models answered), want 75"; fail=1; }
( sleep 2; rm -f "$scratch/loading" ) &
env TICKETED_WAIT_DEMAND="$scratch/p.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$port" 20 2> "$scratch/p2.wait"; rc=$?
[ "$rc" = 0 ] && grep -q 'answers after' "$scratch/p2.wait" || { echo "FAIL P: rc $rc once loaded, want 0 after a wait ($(tail -1 "$scratch/p2.wait"))"; fail=1; }
kill "$pf" 2> /dev/null; wait "$pf" 2> /dev/null; rm -f "$scratch/loading"

# ---- G: the default queue wait keeps the ticket's place on a busy box (a give-up re-queues at the back) ------------
sed -n 's/^QUEUE_WAIT=\${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw"
[ "$(cat "$scratch/qw")" -ge 14400 ] 2> /dev/null || { echo "FAIL G: default queue wait $(cat "$scratch/qw")s; under 4 h a busy queue makes it re-queue at the back and starve"; fail=1; }

[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits, gives up with 75 and withdraws its demand, and asks for nothing when the model is up)"
[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits, gives up with 75 and withdraws its demand, and asks for nothing when the model is up; bound-but-loading is not an answer)"
exit "$fail"
Loading