From 8e770aa7316e3f65e228d9b87ee9a48ca096eaee Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 18:00:47 -0400 Subject: [PATCH 1/2] 1bit-canonical runs in gpu-hold windows like coder-llm; scripts/ticketed-wait lets callers ask for a model engine#2579 measured the resident 1bit-canonical.service (Qwen3-4B on HRX, :8095) costing 3-9 % of every other GPU job while it sat idle. The same treatment as coder-llm (#2496): the model runs only inside gpu-hold tickets, in on-demand windows. - coder-llm-ticketed: CODER_LLM_LABEL names the windows' gpu-hold label (default coder-llm, so nothing changes for coder-llm). - install-coder-llm-ticketed.sh: CODER_LLM_UNIT picks the unit (default coder-llm, whose render is unchanged). Any other unit gets CODER_LLM_LABEL=, CODER_LLM_DEMAND=~/.cache//demand, CODER_LLM_URL from the --port of its serve line (refused when there is none), and CODER_LLM_GH_POLL=0. --diff renders to a temp file outside the drop-in directory. - scripts/ticketed-wait UNIT URL [SECONDS]: what a caller runs before its requests. It touches the unit's demand file every 30 s while it waits and polls /v1/models. Exit 0 once the model answers (at once when resident or already open), 75 after the limit (the window is queued: "not run", never a fake answer), 2 for usage. - docs/ci-runners.md: "Other resident models: 1bit-canonical on :8095"; ticketed-wait joins the 75 row of the exit-code table (and the table test). Tests: - coder_llm_ticketed arm L: the window's ticket carries its label. - Arm M: ticketed-wait asks a waiting wrapper for the model, the real gpu-hold grants a window, and it returns 0 once /v1/models answers. - Arm N: a model that never answers gives 75 after the limit, with the demand touched. - install arm 5: 1bit-canonical gets its own label, demand and URL, the poll is off, the restart is of that unit, and a serve line with no --port is refused. Mutants: label ignored -> FAIL L; ticketed-wait never touches the demand -> FAIL M (rc 75 after the full 40 s); installer ignores the unit -> FAIL 5. Co-Authored-By: Claude Opus 5.5 --- docs/ci-runners.md | 21 +++++++++- scripts/coder-llm-ticketed | 8 +++- scripts/install-coder-llm-ticketed.sh | 52 +++++++++++++++++++----- scripts/ticketed-wait | 50 +++++++++++++++++++++++ tests/coder_llm_ticketed_test.sh | 43 +++++++++++++++++++- tests/exit_code_table_test.py | 3 +- tests/install_coder_llm_ticketed_test.sh | 28 ++++++++++++- 7 files changed, 185 insertions(+), 20 deletions(-) create mode 100755 scripts/ticketed-wait diff --git a/docs/ci-runners.md b/docs/ci-runners.md index dd5864c74..35f0538d4 100644 --- a/docs/ci-runners.md +++ b/docs/ci-runners.md @@ -319,7 +319,24 @@ A silent `:8090` is therefore **not** evidence of a failed restart on its own purpose, and a window that reports them as failures is how lanes learn to ignore failures (#2496, #2547, #2566). `--restart-rc` encodes the same table, and its self-test asserts each row. -`scripts/install-coder-llm-ticketed.sh` does the switch: +### Other resident models: 1bit-canonical on :8095 (engine#2579) + +The same wrapper runs any model that used to stay resident. engine#2579 measured `1bit-canonical.service` (Qwen3-4B +on HRX, `:8095`) costing 3-9 % of every other GPU job while it sat idle. Ticketed, it gets its own gpu-hold label +(`1bit-canonical`), its own demand file (`~/.cache/1bit-canonical/demand`) and its own URL, read from its serve line's +`--port`. The pr-agent poll is coder-llm's alone. + + CODER_LLM_UNIT=1bit-canonical scripts/install-coder-llm-ticketed.sh --diff # then --install, --status + +**A caller asks before it requests**, with `scripts/ticketed-wait UNIT URL [SECONDS]`: + + ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 || { echo "not run: :8095 queued"; exit 75; } + +The script touches the demand file every 30 s while it waits, and polls `/v1/models`. It returns 0 once the model +answers, at once if the unit is resident or its window is already open, and 75 after the limit. A 75 means the window +is queued behind other tickets, so the run is **not run**, never a fake answer. A caller that works longer than the +unit's idle time (5 min) keeps touching the demand file, or the window closes under it. + - with no argument, or `--diff`, it prints the drop-in change; - `--status` says whether the wrapper is installed and the running unit executes it (exit 0 only when both are true); @@ -355,7 +372,7 @@ Each script's own header stays the reference for its edge cases. | 2 | one code, four causes; **the stderr line says which** and is part of the contract (each phrase below is what the script prints, so grep your log for it): (a) a **usage error**, `gpu-hold: unknown option` or `gpu-hold: no command given`: the command line is wrong, not the box; (b) **could not run / INCOMPLETE**, `INCOMPLETE (an input was absent: nothing was tested)`: an input was absent (a model, text, a binary, a checkout) or a build or memory floor refused it. hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question; (c) the box could not set up the ticket, `gpu-hold: cannot create` or `gpu-hold: cannot write`; (d) a **policy refusal**, `npu-hold: REFUSED:` (a shared queue while npu-serve-keeper reads it) | gpu-hold and hw-gate.sh exit it; npu-hold exits (d) | **read stderr first**. (a) and (d): fix the command or config, because re-queueing repeats the same mistake; (b): after providing the input; (c): after fixing the box | | 3 | the server under test never came up | hw-gate.sh exits it from its server-start step | read the server log | | 69 | the job ran, but coder-llm did not answer `:8090` after the window | window-run.sh (computed by `final_rc`, not a literal exit) | see "The PR-Agent model runs in gpu-hold windows" | -| 75 | the queue never came round within `-w` | gpu-hold exits it; npu-hold and coder-llm-ticketed pass it through | yes, but **at the back** of the queue. That is why coder-llm queues with a 24 h `-w`, and why a lane should size `-w` for the queue instead of treating 75 as a free retry | +| 75 | the queue never came round within `-w`, or a ticketed model did not answer within `ticketed-wait`'s limit | gpu-hold exits it; ticketed-wait exits it; npu-hold and coder-llm-ticketed pass it through | yes, but **at the back** of the queue. That is why coder-llm queues with a 24 h `-w`, and why a lane should size `-w` for the queue instead of treating 75 as a free retry | | 76 | the memory pre-check refused the run | hw-gate.sh `--mem` exits it | when MemAvailable is back | | **78** | a GPU/NPU fault line in the kernel log: **during** the hold (the hold was ended) or **at take time** (the ticket was refused, nothing ran). The coder-llm journal says which one | gpu-hold exits it; npu-hold and coder-llm-ticketed pass it through (the coder-llm journal names the cause) | after the fault is understood. Never in a loop | | **124** | the max hold (`-t`) ended the job: a deadline, and planned. coder-llm-ticketed also maps a KILL after its max hold to 124 | gpu-hold sets it (its own code, after `timeout(1)`'s convention; it also maps a TERM at its deadline from 143); npu-hold passes it through; coder-llm-ticketed passes it through and relabels a post-deadline 137 to it | yes | diff --git a/scripts/coder-llm-ticketed b/scripts/coder-llm-ticketed index 4e973c627..81dac6170 100755 --- a/scripts/coder-llm-ticketed +++ b/scripts/coder-llm-ticketed @@ -62,6 +62,9 @@ # so the window can end its own server before gpu-hold KILLs the window; an explicit # GPU_HOLD_KILL_GRACE in the environment wins # CODER_LLM_GPU_HOLD gpu-hold to use, default the one on PATH +# CODER_LLM_LABEL the gpu-hold label of the windows, default coder-llm. The same wrapper runs other resident +# models the same way (1bit-canonical on :8095, engine#2579): one label, demand file and +# URL per unit, set by scripts/install-coder-llm-ticketed.sh (CODER_LLM_UNIT) # CODER_LLM_ALWAYS=1 skip the demand wait (a resident window, still ticketed and still max-hold bounded) # # Internal: "coder-llm-ticketed --window SERVER_CMD..." is what gpu-hold runs; it is the idle-aware window itself. @@ -94,6 +97,7 @@ BACKOFF=${CODER_LLM_FAULT_BACKOFF_S:-600} STOP_WAIT=${CODER_LLM_STOP_WAIT_S:-30} KILL_GRACE=${GPU_HOLD_KILL_GRACE:-${CODER_LLM_KILL_GRACE_S:-$(( STOP_WAIT + 10 ))}} GPU_HOLD=${CODER_LLM_GPU_HOLD:-gpu-hold} +LABEL=${CODER_LLM_LABEL:-coder-llm} log() { echo "coder-llm-ticketed[$$] $*"; } @@ -204,12 +208,12 @@ holder=$(head -c 120 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1 # inside its stop wait): say so, do not override the operator's choice [ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] 2> /dev/null \ || log "WARNING: the kill grace ${KILL_GRACE}s is not at least 5 s longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: gpu-hold will KILL the window before the server can exit" -log "demand: queueing for the GPU (gpu-hold -l coder-llm -t $MAX_HOLD -w $QUEUE_WAIT)${holder:+; queued behind $holder}" +log "demand: queueing for the GPU (gpu-hold -l $LABEL -t $MAX_HOLD -w $QUEUE_WAIT)${holder:+; queued behind $holder}" # gpu-hold's default 10 s grace KILLed the window at the first live max hold while the server was still exiting: # rc 137 and an unclean stop instead of 124 (11-15 10:56Z) export CODER_LLM_WINDOW_STAMP="${TMPDIR:-/tmp}/coder-llm-window.$$" rm -f "$CODER_LLM_WINDOW_STAMP" -GPU_HOLD_KILL_GRACE=$KILL_GRACE "$GPU_HOLD" -t "$MAX_HOLD" -l coder-llm -w "$QUEUE_WAIT" -- "$0" --window "$@" +GPU_HOLD_KILL_GRACE=$KILL_GRACE "$GPU_HOLD" -t "$MAX_HOLD" -l "$LABEL" -w "$QUEUE_WAIT" -- "$0" --window "$@" rc=$? # a KILL once the window has been open for the max hold is still the max hold (as 143 is, which gpu-hold maps to # 124); timed from the window's own start, so a long queue wait followed by some other KILL is not relabelled diff --git a/scripts/install-coder-llm-ticketed.sh b/scripts/install-coder-llm-ticketed.sh index 7c956af78..a70835c0d 100755 --- a/scripts/install-coder-llm-ticketed.sh +++ b/scripts/install-coder-llm-ticketed.sh @@ -25,12 +25,20 @@ # --install back up the drop-in as .bak-, write the new one, install the wrapper into ~/.local/bin, # daemon-reload, restart the unit, print --status. Already ticketed: nothing to do, exit 0. # -# Environment (tests): CODER_LLM_UNIT_DIR (default ~/.config/systemd/user/coder-llm.service.d), CODER_LLM_DROPIN +# CODER_LLM_UNIT picks the unit, default coder-llm. Any other resident model is ticketed the same way, with its own +# label, demand file and URL (engine#2579: 1bit-canonical on :8095 cost 3-9 % of every other GPU job while idle): +# CODER_LLM_UNIT=1bit-canonical install-coder-llm-ticketed.sh --diff +# For a unit other than coder-llm the drop-in gets CODER_LLM_LABEL=, CODER_LLM_DEMAND=~/.cache//demand, +# CODER_LLM_URL=http://127.0.0.1: (refused when there is none) and CODER_LLM_GH_POLL=0 +# (the pr-agent fallback is coder-llm's only). Its callers ask for it with scripts/ticketed-wait. +# +# Environment (tests): CODER_LLM_UNIT_DIR (default ~/.config/systemd/user/.service.d), CODER_LLM_DROPIN # (default zz-c22-selfcontained.conf), CODER_LLM_BIN_DIR (default ~/.local/bin), CODER_LLM_SYSTEMCTL (default # "systemctl --user"). set -u here=$(cd "$(dirname "$0")" && pwd) -UNIT_DIR=${CODER_LLM_UNIT_DIR:-$HOME/.config/systemd/user/coder-llm.service.d} +UNIT=${CODER_LLM_UNIT:-coder-llm} +UNIT_DIR=${CODER_LLM_UNIT_DIR:-$HOME/.config/systemd/user/$UNIT.service.d} DROPIN=$UNIT_DIR/${CODER_LLM_DROPIN:-zz-c22-selfcontained.conf} BIN_DIR=${CODER_LLM_BIN_DIR:-$HOME/.local/bin} read -r -a SYSTEMCTL <<< "${CODER_LLM_SYSTEMCTL:-systemctl --user}" @@ -38,13 +46,32 @@ WRAPPER=$BIN_DIR/coder-llm-ticketed mode=${1:---diff} # the new drop-in, from the live one: env lines before the final "ExecStart=" reset, wrapper in front of the command +# the serve line's --port, for a unit's own URL (coder-llm keeps the wrapper's :8090 default) +unit_port() { awk '/^ExecStart=./ { l = $0 } END { if (match(l, /--port[ =][0-9]+/)) { p = substr(l, RSTART, RLENGTH); sub(/--port[ =]/, "", p); print p } }' "$DROPIN"; } + render() { - awk -v w="$WRAPPER" -v b="$BIN_DIR" ' + local extra="" port + if [ "$UNIT" != coder-llm ]; then + port=$(unit_port) + [ -n "$port" ] || { echo "no --port on $UNIT's serve line in $DROPIN: refusing to guess its URL" >&2; return 2; } + extra="Environment=CODER_LLM_LABEL=$UNIT +Environment=CODER_LLM_DEMAND=$HOME/.cache/$UNIT/demand +Environment=CODER_LLM_URL=http://127.0.0.1:$port" + fi + awk -v w="$WRAPPER" -v b="$BIN_DIR" -v unit="$UNIT" -v extra="$extra" ' /^ExecStart=$/ && !done { - print "# 2026-11-15 CODER-LLM-TICKETED (engine#2494): the model runs ONLY inside gpu-hold tickets, in on-demand windows" - print "# (label coder-llm, max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged." - print "# `systemctl --user start coder-llm` now means QUEUED, not \":8090 answers\" (docs/ci-runners.md)." - print "Environment=CODER_LLM_GH_POLL=1" + if (unit == "coder-llm") { + print "# 2026-11-15 CODER-LLM-TICKETED (engine#2494): the model runs ONLY inside gpu-hold tickets, in on-demand windows" + print "# (label coder-llm, max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged." + print "# `systemctl --user start coder-llm` now means QUEUED, not \":8090 answers\" (docs/ci-runners.md)." + print "Environment=CODER_LLM_GH_POLL=1" + } else { + print "# engine#2579: " unit " runs ONLY inside gpu-hold tickets, in on-demand windows, like coder-llm (label " unit "," + print "# max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged. Callers ask for it" + print "# with scripts/ticketed-wait; `systemctl --user start " unit "` now means QUEUED (docs/ci-runners.md)." + print extra + print "Environment=CODER_LLM_GH_POLL=0" + } print "Environment=CODER_LLM_GPU_HOLD=" b "/gpu-hold" print "Environment=PATH=" b ":/usr/local/bin:/usr/bin:/bin" print "Environment=CODER_LLM_MAX_HOLD_MIN=25" @@ -71,7 +98,7 @@ status() { echo "wrapper: NOT installed at $WRAPPER"; ok=1 fi if is_ticketed_file; then echo "drop-in: $DROPIN runs the wrapper"; else echo "drop-in: $DROPIN does NOT run the wrapper"; ok=1; fi - exec_line=$("${SYSTEMCTL[@]}" show coder-llm -p ExecStart --value 2> /dev/null | head -1) + exec_line=$("${SYSTEMCTL[@]}" show "$UNIT" -p ExecStart --value 2> /dev/null | head -1) case "$exec_line" in *coder-llm-ticketed*) echo "running unit: executes coder-llm-ticketed" ;; *) echo "running unit: does NOT execute coder-llm-ticketed (${exec_line:0:120})"; ok=1 ;; @@ -85,7 +112,10 @@ status() { case "$mode" in --diff) if is_ticketed_file; then echo "already ticketed: $DROPIN"; exit 0; fi - diff -u "$DROPIN" <(render) | sed "2s|/dev/fd/[0-9]*|$DROPIN (new)|" + tmp=$(mktemp) || exit 2 + render > "$tmp" || { rm -f "$tmp"; exit 2; } + diff -u "$DROPIN" "$tmp" | sed "2s|$tmp|$DROPIN (new)|" + rm -f "$tmp" exit 0 ;; --status) status; exit $? ;; @@ -95,10 +125,10 @@ case "$mode" in bak=$DROPIN.bak-$(date +%Y%m%d) [ -e "$bak" ] && bak=$bak-$(date +%H%M%S) cp -p "$DROPIN" "$bak" || exit 1 - render > "$DROPIN.new" && mv "$DROPIN.new" "$DROPIN" || exit 1 + render > "$DROPIN.new" && mv "$DROPIN.new" "$DROPIN" || { rm -f "$DROPIN.new"; exit 1; } mkdir -p "$BIN_DIR" && install -m 0755 "$here/coder-llm-ticketed" "$WRAPPER" || exit 1 echo "backed up: $bak" - "${SYSTEMCTL[@]}" daemon-reload && "${SYSTEMCTL[@]}" restart coder-llm || { echo "systemctl failed; roll back with: cp -p $bak $DROPIN" >&2; exit 1; } + "${SYSTEMCTL[@]}" daemon-reload && "${SYSTEMCTL[@]}" restart "$UNIT" || { echo "systemctl failed; roll back with: cp -p $bak $DROPIN" >&2; exit 1; } status; exit $? ;; *) echo "usage: $0 [--diff | --status | --install]" >&2; exit 2 ;; diff --git a/scripts/ticketed-wait b/scripts/ticketed-wait new file mode 100755 index 000000000..76852591f --- /dev/null +++ b/scripts/ticketed-wait @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# ticketed-wait UNIT URL [SECONDS] +# +# Ask for a model that runs in gpu-hold windows (scripts/coder-llm-ticketed: coder-llm on :8090, 1bit-canonical on +# :8095; engine#2494, #2579) and wait until it answers, bounded. It touches ~/.cache/UNIT/demand (the file the +# unit's window waits for) every 30 s while it waits, and polls URL/v1/models. /v1/models, not /health: /health +# answers before a model is loaded. Run it right before the requests, and keep touching the demand file while you +# work if that takes longer than the unit's idle time (5 min), or the window closes under you. +# +# ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl ... http://127.0.0.1:8095/v1/chat/completions +# +# Exit 0: the model answers (at once, when the unit is resident or its window is already open). +# Exit 75: it did not answer within SECONDS (default 900). The window is queued behind other GPU tickets and comes +# round in FIFO order, so a long wait is the queue, not a fault: report "not run", never a fake answer. +# Exit 2: usage. +set -u +unit=${1:-}; url=${2:-}; limit=${3:-900} +[ -n "$unit" ] && [ -n "$url" ] && [ "$limit" -ge 0 ] 2> /dev/null || { echo "usage: ticketed-wait UNIT URL [SECONDS]" >&2; exit 2; } +demand=${TICKETED_WAIT_DEMAND:-$HOME/.cache/$unit/demand} +poll=${TICKETED_WAIT_POLL_S:-5} +up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; } +up && exit 0 +mkdir -p "$(dirname "$demand")" && touch "$demand" || { echo "ticketed-wait: cannot touch $demand" >&2; exit 2; } +start=$(date +%s); touched=$start +echo "ticketed-wait: asked for $unit (touched $demand); waiting up to ${limit}s for $url; GPU token: $(head -c 80 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1)" >&2 +while :; do + now=$(date +%s) + if [ $(( now - start )) -ge "$limit" ]; then + echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run" >&2 + exit 75 + fi + [ $(( now - touched )) -ge 30 ] && { touch "$demand"; touched=$now; } + sleep "$poll" + up && { echo "ticketed-wait: $unit answers after $(( $(date +%s) - start ))s" >&2; touch "$demand"; exit 0; } +done diff --git a/tests/coder_llm_ticketed_test.sh b/tests/coder_llm_ticketed_test.sh index 2ff125dd3..897dfd0cf 100755 --- a/tests/coder_llm_ticketed_test.sh +++ b/tests/coder_llm_ticketed_test.sh @@ -38,7 +38,12 @@ # window's start, not the queue's, so a real KILL is not reported as a max hold; every outcome line says how long # the window was open; # K a fault line already in the kernel log makes gpu-hold refuse the grant (rc 78): the journal says "ticket refused -# before the window", not a fault during one (arm D says "fault during the window"), and no server starts. +# before the window", not a fault during one (arm D says "fault during the window"), and no server starts; +# L CODER_LLM_LABEL names the window's ticket (another resident model, 1bit-canonical, runs in its own windows; +# engine#2579); +# M scripts/ticketed-wait asks a waiting unit for the model and returns 0 once it answers: it touches the demand +# file, the wrapper queues and opens a window, and /v1/models answers; +# N ticketed-wait on a model that never answers gives up with 75 after its limit, having touched the demand file. set -u root=$(cd "$(dirname "$0")/.." && pwd) # the defaults work from any directory, not only the repo root tick=${1:-$root/scripts/coder-llm-ticketed} @@ -227,9 +232,43 @@ rc=$? grep -q "rc 78: ticket refused before the window" "$scratch/k.log" || { echo "FAIL K: a refused grant is not reported as such ($(grep -m1 'rc 78' "$scratch/k.log"))"; fail=1; } [ -s "$scratch/server.pid" ] && { echo "FAIL K: a server started on a refused grant"; fail=1; } +# ---- L: the window holds the token under its own label ---------------------------------------------------------- +echo idle > "$scratch/state"; rm -f "$scratch/server.pid"; : > "$scratch/klog"; touch "$scratch/demand" +env "${envs[@]}" CODER_LLM_LABEL=1bit-canonical CODER_LLM_IDLE_S=4 CODER_LLM_MAX_HOLD_MIN=1 timeout 60 "$tick" -- "$scratch/fake" > "$scratch/l.log" 2>&1 & +ll=$!; pids+=($ll) +wait_for 15 server_alive || { echo "FAIL L: no window opened"; fail=1; } +grep -qs '1bit-canonical' "$scratch/TOKEN" || { echo "FAIL L: the token is not held as 1bit-canonical ($(head -c 60 "$scratch/TOKEN" 2> /dev/null))"; fail=1; } +grep -qs 'coder-llm' "$scratch/TOKEN" && { echo "FAIL L: the token still says coder-llm"; fail=1; } +wait "$ll" + +# ---- M: ticketed-wait asks for the model and returns once it answers --------------------------------------------- +waiter=$root/scripts/ticketed-wait +echo idle > "$scratch/state"; rm -f "$scratch/server.pid" "$scratch/demand"; : > "$scratch/klog" +touch -d '-10 min' "$scratch/demand" +env "${envs[@]}" CODER_LLM_IDLE_S=6 CODER_LLM_MAX_HOLD_MIN=1 timeout 90 "$tick" -- "$scratch/fake" > "$scratch/m.log" 2>&1 & +mm=$!; pids+=($mm) +sleep 3 +if [ -e "$scratch/TOKEN" ] || server_alive; then echo "FAIL M: a window opened before anyone asked"; fail=1; fi +t0=$(date +%s) +env TICKETED_WAIT_DEMAND="$scratch/demand" TICKETED_WAIT_POLL_S=1 GPU_TOKEN_FILE="$scratch/TOKEN" \ + "$waiter" test-unit "http://127.0.0.1:$port" 40 2> "$scratch/m.wait"; rc=$? +[ "$rc" = 0 ] || { echo "FAIL M: ticketed-wait rc $rc, want 0 ($(tail -1 "$scratch/m.wait"))"; fail=1; } +[ $(( $(date +%s) - t0 )) -lt 40 ] || { echo "FAIL M: ticketed-wait took the whole limit"; fail=1; } +grep -q 'answers after' "$scratch/m.wait" || { echo "FAIL M: ticketed-wait returned without the model answering"; fail=1; } +grep -qs coder-llm "$scratch/TOKEN" || { echo "FAIL M: the model answered outside a ticket"; fail=1; } +wait "$mm" + +# ---- N: a model that never answers: 75 after the limit, demand touched ------------------------------------------- +dead=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])') +rm -f "$scratch/n.demand"; t0=$(date +%s) +env TICKETED_WAIT_DEMAND="$scratch/n.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$dead" 3 2> "$scratch/n.wait"; rc=$? +[ "$rc" = 75 ] || { echo "FAIL N: ticketed-wait rc $rc for a model that never answers, want 75"; fail=1; } +[ -e "$scratch/n.demand" ] || { echo "FAIL N: ticketed-wait did not touch the demand file"; fail=1; } +[ $(( $(date +%s) - t0 )) -le 10 ] || { echo "FAIL N: ticketed-wait overran its 3 s limit"; fail=1; } + # ---- G: the default queue wait keeps the ticket's place on a busy box (a give-up re-queues at the back) ------------ sed -n 's/^QUEUE_WAIT=\${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw" [ "$(cat "$scratch/qw")" -ge 14400 ] 2> /dev/null || { echo "FAIL G: default queue wait $(cat "$scratch/qw")s; under 4 h a busy queue makes it re-queue at the back and starve"; fail=1; } -[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window)" +[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits and gives up with 75)" exit "$fail" diff --git a/tests/exit_code_table_test.py b/tests/exit_code_table_test.py index c8e51b5f5..37da82348 100755 --- a/tests/exit_code_table_test.py +++ b/tests/exit_code_table_test.py @@ -33,7 +33,8 @@ help="engine checkout to check (default: this test's own tree)") root = Path(ap.parse_args().root) SCRIPTS = {"gpu-hold": "scripts/gpu-hold", "npu-hold": "scripts/npu-hold", "hw-gate.sh": "scripts/hw-gate.sh", - "window-run.sh": "scripts/window-run.sh", "coder-llm-ticketed": "scripts/coder-llm-ticketed"} + "window-run.sh": "scripts/window-run.sh", "coder-llm-ticketed": "scripts/coder-llm-ticketed", + "ticketed-wait": "scripts/ticketed-wait"} doc = (root / "docs/ci-runners.md").read_text() m = re.search(r"^## Exit codes on the GPU and NPU path.*?(?=^## )", doc, re.S | re.M) diff --git a/tests/install_coder_llm_ticketed_test.sh b/tests/install_coder_llm_ticketed_test.sh index 5ab8d33c7..bc2b4a639 100755 --- a/tests/install_coder_llm_ticketed_test.sh +++ b/tests/install_coder_llm_ticketed_test.sh @@ -20,7 +20,10 @@ # 2 --status says NOT ticketed (exit 1) before the install; # 3 --install backs up the drop-in, keeps the server command byte for byte behind the wrapper, installs the wrapper, # reloads and restarts (the stub records both), and --status then says ticketed (exit 0); -# 4 a second --install changes nothing (idempotent). +# 4 a second --install changes nothing (idempotent); +# 5 CODER_LLM_UNIT=1bit-canonical (engine#2579): the drop-in gets that unit's own label, demand file and URL (from the +# --port of its serve line), the pr-agent poll off, the command byte for byte; the restart is of that unit; a serve +# line with no --port is refused rather than guessed. set -u root=$(cd "$(dirname "$0")/.." && pwd) inst=${1:-$root/scripts/install-coder-llm-ticketed.sh} @@ -73,5 +76,26 @@ env "${envs[@]}" "$inst" --install > "$scratch/i2" 2>&1; rc=$? || { echo "FAIL 4: a second --install was not a no-op (rc $rc)"; fail=1; } [ "$(ls "$scratch/unit"/zz.conf.bak-* | wc -l)" = 1 ] || { echo "FAIL 4: a second backup was written"; fail=1; } -[ "$fail" = 0 ] && echo "install_coder_llm_ticketed: ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte, backs up, reloads+restarts; idempotent)" +# ---- 5: another resident model, 1bit-canonical ------------------------------------------------------------------- +mkdir -p "$scratch/cunit"; : > "$scratch/calls" +cserve='/opt/x/1bit serve -m /models/q.gguf --device hrx --host 127.0.0.1 --port 8095 --ctx-size 8192 --alias qwen3-4b' +printf '[Service]\nEnvironment=HSA_USERPTR_FOR_PAGED_MEM=0\nExecStart=\nExecStart=%s\n' "$cserve" > "$scratch/cunit/zz.conf" +sed "s|$scratch/unit/zz.conf|$scratch/cunit/zz.conf|" "$scratch/systemctl" > "$scratch/csystemctl"; chmod +x "$scratch/csystemctl" +cenvs=(CODER_LLM_UNIT=1bit-canonical CODER_LLM_UNIT_DIR="$scratch/cunit" CODER_LLM_DROPIN=zz.conf CODER_LLM_BIN_DIR="$scratch/bin" + CODER_LLM_SYSTEMCTL="$scratch/csystemctl" GPU_TOKEN_FILE="$scratch/none") +env "${cenvs[@]}" "$inst" --install > "$scratch/c1" 2>&1; rc=$? +[ "$rc" = 0 ] || { echo "FAIL 5: --install of 1bit-canonical rc $rc: $(tail -2 "$scratch/c1")"; fail=1; } +for want in "Environment=CODER_LLM_LABEL=1bit-canonical" "Environment=CODER_LLM_DEMAND=$HOME/.cache/1bit-canonical/demand" \ + "Environment=CODER_LLM_URL=http://127.0.0.1:8095" "Environment=CODER_LLM_GH_POLL=0" \ + "ExecStart=$scratch/bin/coder-llm-ticketed -- $cserve"; do + grep -qxF "$want" "$scratch/cunit/zz.conf" || { echo "FAIL 5: missing '$want'"; fail=1; } +done +grep -q 'CODER_LLM_GH_POLL=1' "$scratch/cunit/zz.conf" && { echo "FAIL 5: the pr-agent poll is on for 1bit-canonical"; fail=1; } +grep -qx 'restart 1bit-canonical' "$scratch/calls" || { echo "FAIL 5: 1bit-canonical was not restarted ($(tr '\n' ';' < "$scratch/calls"))"; fail=1; } +grep -qx 'restart coder-llm' "$scratch/calls" && { echo "FAIL 5: coder-llm was restarted for a 1bit-canonical install"; fail=1; } +printf '[Service]\nExecStart=\nExecStart=/opt/x/1bit serve -m /models/q.gguf\n' > "$scratch/cunit/zz.conf" +env "${cenvs[@]}" "$inst" --diff > "$scratch/c2" 2>&1; rc=$? +[ "$rc" = 2 ] && grep -q 'no --port' "$scratch/c2" || { echo "FAIL 5: a serve line with no --port was not refused (rc $rc)"; fail=1; } + +[ "$fail" = 0 ] && echo "install_coder_llm_ticketed: ok (--diff changes nothing; status before = NOT; install keeps the command byte for byte, backs up, reloads+restarts; idempotent; another unit gets its own label, demand and URL, and no --port is refused)" exit "$fail" From a90b52320280a2e3aef3b3f0f1741054c65ac878 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 18:19:52 -0400 Subject: [PATCH 2/2] ticketed-wait withdraws its demand when it gives up; other units take demand as stale after 5 min Review of #2588 (CHANGES, 22:09Z): a caller that timed out left its demand fresh for up to the 30-min TTL, so the unit would queue, take a GPU ticket and serve nobody for its 5-min idle time. - ticketed-wait removes the demand file on the 75 path ("demand withdrawn"). A waiter that is still alive re-touches it within 30 s, so the removal starves no one. - The installer gives units other than coder-llm CODER_LLM_DEMAND_TTL_MIN=5. Their callers re-touch every 30 s through ticketed-wait, so a crashed caller's touch expires in minutes. coder-llm keeps 30, because its workflow touches once. - Arm N asserts the demand is gone after a timeout. New arm O: a model that already answers gives 0 at once and no demand file is created. Install arm 5 asserts the TTL line. Mutants: no withdrawal -> FAIL N; no up-first check -> FAIL O; no TTL line -> FAIL 5. Co-Authored-By: Claude Opus 5.5 --- docs/ci-runners.md | 4 +++- scripts/install-coder-llm-ticketed.sh | 7 +++++-- scripts/ticketed-wait | 5 ++++- tests/coder_llm_ticketed_test.sh | 18 +++++++++++++++--- tests/install_coder_llm_ticketed_test.sh | 2 +- 5 files changed, 28 insertions(+), 8 deletions(-) diff --git a/docs/ci-runners.md b/docs/ci-runners.md index 35f0538d4..a1688590a 100644 --- a/docs/ci-runners.md +++ b/docs/ci-runners.md @@ -334,7 +334,9 @@ on HRX, `:8095`) costing 3-9 % of every other GPU job while it sat idle. Tickete The script touches the demand file every 30 s while it waits, and polls `/v1/models`. It returns 0 once the model answers, at once if the unit is resident or its window is already open, and 75 after the limit. A 75 means the window -is queued behind other tickets, so the run is **not run**, never a fake answer. A caller that works longer than the +is queued behind other tickets, so the run is **not run**, never a fake answer. On a 75 it also **removes the demand +file**, so a caller that has gone does not buy a GPU ticket for nobody. A unit other than coder-llm also takes demand +as stale after 5 minutes (`CODER_LLM_DEMAND_TTL_MIN=5`), so even a crashed caller's touch expires quickly. A caller that works longer than the unit's idle time (5 min) keeps touching the demand file, or the window closes under it. diff --git a/scripts/install-coder-llm-ticketed.sh b/scripts/install-coder-llm-ticketed.sh index a70835c0d..0aaabd333 100755 --- a/scripts/install-coder-llm-ticketed.sh +++ b/scripts/install-coder-llm-ticketed.sh @@ -29,7 +29,9 @@ # label, demand file and URL (engine#2579: 1bit-canonical on :8095 cost 3-9 % of every other GPU job while idle): # CODER_LLM_UNIT=1bit-canonical install-coder-llm-ticketed.sh --diff # For a unit other than coder-llm the drop-in gets CODER_LLM_LABEL=, CODER_LLM_DEMAND=~/.cache//demand, -# CODER_LLM_URL=http://127.0.0.1: (refused when there is none) and CODER_LLM_GH_POLL=0 +# CODER_LLM_URL=http://127.0.0.1: (refused when there is none), CODER_LLM_DEMAND_TTL_MIN=5 +# (its callers re-touch every 30 s through ticketed-wait, so a crashed caller's demand expires in minutes, not 30) and +# CODER_LLM_GH_POLL=0 # (the pr-agent fallback is coder-llm's only). Its callers ask for it with scripts/ticketed-wait. # # Environment (tests): CODER_LLM_UNIT_DIR (default ~/.config/systemd/user/.service.d), CODER_LLM_DROPIN @@ -56,7 +58,8 @@ render() { [ -n "$port" ] || { echo "no --port on $UNIT's serve line in $DROPIN: refusing to guess its URL" >&2; return 2; } extra="Environment=CODER_LLM_LABEL=$UNIT Environment=CODER_LLM_DEMAND=$HOME/.cache/$UNIT/demand -Environment=CODER_LLM_URL=http://127.0.0.1:$port" +Environment=CODER_LLM_URL=http://127.0.0.1:$port +Environment=CODER_LLM_DEMAND_TTL_MIN=5" fi awk -v w="$WRAPPER" -v b="$BIN_DIR" -v unit="$UNIT" -v extra="$extra" ' /^ExecStart=$/ && !done { diff --git a/scripts/ticketed-wait b/scripts/ticketed-wait index 76852591f..03eb3c29a 100755 --- a/scripts/ticketed-wait +++ b/scripts/ticketed-wait @@ -27,6 +27,8 @@ # Exit 0: the model answers (at once, when the unit is resident or its window is already open). # Exit 75: it did not answer within SECONDS (default 900). The window is queued behind other GPU tickets and comes # round in FIFO order, so a long wait is the queue, not a fault: report "not run", never a fake answer. +# The demand file is removed on the way out: a demand that outlives its caller would buy a GPU ticket for +# nobody (review of #2588). A waiter that is still alive re-touches it within 30 s, so this starves no one. # Exit 2: usage. set -u unit=${1:-}; url=${2:-}; limit=${3:-900} @@ -41,7 +43,8 @@ echo "ticketed-wait: asked for $unit (touched $demand); waiting up to ${limit}s while :; do now=$(date +%s) if [ $(( now - start )) -ge "$limit" ]; then - echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run" >&2 + echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn" >&2 + rm -f "$demand" exit 75 fi [ $(( now - touched )) -ge 30 ] && { touch "$demand"; touched=$now; } diff --git a/tests/coder_llm_ticketed_test.sh b/tests/coder_llm_ticketed_test.sh index 897dfd0cf..e62135396 100755 --- a/tests/coder_llm_ticketed_test.sh +++ b/tests/coder_llm_ticketed_test.sh @@ -43,7 +43,9 @@ # engine#2579); # M scripts/ticketed-wait asks a waiting unit for the model and returns 0 once it answers: it touches the demand # file, the wrapper queues and opens a window, and /v1/models answers; -# N ticketed-wait on a model that never answers gives up with 75 after its limit, having touched the demand file. +# N ticketed-wait on a model that never answers gives up with 75 after its limit and WITHDRAWS the demand it made, +# so the unit does not take a ticket for a caller that has gone (review of #2588); +# O ticketed-wait on a model that already answers returns 0 at once and creates no demand at all. set -u root=$(cd "$(dirname "$0")/.." && pwd) # the defaults work from any directory, not only the repo root tick=${1:-$root/scripts/coder-llm-ticketed} @@ -263,12 +265,22 @@ dead=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); pr rm -f "$scratch/n.demand"; t0=$(date +%s) env TICKETED_WAIT_DEMAND="$scratch/n.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$dead" 3 2> "$scratch/n.wait"; rc=$? [ "$rc" = 75 ] || { echo "FAIL N: ticketed-wait rc $rc for a model that never answers, want 75"; fail=1; } -[ -e "$scratch/n.demand" ] || { echo "FAIL N: ticketed-wait did not touch the demand file"; fail=1; } +grep -q 'demand withdrawn' "$scratch/n.wait" || { echo "FAIL N: ticketed-wait did not say it withdrew the demand"; fail=1; } +[ -e "$scratch/n.demand" ] && { echo "FAIL N: the demand outlived its caller: the unit would take a ticket for nobody"; fail=1; } + +# ---- O: a model that already answers: 0 at once, no demand created -------------------------------------------------- +echo idle > "$scratch/state"; rm -f "$scratch/o.demand" +"$scratch/fake" > /dev/null 2>&1 & of=$!; pids+=($of) +wait_for 10 curl -sf -o /dev/null "http://127.0.0.1:$port/v1/models" || { echo "FAIL O: the fake did not come up"; fail=1; } +env TICKETED_WAIT_DEMAND="$scratch/o.demand" TICKETED_WAIT_POLL_S=1 "$waiter" test-unit "http://127.0.0.1:$port" 5 2> "$scratch/o.wait"; rc=$? +[ "$rc" = 0 ] || { echo "FAIL O: ticketed-wait rc $rc on a model that already answers, want 0"; fail=1; } +[ -e "$scratch/o.demand" ] && { echo "FAIL O: ticketed-wait created a demand for a model that already answers"; fail=1; } +kill "$of" 2> /dev/null; wait "$of" 2> /dev/null [ $(( $(date +%s) - t0 )) -le 10 ] || { echo "FAIL N: ticketed-wait overran its 3 s limit"; fail=1; } # ---- G: the default queue wait keeps the ticket's place on a busy box (a give-up re-queues at the back) ------------ sed -n 's/^QUEUE_WAIT=\${CODER_LLM_QUEUE_WAIT_S:-\([0-9]*\)}$/\1/p' "$tick" > "$scratch/qw" [ "$(cat "$scratch/qw")" -ge 14400 ] 2> /dev/null || { echo "FAIL G: default queue wait $(cat "$scratch/qw")s; under 4 h a busy queue makes it re-queue at the back and starve"; fail=1; } -[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits and gives up with 75)" +[ "$fail" = 0 ] && echo "coder_llm_ticketed: ok (no demand -> no ticket; idle -> rc 0; busy -> 124 at max hold; fault -> 78 + back-off; FIFO: coder-llm, lane, coder-llm; a failing gh poll is logged once, no window; default queue wait keeps its place; a stale grant releases without a load; a slow-exiting server gets its grace, a KILL after the max hold is 124, one after a queue wait stays 137; a refused grant is told from a fault during a window; a window takes its label; ticketed-wait asks, waits, gives up with 75 and withdraws its demand, and asks for nothing when the model is up)" exit "$fail" diff --git a/tests/install_coder_llm_ticketed_test.sh b/tests/install_coder_llm_ticketed_test.sh index bc2b4a639..f9926b5b2 100755 --- a/tests/install_coder_llm_ticketed_test.sh +++ b/tests/install_coder_llm_ticketed_test.sh @@ -86,7 +86,7 @@ cenvs=(CODER_LLM_UNIT=1bit-canonical CODER_LLM_UNIT_DIR="$scratch/cunit" CODER_L env "${cenvs[@]}" "$inst" --install > "$scratch/c1" 2>&1; rc=$? [ "$rc" = 0 ] || { echo "FAIL 5: --install of 1bit-canonical rc $rc: $(tail -2 "$scratch/c1")"; fail=1; } for want in "Environment=CODER_LLM_LABEL=1bit-canonical" "Environment=CODER_LLM_DEMAND=$HOME/.cache/1bit-canonical/demand" \ - "Environment=CODER_LLM_URL=http://127.0.0.1:8095" "Environment=CODER_LLM_GH_POLL=0" \ + "Environment=CODER_LLM_URL=http://127.0.0.1:8095" "Environment=CODER_LLM_GH_POLL=0" "Environment=CODER_LLM_DEMAND_TTL_MIN=5" \ "ExecStart=$scratch/bin/coder-llm-ticketed -- $cserve"; do grep -qxF "$want" "$scratch/cunit/zz.conf" || { echo "FAIL 5: missing '$want'"; fail=1; } done