Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions docs/ci-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,26 @@ A silent `:8090` is therefore **not** evidence of a failed restart on its own
purpose, and a window that reports them as failures is how lanes learn to ignore failures (#2496, #2547, #2566).
`--restart-rc` encodes the same table, and its self-test asserts each row.

`scripts/install-coder-llm-ticketed.sh` does the switch:
### Other resident models: 1bit-canonical on :8095 (engine#2579)

The same wrapper runs any model that used to stay resident. engine#2579 measured `1bit-canonical.service` (Qwen3-4B
on HRX, `:8095`) costing 3-9 % of every other GPU job while it sat idle. Ticketed, it gets its own gpu-hold label
(`1bit-canonical`), its own demand file (`~/.cache/1bit-canonical/demand`) and its own URL, read from its serve line's
`--port`. The pr-agent poll is coder-llm's alone.

CODER_LLM_UNIT=1bit-canonical scripts/install-coder-llm-ticketed.sh --diff # then --install, --status

**A caller asks before it requests**, with `scripts/ticketed-wait UNIT URL [SECONDS]`:

ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 || { echo "not run: :8095 queued"; exit 75; }

The script touches the demand file every 30 s while it waits, and polls `/v1/models`. It returns 0 once the model
answers, at once if the unit is resident or its window is already open, and 75 after the limit. A 75 means the window
is queued behind other tickets, so the run is **not run**, never a fake answer. On a 75 it also **removes the demand
file**, so a caller that has gone does not buy a GPU ticket for nobody. A unit other than coder-llm also takes demand
as stale after 5 minutes (`CODER_LLM_DEMAND_TTL_MIN=5`), so even a crashed caller's touch expires quickly. A caller that works longer than the
unit's idle time (5 min) keeps touching the demand file, or the window closes under it.


- with no argument, or `--diff`, it prints the drop-in change;
- `--status` says whether the wrapper is installed and the running unit executes it (exit 0 only when both are true);
Expand Down Expand Up @@ -355,7 +374,7 @@ Each script's own header stays the reference for its edge cases.
| 2 | one code, four causes; **the stderr line says which** and is part of the contract (each phrase below is what the script prints, so grep your log for it): (a) a **usage error**, `gpu-hold: unknown option` or `gpu-hold: no command given`: the command line is wrong, not the box; (b) **could not run / INCOMPLETE**, `INCOMPLETE (an input was absent: nothing was tested)`: an input was absent (a model, text, a binary, a checkout) or a build or memory floor refused it. hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question; (c) the box could not set up the ticket, `gpu-hold: cannot create` or `gpu-hold: cannot write`; (d) a **policy refusal**, `npu-hold: REFUSED:` (a shared queue while npu-serve-keeper reads it) | gpu-hold and hw-gate.sh exit it; npu-hold exits (d) | **read stderr first**. (a) and (d): fix the command or config, because re-queueing repeats the same mistake; (b): after providing the input; (c): after fixing the box |
| 3 | the server under test never came up | hw-gate.sh exits it from its server-start step | read the server log |
| 69 | the job ran, but coder-llm did not answer `:8090` after the window | window-run.sh (computed by `final_rc`, not a literal exit) | see "The PR-Agent model runs in gpu-hold windows" |
| 75 | the queue never came round within `-w` | gpu-hold exits it; npu-hold and coder-llm-ticketed pass it through | yes, but **at the back** of the queue. That is why coder-llm queues with a 24 h `-w`, and why a lane should size `-w` for the queue instead of treating 75 as a free retry |
| 75 | the queue never came round within `-w`, or a ticketed model did not answer within `ticketed-wait`'s limit | gpu-hold exits it; ticketed-wait exits it; npu-hold and coder-llm-ticketed pass it through | yes, but **at the back** of the queue. That is why coder-llm queues with a 24 h `-w`, and why a lane should size `-w` for the queue instead of treating 75 as a free retry |
| 76 | the memory pre-check refused the run | hw-gate.sh `--mem` exits it | when MemAvailable is back |
| **78** | a GPU/NPU fault line in the kernel log: **during** the hold (the hold was ended) or **at take time** (the ticket was refused, nothing ran). The coder-llm journal says which one | gpu-hold exits it; npu-hold and coder-llm-ticketed pass it through (the coder-llm journal names the cause) | after the fault is understood. Never in a loop |
| **124** | the max hold (`-t`) ended the job: a deadline, and planned. coder-llm-ticketed also maps a KILL after its max hold to 124 | gpu-hold sets it (its own code, after `timeout(1)`'s convention; it also maps a TERM at its deadline from 143); npu-hold passes it through; coder-llm-ticketed passes it through and relabels a post-deadline 137 to it | yes |
Expand Down
8 changes: 6 additions & 2 deletions scripts/coder-llm-ticketed
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@
# so the window can end its own server before gpu-hold KILLs the window; an explicit
# GPU_HOLD_KILL_GRACE in the environment wins
# CODER_LLM_GPU_HOLD gpu-hold to use, default the one on PATH
# CODER_LLM_LABEL the gpu-hold label of the windows, default coder-llm. The same wrapper runs other resident
# models the same way (1bit-canonical on :8095, engine#2579): one label, demand file and
# URL per unit, set by scripts/install-coder-llm-ticketed.sh (CODER_LLM_UNIT)
# CODER_LLM_ALWAYS=1 skip the demand wait (a resident window, still ticketed and still max-hold bounded)
#
# Internal: "coder-llm-ticketed --window SERVER_CMD..." is what gpu-hold runs; it is the idle-aware window itself.
Expand Down Expand Up @@ -94,6 +97,7 @@ BACKOFF=${CODER_LLM_FAULT_BACKOFF_S:-600}
STOP_WAIT=${CODER_LLM_STOP_WAIT_S:-30}
KILL_GRACE=${GPU_HOLD_KILL_GRACE:-${CODER_LLM_KILL_GRACE_S:-$(( STOP_WAIT + 10 ))}}
GPU_HOLD=${CODER_LLM_GPU_HOLD:-gpu-hold}
LABEL=${CODER_LLM_LABEL:-coder-llm}

log() { echo "coder-llm-ticketed[$$] $*"; }

Expand Down Expand Up @@ -204,12 +208,12 @@ holder=$(head -c 120 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1
# inside its stop wait): say so, do not override the operator's choice
[ "$KILL_GRACE" -ge $(( STOP_WAIT + 5 )) ] 2> /dev/null \
|| log "WARNING: the kill grace ${KILL_GRACE}s is not at least 5 s longer than CODER_LLM_STOP_WAIT_S=${STOP_WAIT}s: gpu-hold will KILL the window before the server can exit"
log "demand: queueing for the GPU (gpu-hold -l coder-llm -t $MAX_HOLD -w $QUEUE_WAIT)${holder:+; queued behind $holder}"
log "demand: queueing for the GPU (gpu-hold -l $LABEL -t $MAX_HOLD -w $QUEUE_WAIT)${holder:+; queued behind $holder}"
# gpu-hold's default 10 s grace KILLed the window at the first live max hold while the server was still exiting:
# rc 137 and an unclean stop instead of 124 (11-15 10:56Z)
export CODER_LLM_WINDOW_STAMP="${TMPDIR:-/tmp}/coder-llm-window.$$"
rm -f "$CODER_LLM_WINDOW_STAMP"
GPU_HOLD_KILL_GRACE=$KILL_GRACE "$GPU_HOLD" -t "$MAX_HOLD" -l coder-llm -w "$QUEUE_WAIT" -- "$0" --window "$@"
GPU_HOLD_KILL_GRACE=$KILL_GRACE "$GPU_HOLD" -t "$MAX_HOLD" -l "$LABEL" -w "$QUEUE_WAIT" -- "$0" --window "$@"
rc=$?
# a KILL once the window has been open for the max hold is still the max hold (as 143 is, which gpu-hold maps to
# 124); timed from the window's own start, so a long queue wait followed by some other KILL is not relabelled
Expand Down
55 changes: 44 additions & 11 deletions scripts/install-coder-llm-ticketed.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,26 +25,56 @@
# --install back up the drop-in as <name>.bak-<YYYYMMDD>, write the new one, install the wrapper into ~/.local/bin,
# daemon-reload, restart the unit, print --status. Already ticketed: nothing to do, exit 0.
#
# Environment (tests): CODER_LLM_UNIT_DIR (default ~/.config/systemd/user/coder-llm.service.d), CODER_LLM_DROPIN
# CODER_LLM_UNIT picks the unit, default coder-llm. Any other resident model is ticketed the same way, with its own
# label, demand file and URL (engine#2579: 1bit-canonical on :8095 cost 3-9 % of every other GPU job while idle):
# CODER_LLM_UNIT=1bit-canonical install-coder-llm-ticketed.sh --diff
# For a unit other than coder-llm the drop-in gets CODER_LLM_LABEL=<unit>, CODER_LLM_DEMAND=~/.cache/<unit>/demand,
# CODER_LLM_URL=http://127.0.0.1:<the --port of its serve line> (refused when there is none), CODER_LLM_DEMAND_TTL_MIN=5
# (its callers re-touch every 30 s through ticketed-wait, so a crashed caller's demand expires in minutes, not 30) and
# CODER_LLM_GH_POLL=0
# (the pr-agent fallback is coder-llm's only). Its callers ask for it with scripts/ticketed-wait.
#
# Environment (tests): CODER_LLM_UNIT_DIR (default ~/.config/systemd/user/<unit>.service.d), CODER_LLM_DROPIN
# (default zz-c22-selfcontained.conf), CODER_LLM_BIN_DIR (default ~/.local/bin), CODER_LLM_SYSTEMCTL (default
# "systemctl --user").
set -u
here=$(cd "$(dirname "$0")" && pwd)
UNIT_DIR=${CODER_LLM_UNIT_DIR:-$HOME/.config/systemd/user/coder-llm.service.d}
UNIT=${CODER_LLM_UNIT:-coder-llm}
UNIT_DIR=${CODER_LLM_UNIT_DIR:-$HOME/.config/systemd/user/$UNIT.service.d}
DROPIN=$UNIT_DIR/${CODER_LLM_DROPIN:-zz-c22-selfcontained.conf}
BIN_DIR=${CODER_LLM_BIN_DIR:-$HOME/.local/bin}
read -r -a SYSTEMCTL <<< "${CODER_LLM_SYSTEMCTL:-systemctl --user}"
WRAPPER=$BIN_DIR/coder-llm-ticketed
mode=${1:---diff}

# the new drop-in, from the live one: env lines before the final "ExecStart=" reset, wrapper in front of the command
# the serve line's --port, for a unit's own URL (coder-llm keeps the wrapper's :8090 default)
unit_port() { awk '/^ExecStart=./ { l = $0 } END { if (match(l, /--port[ =][0-9]+/)) { p = substr(l, RSTART, RLENGTH); sub(/--port[ =]/, "", p); print p } }' "$DROPIN"; }

render() {
awk -v w="$WRAPPER" -v b="$BIN_DIR" '
local extra="" port
if [ "$UNIT" != coder-llm ]; then
port=$(unit_port)
[ -n "$port" ] || { echo "no --port on $UNIT's serve line in $DROPIN: refusing to guess its URL" >&2; return 2; }
extra="Environment=CODER_LLM_LABEL=$UNIT
Environment=CODER_LLM_DEMAND=$HOME/.cache/$UNIT/demand
Environment=CODER_LLM_URL=http://127.0.0.1:$port
Environment=CODER_LLM_DEMAND_TTL_MIN=5"
fi
awk -v w="$WRAPPER" -v b="$BIN_DIR" -v unit="$UNIT" -v extra="$extra" '
/^ExecStart=$/ && !done {
print "# 2026-11-15 CODER-LLM-TICKETED (engine#2494): the model runs ONLY inside gpu-hold tickets, in on-demand windows"
print "# (label coder-llm, max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged."
print "# `systemctl --user start coder-llm` now means QUEUED, not \":8090 answers\" (docs/ci-runners.md)."
print "Environment=CODER_LLM_GH_POLL=1"
if (unit == "coder-llm") {
print "# 2026-11-15 CODER-LLM-TICKETED (engine#2494): the model runs ONLY inside gpu-hold tickets, in on-demand windows"
print "# (label coder-llm, max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged."
print "# `systemctl --user start coder-llm` now means QUEUED, not \":8090 answers\" (docs/ci-runners.md)."
print "Environment=CODER_LLM_GH_POLL=1"
} else {
print "# engine#2579: " unit " runs ONLY inside gpu-hold tickets, in on-demand windows, like coder-llm (label " unit ","
print "# max hold 25 min, re-queued FIFO by Restart=always); the server command below is unchanged. Callers ask for it"
print "# with scripts/ticketed-wait; `systemctl --user start " unit "` now means QUEUED (docs/ci-runners.md)."
print extra
print "Environment=CODER_LLM_GH_POLL=0"
}
print "Environment=CODER_LLM_GPU_HOLD=" b "/gpu-hold"
print "Environment=PATH=" b ":/usr/local/bin:/usr/bin:/bin"
print "Environment=CODER_LLM_MAX_HOLD_MIN=25"
Expand All @@ -71,7 +101,7 @@ status() {
echo "wrapper: NOT installed at $WRAPPER"; ok=1
fi
if is_ticketed_file; then echo "drop-in: $DROPIN runs the wrapper"; else echo "drop-in: $DROPIN does NOT run the wrapper"; ok=1; fi
exec_line=$("${SYSTEMCTL[@]}" show coder-llm -p ExecStart --value 2> /dev/null | head -1)
exec_line=$("${SYSTEMCTL[@]}" show "$UNIT" -p ExecStart --value 2> /dev/null | head -1)
case "$exec_line" in
*coder-llm-ticketed*) echo "running unit: executes coder-llm-ticketed" ;;
*) echo "running unit: does NOT execute coder-llm-ticketed (${exec_line:0:120})"; ok=1 ;;
Expand All @@ -85,7 +115,10 @@ status() {
case "$mode" in
--diff)
if is_ticketed_file; then echo "already ticketed: $DROPIN"; exit 0; fi
diff -u "$DROPIN" <(render) | sed "2s|/dev/fd/[0-9]*|$DROPIN (new)|"
tmp=$(mktemp) || exit 2
render > "$tmp" || { rm -f "$tmp"; exit 2; }
diff -u "$DROPIN" "$tmp" | sed "2s|$tmp|$DROPIN (new)|"
rm -f "$tmp"
exit 0 ;;
--status)
status; exit $? ;;
Expand All @@ -95,10 +128,10 @@ case "$mode" in
bak=$DROPIN.bak-$(date +%Y%m%d)
[ -e "$bak" ] && bak=$bak-$(date +%H%M%S)
cp -p "$DROPIN" "$bak" || exit 1
render > "$DROPIN.new" && mv "$DROPIN.new" "$DROPIN" || exit 1
render > "$DROPIN.new" && mv "$DROPIN.new" "$DROPIN" || { rm -f "$DROPIN.new"; exit 1; }
mkdir -p "$BIN_DIR" && install -m 0755 "$here/coder-llm-ticketed" "$WRAPPER" || exit 1
echo "backed up: $bak"
"${SYSTEMCTL[@]}" daemon-reload && "${SYSTEMCTL[@]}" restart coder-llm || { echo "systemctl failed; roll back with: cp -p $bak $DROPIN" >&2; exit 1; }
"${SYSTEMCTL[@]}" daemon-reload && "${SYSTEMCTL[@]}" restart "$UNIT" || { echo "systemctl failed; roll back with: cp -p $bak $DROPIN" >&2; exit 1; }
status; exit $? ;;
*)
echo "usage: $0 [--diff | --status | --install]" >&2; exit 2 ;;
Expand Down
53 changes: 53 additions & 0 deletions scripts/ticketed-wait
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ticketed-wait UNIT URL [SECONDS]
#
# Ask for a model that runs in gpu-hold windows (scripts/coder-llm-ticketed: coder-llm on :8090, 1bit-canonical on
# :8095; engine#2494, #2579) and wait until it answers, bounded. It touches ~/.cache/UNIT/demand (the file the
# unit's window waits for) every 30 s while it waits, and polls URL/v1/models. /v1/models, not /health: /health
# answers before a model is loaded. Run it right before the requests, and keep touching the demand file while you
# work if that takes longer than the unit's idle time (5 min), or the window closes under you.
#
# ticketed-wait 1bit-canonical http://127.0.0.1:8095 900 && curl ... http://127.0.0.1:8095/v1/chat/completions
#
# Exit 0: the model answers (at once, when the unit is resident or its window is already open).
# Exit 75: it did not answer within SECONDS (default 900). The window is queued behind other GPU tickets and comes
# round in FIFO order, so a long wait is the queue, not a fault: report "not run", never a fake answer.
# The demand file is removed on the way out: a demand that outlives its caller would buy a GPU ticket for
# nobody (review of #2588). A waiter that is still alive re-touches it within 30 s, so this starves no one.
# Exit 2: usage.
set -u
unit=${1:-}; url=${2:-}; limit=${3:-900}
[ -n "$unit" ] && [ -n "$url" ] && [ "$limit" -ge 0 ] 2> /dev/null || { echo "usage: ticketed-wait UNIT URL [SECONDS]" >&2; exit 2; }
demand=${TICKETED_WAIT_DEMAND:-$HOME/.cache/$unit/demand}
poll=${TICKETED_WAIT_POLL_S:-5}
up() { [ "$(curl -s -o /dev/null -m 5 -w '%{http_code}' "${url%/}/v1/models" 2> /dev/null)" = 200 ]; }
up && exit 0
mkdir -p "$(dirname "$demand")" && touch "$demand" || { echo "ticketed-wait: cannot touch $demand" >&2; exit 2; }
start=$(date +%s); touched=$start
echo "ticketed-wait: asked for $unit (touched $demand); waiting up to ${limit}s for $url; GPU token: $(head -c 80 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1)" >&2
while :; do
now=$(date +%s)
if [ $(( now - start )) -ge "$limit" ]; then
echo "ticketed-wait: $unit did not answer in ${limit}s (still queued for the GPU?): not run; demand withdrawn" >&2
rm -f "$demand"
exit 75
fi
[ $(( now - touched )) -ge 30 ] && { touch "$demand"; touched=$now; }
sleep "$poll"
up && { echo "ticketed-wait: $unit answers after $(( $(date +%s) - start ))s" >&2; touch "$demand"; exit 0; }
done
Loading
Loading