Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions cmake/tests-gpu-hold.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,17 @@ add_test(NAME gpu_hold_mem_abort COMMAND bash ${CMAKE_SOURCE_DIR}/tests/gpu_hold
# when the token is taken; the 11-13 crash was nine resets inside two tickets. The kernel log is a fixture file.
add_test(NAME gpu_hold_fault_abort COMMAND bash ${CMAKE_SOURCE_DIR}/tests/gpu_hold_fault_abort_test.sh ${CMAKE_SOURCE_DIR}/scripts/gpu-hold)

# CODER-LLM-TICKETED 2026-11-15 (engine#2494): the PR-Agent model (coder-llm, :8090) runs only inside a gpu-hold ticket,
# in on-demand windows. Drives scripts/coder-llm-ticketed with the real scripts/gpu-hold on a private queue and a fake
# /slots server: no demand -> no ticket, idle -> window closes, busy -> max hold, kernel fault -> rc 78, FIFO with a lane.
add_test(NAME coder_llm_ticketed COMMAND bash ${CMAKE_SOURCE_DIR}/tests/coder_llm_ticketed_test.sh
${CMAKE_SOURCE_DIR}/scripts/coder-llm-ticketed ${CMAKE_SOURCE_DIR}/scripts/gpu-hold)
# its fake /slots server binds a loopback port, so it takes the route-port lock like every other route test (#1070)
set_tests_properties(coder_llm_ticketed PROPERTIES RESOURCE_LOCK route_ports)
# ...and scripts/install-coder-llm-ticketed.sh, on a fake unit dir and a stub systemctl: --diff changes nothing, --install
# backs up and keeps the server command byte for byte behind the wrapper, --status tells ticketed from not, idempotent.
add_test(NAME install_coder_llm_ticketed COMMAND bash ${CMAKE_SOURCE_DIR}/tests/install_coder_llm_ticketed_test.sh)

# engine#1635: a queued ticket that is removed (reaped, or deleted by another lane) used to leave its owner waiting for a
# queue position that no longer existed -- the full -w, 5400 s by default, looking exactly like "queued behind a busy box".
# Drives scripts/gpu-hold on a private queue: the ticket is deleted mid-wait, and separately rewritten to a dead pid so a
Expand Down
49 changes: 49 additions & 0 deletions docs/ci-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,55 @@ allocation failed as an exception rather than as an OOM kill, so that experiment
directive is the documented mechanism and it is what the fresh install's single end-to-end run should confirm,
together with the rest of #1504's acceptance.

## The PR-Agent model runs in gpu-hold windows (engine#2494)

The `pr-agent` job talks to a local model, Qwen3-Coder-30B-A3B served by `1bit serve` on HRX at `127.0.0.1:8090` (the
`coder-llm` user service). Until 2026-11-15 that server ran around the clock **outside any gpu-hold ticket**, so "one GPU
job at a time" never held, and its model is the one whose prefill faults the GPU (engine#2494). It now runs only inside a
ticket, in on-demand windows, through `scripts/coder-llm-ticketed` (deployed to `~/.local/bin`):

1. **It waits for demand.** Demand is the file `~/.cache/coder-llm/demand` touched within the last 30 minutes (the
`pr-agent` workflow's `reviewer` job touches it), or, as the fallback that is on by default (`CODER_LLM_GH_POLL=1`),
a `pr-agent` **pull_request** run that is queued or in progress (comment runs are not counted: most never review). So
a review is never starved by a workflow that forgot to ask. A poll that cannot ask (an expired token, a rate limit) is
logged once per wait as `gh poll failed (<reason>)` and treated as no demand.
2. **It queues for the GPU like any other job**: `gpu-hold -l coder-llm -t 25`. The FIFO queue, the temperature, memory
and kernel-fault watchdogs, and the release wait that lets the next label in all apply.
3. **Inside the ticket it serves**, and ends the window when no slot in `/slots` has been processing **and** the demand
file has not been touched for 5 minutes, or when the 25-minute max hold ends it.
4. **It exits**, and the unit's `Restart=always` starts the next invocation, which waits for demand and queues again at
the back of the line. After a kernel fault (gpu-hold rc 78) it backs off 10 minutes first.

Anything else that needs the model asks the same way: touch `~/.cache/coder-llm/demand`, keep touching it while it
works, and wait for `:8090` to answer a real completion. Until a window opens, the port refuses.

**The contract this changes: `systemctl --user start coder-llm` now means *queued*, not ":8090 answers".** A window
wrapper that restarts the unit from inside its own gpu-hold ticket and then polls `:8090` is polling a server that is queued
behind that very ticket. The server can answer only after the caller's ticket is released, so such a poll must not count a
silent `:8090` as "the fleet LLM did not come back". The unit's journal says `queued behind <holder>` when it queues. With
the model ticketed, a memory window no longer needs to stop it: it is not resident outside its own windows.

`scripts/install-coder-llm-ticketed.sh` does the switch:

- with no argument, or `--diff`, it prints the drop-in change;
- `--status` says whether the wrapper is installed and the running unit executes it (exit 0 only when both are true);
- `--install` backs the drop-in up as `<name>.bak-<YYYYMMDD>`, rewrites it, installs the wrapper, reloads and restarts.

The server command stays byte for byte; the wrapper goes in front of it:

```ini
[Service]
ExecStart=
ExecStart=%h/.local/bin/coder-llm-ticketed -- <the 1bit serve command line>
Environment=CODER_LLM_GH_POLL=1
Restart=always
RestartSec=15
TimeoutStopSec=90
```

`tests/coder_llm_ticketed_test.sh` (ctest `coder_llm_ticketed`) drives the wrapper with the real `scripts/gpu-hold` on a
private queue and a fake `/slots` server. No device is used.

## What a restart policy cannot do

It cannot notice that a runner is *not* coming back (a version mismatch, a revoked token, a full disk under
Expand Down
177 changes: 177 additions & 0 deletions scripts/coder-llm-ticketed
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# coder-llm-ticketed [--] SERVER_CMD [ARGS...]
#
# Runs the PR-Agent reviewer model (the coder-llm user service, :8090) ONLY inside a gpu-hold ticket, in on-demand
# windows (owner 2026-11-15, CODER-LLM-TICKETED; engine#2494). It used to run 24/7 outside any ticket, so "one GPU job
# at a time" never held, and the model it serves (Qwen3-Coder-30B on HRX) is the one whose prefill faults the GPU.
#
# One invocation is ONE window, and systemd's Restart= makes the next one:
# 1. wait for demand: the demand file was touched within CODER_LLM_DEMAND_TTL_MIN (the pr-agent workflow's reviewer
# job touches it, engine#2509), or a pr-agent workflow run is queued or in progress (CODER_LLM_GH_POLL, on by
# default as the fallback);
# 2. queue for the GPU like any other job: gpu-hold -l coder-llm -t CODER_LLM_MAX_HOLD_MIN (FIFO; gpu-hold's
# temperature, memory and kernel-fault watchdogs apply, and its release wait lets the next label in);
# 3. inside the ticket, start SERVER_CMD and end the window once the server has been idle (no slot in /slots is
# processing) AND the demand file has not been touched for CODER_LLM_IDLE_MIN - or when the max hold ends it;
# 4. exit, so systemd starts the next invocation, which goes back to waiting for demand and re-queues.
# A window cut short by a kernel fault (gpu-hold rc 78) or a refused ticket backs off CODER_LLM_FAULT_BACKOFF_S
# before exiting, so a faulting GPU does not turn into a restart loop.
#
# Anything else that needs the model asks for it the same way the workflow does: touch "$CODER_LLM_DEMAND" (and
# keep touching it while working), then wait for :8090 to answer.
#
# The contract this changes: `systemctl --user start coder-llm` now means QUEUED, not ":8090 answers". A window
# wrapper that restarts the unit from inside its own gpu-hold ticket and then polls :8090 is polling a server that is
# queued behind that very ticket; it answers only after the caller's ticket is released. The journal line
# "queued behind <holder>" says which state it is in.
#
# Environment (all optional):
# CODER_LLM_DEMAND demand file, default ~/.cache/coder-llm/demand
# CODER_LLM_DEMAND_TTL_MIN a touch this recent is demand, default 30 (CODER_LLM_DEMAND_TTL_S in seconds wins)
# CODER_LLM_IDLE_MIN idle minutes that end a window, default 5 (CODER_LLM_IDLE_S in seconds wins)
# CODER_LLM_MAX_HOLD_MIN max hold per window (gpu-hold -t), default 25
# CODER_LLM_QUEUE_WAIT_S how long one window queues for the GPU (gpu-hold -w), default 5400
# CODER_LLM_URL the server's base URL for /slots, default http://127.0.0.1:8090
# CODER_LLM_POLL_S poll interval, default 15
# CODER_LLM_GH_POLL 1 (the default) = also treat a queued/in-progress pr-agent run as demand (REST,
# every CODER_LLM_GH_POLL_S, default 60), so a review is never starved even if nothing
# touched the demand file; 0 = the demand file only
# CODER_LLM_GH_REPO default 1bit-MONSTER/engine
# CODER_LLM_FAULT_BACKOFF_S back-off after gpu-hold rc 78 (a fault, or a refused ticket), default 600
# CODER_LLM_GPU_HOLD gpu-hold to use, default the one on PATH
# CODER_LLM_ALWAYS=1 skip the demand wait (a resident window, still ticketed and still max-hold bounded)
#
# Internal: "coder-llm-ticketed --window SERVER_CMD..." is what gpu-hold runs; it is the idle-aware window itself.
set -u

DEMAND=${CODER_LLM_DEMAND:-$HOME/.cache/coder-llm/demand}
DEMAND_TTL=${CODER_LLM_DEMAND_TTL_S:-$(( ${CODER_LLM_DEMAND_TTL_MIN:-30} * 60 ))}
IDLE=${CODER_LLM_IDLE_S:-$(( ${CODER_LLM_IDLE_MIN:-5} * 60 ))}
MAX_HOLD=${CODER_LLM_MAX_HOLD_MIN:-25}
QUEUE_WAIT=${CODER_LLM_QUEUE_WAIT_S:-5400}
URL=${CODER_LLM_URL:-http://127.0.0.1:8090}
POLL=${CODER_LLM_POLL_S:-15}
GH_POLL=${CODER_LLM_GH_POLL:-1}
GH_POLL_S=${CODER_LLM_GH_POLL_S:-60}
GH_REPO=${CODER_LLM_GH_REPO:-1bit-MONSTER/engine}
BACKOFF=${CODER_LLM_FAULT_BACKOFF_S:-600}
GPU_HOLD=${CODER_LLM_GPU_HOLD:-gpu-hold}

log() { echo "coder-llm-ticketed[$$] $*"; }

mtime() { stat -c %Y "$1" 2> /dev/null || echo 0; }

demand_fresh() { # $1 = window in seconds
[ $(( $(date +%s) - $(mtime "$DEMAND") )) -lt "$1" ]
}

# Only pull_request runs count: every comment on a PR also creates a pr-agent run (about one every two minutes on a
# busy night) and most never review, so counting them would keep a 30B window open on chatter. Command comments
# (/review, /ask ...) reach the model through the demand file the workflow's reviewer job touches (engine#2509).
GH_FAIL_LOGGED=0
gh_demand() {
[ "$GH_POLL" = 1 ] || return 1
local s n err errf="${TMPDIR:-/tmp}/coder-llm-gh.$$"
for s in queued in_progress; do
n=$(gh api "repos/$GH_REPO/actions/workflows/pr-agent.yml/runs?status=$s&event=pull_request&per_page=1" \
-q .total_count 2> "$errf")
if [ $? -ne 0 ] || ! [ "$n" -ge 0 ] 2> /dev/null; then
# an absent answer is not "no demand": say why, once per wait (the behaviour stays "no demand", the safe side)
err=$(head -c 200 "$errf" 2> /dev/null | tr '\n' ' ')
[ "$GH_FAIL_LOGGED" = 1 ] || log "gh poll failed (${err:-answer '$n'}): treating it as no demand; check with: gh api repos/$GH_REPO/actions/workflows/pr-agent.yml/runs"
GH_FAIL_LOGGED=1
rm -f "$errf"
return 1
fi
rm -f "$errf"
[ "$n" -gt 0 ] && return 0
done
return 1
}

slots_busy() { # 0 = some slot is processing, or the server cannot say (not up yet counts as busy)
local j
j=$(curl -sf --max-time 5 "$URL/slots" 2> /dev/null) || return 0
grep -Eq '"is_processing"[[:space:]]*:[[:space:]]*true' <<< "$j" && return 0
return 1
}

# ---- the window: runs INSIDE the gpu-hold ticket --------------------------------------------------------------------
if [ "${1:-}" = "--window" ]; then
shift
[ "${1:-}" = "--" ] && shift
[ $# -gt 0 ] || { log "--window needs a server command"; exit 2; }
"$@" &
server=$!
stop_server() {
kill -TERM "$server" 2> /dev/null
for _ in $(seq 1 30); do kill -0 "$server" 2> /dev/null || return; sleep 1; done
kill -KILL "$server" 2> /dev/null
}
trap 'stop_server; exit 143' TERM INT HUP
log "window open: server pid $server"
last_busy=$(date +%s)
while kill -0 "$server" 2> /dev/null; do
sleep "$POLL"
now=$(date +%s)
if slots_busy || demand_fresh "$IDLE"; then
last_busy=$now
elif [ $(( now - last_busy )) -ge "$IDLE" ]; then
log "idle ${IDLE}s and no fresh demand: closing the window"
stop_server
exit 0
fi
done
wait "$server"
rc=$?
log "server exited by itself, rc $rc"
exit "$rc"
fi

# ---- one service invocation: wait for demand, then one ticketed window ----------------------------------------------
[ "${1:-}" = "--" ] && shift
[ $# -gt 0 ] || { echo "usage: coder-llm-ticketed [--] SERVER_CMD [ARGS...]" >&2; exit 2; }
mkdir -p "$(dirname "$DEMAND")"

if [ "${CODER_LLM_ALWAYS:-0}" != 1 ]; then
log "waiting for demand ($DEMAND touched within ${DEMAND_TTL}s$([ "$GH_POLL" = 1 ] && echo ", or a pr-agent pull_request run on $GH_REPO"))"
if [ "$GH_POLL" = 1 ] && ! gh auth status > /dev/null 2>&1; then
log "WARNING: gh is not logged in, so the pr-agent-run fallback cannot see demand; only the demand file will wake the model"
fi
next_gh=0
while :; do
demand_fresh "$DEMAND_TTL" && break
if [ "$GH_POLL" = 1 ] && [ "$(date +%s)" -ge "$next_gh" ]; then
gh_demand && { touch "$DEMAND"; break; }
next_gh=$(( $(date +%s) + GH_POLL_S ))
fi
sleep "$POLL"
done
fi
holder=$(head -c 120 "${GPU_TOKEN_FILE:-$HOME/GPU-TOKEN}" 2> /dev/null | head -1)
log "demand: queueing for the GPU (gpu-hold -l coder-llm -t $MAX_HOLD -w $QUEUE_WAIT)${holder:+; queued behind $holder}"
"$GPU_HOLD" -t "$MAX_HOLD" -l coder-llm -w "$QUEUE_WAIT" -- "$0" --window "$@"
rc=$?
case "$rc" in
0) log "window closed (idle)" ;;
124) log "window closed (max hold ${MAX_HOLD} min); re-queueing" ;;
75) log "the queue never came round in ${QUEUE_WAIT}s" ;;
78) log "gpu-hold rc 78 (a GPU fault in the kernel log, or a refused ticket): backing off ${BACKOFF}s"; sleep "$BACKOFF" ;;
*) log "window ended rc $rc" ;;
esac
exit "$rc"
Loading
Loading