Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/ci-runners.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,36 @@ TimeoutStopSec=90
`tests/coder_llm_ticketed_test.sh` (ctest `coder_llm_ticketed`) drives the wrapper with the real `scripts/gpu-hold` on a
private queue and a fake `/slots` server. No device is used.

## Exit codes on the GPU and NPU path: one table

A lane reading an exit code wants one answer: **did my job fail, or did the box not let it run?** The fleet's scheduler
scripts answer it with a few shared codes. Each code below means the same thing in every script that uses it, and each
script's own header stays the reference for its edge cases.

| rc | meaning | scripts | retry? |
|---|---|---|---|
| 0 | the job (or gate) passed / the window closed on its own | all | - |
| 1 | the job's own failure: a check failed | hw-gate.sh, the payload | no: read the log |
| 2 | **could not run / INCOMPLETE**: an input was absent (a model, text, a binary, a checkout), a build or memory floor refused it, a refused shared queue, or a usage error. hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question | hw-gate.sh, npu-hold | after fixing the input |
| 3 | the server under test never came up | hw-gate.sh (internal) | read the server log |
| 69 | the job ran, but coder-llm did not answer `:8090` after the window | window-run.sh | see "The PR-Agent model runs in gpu-hold windows" |
| 75 | the queue never came round within `-w` | gpu-hold, npu-hold, coder-llm-ticketed | yes: re-queue (at the back) |
| 76 | the memory pre-check refused the run | hw-gate.sh `--mem` | when MemAvailable is back |
| **78** | a GPU/NPU fault line in the kernel log: **during** the hold (the hold was ended) or **at take time** (the ticket was refused, nothing ran). The coder-llm journal says which one | gpu-hold, npu-hold, coder-llm-ticketed | after the fault is understood. Never in a loop |
| **124** | the max hold (`-t`) ended the job: a deadline, and planned. coder-llm-ticketed also maps a KILL after its max hold to 124 | gpu-hold, npu-hold, coder-llm-ticketed | yes |
| 130 | a stop was requested and the job refused to die; the token was released anyway | gpu-hold, coder-llm-ticketed | check for a survivor (`~/GPU-QUEUE/.gpu-hold-survivors`) |
| **137** | SIGKILL **before** any deadline: the OOM killer or a `kill -9`. A failure, not a timeout | any payload | no: find the killer |
| 143 | SIGTERM (a cancel) that was not the max hold | any payload | depends on who sent it |

Three rules follow from the table:

- **75, 124 and 78 are not the job's verdict.** A gate or a measurement that ends with one of them has no result. Report
it as "not run", never as a pass or a fail.
- **78 is never retried in a loop.** coder-llm backs off 10 minutes, and a lane stops and logs the time plus the command
in its state file (the fault rule).
- **137 is only a timeout when something says so.** gpu-hold maps a TERM at its deadline to 124. A bare 137 is a kill
from outside the script.

## What a restart policy cannot do

It cannot notice that a runner is *not* coming back (a version mismatch, a revoked token, a full disk under
Expand Down
Loading