diff --git a/docs/ci-runners.md b/docs/ci-runners.md index 92d4c89a0..737c5f6d8 100644 --- a/docs/ci-runners.md +++ b/docs/ci-runners.md @@ -325,6 +325,36 @@ TimeoutStopSec=90 `tests/coder_llm_ticketed_test.sh` (ctest `coder_llm_ticketed`) drives the wrapper with the real `scripts/gpu-hold` on a private queue and a fake `/slots` server. No device is used. +## Exit codes on the GPU and NPU path: one table + +A lane reading an exit code wants one answer: **did my job fail, or did the box not let it run?** The fleet's scheduler +scripts answer it with a few shared codes. Each code below means the same thing in every script that uses it, and each +script's own header stays the reference for its edge cases. + +| rc | meaning | scripts | retry? | +|---|---|---|---| +| 0 | the job (or gate) passed / the window closed on its own | all | - | +| 1 | the job's own failure: a check failed | hw-gate.sh, the payload | no: read the log | +| 2 | **could not run / INCOMPLETE**: an input was absent (a model, text, a binary, a checkout), a build or memory floor refused it, a refused shared queue, or a usage error. hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question | hw-gate.sh, npu-hold | after fixing the input | +| 3 | the server under test never came up | hw-gate.sh (internal) | read the server log | +| 69 | the job ran, but coder-llm did not answer `:8090` after the window | window-run.sh | see "The PR-Agent model runs in gpu-hold windows" | +| 75 | the queue never came round within `-w` | gpu-hold, npu-hold, coder-llm-ticketed | yes: re-queue (at the back) | +| 76 | the memory pre-check refused the run | hw-gate.sh `--mem` | when MemAvailable is back | +| **78** | a GPU/NPU fault line in the kernel log: **during** the hold (the hold was ended) or **at take time** (the ticket was refused, nothing ran). The coder-llm journal says which one | gpu-hold, npu-hold, coder-llm-ticketed | after the fault is understood. Never in a loop | +| **124** | the max hold (`-t`) ended the job: a deadline, and planned. coder-llm-ticketed also maps a KILL after its max hold to 124 | gpu-hold, npu-hold, coder-llm-ticketed | yes | +| 130 | a stop was requested and the job refused to die; the token was released anyway | gpu-hold, coder-llm-ticketed | check for a survivor (`~/GPU-QUEUE/.gpu-hold-survivors`) | +| **137** | SIGKILL **before** any deadline: the OOM killer or a `kill -9`. A failure, not a timeout | any payload | no: find the killer | +| 143 | SIGTERM (a cancel) that was not the max hold | any payload | depends on who sent it | + +Three rules follow from the table: + +- **75, 124 and 78 are not the job's verdict.** A gate or a measurement that ends with one of them has no result. Report + it as "not run", never as a pass or a fail. +- **78 is never retried in a loop.** coder-llm backs off 10 minutes, and a lane stops and logs the time plus the command + in its state file (the fault rule). +- **137 is only a timeout when something says so.** gpu-hold maps a TERM at its deadline to 124. A bare 137 is a kill + from outside the script. + ## What a restart policy cannot do It cannot notice that a runner is *not* coming back (a version mismatch, a revoked token, a full disk under