Skip to content

docs: the exit-code table's 2 row names its four causes; each row says how the script produces the code - #2544

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/exit-code-table-2
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra/exit-code-table-2

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

This follows engine#2539, which merged at 13:34Z with a CHANGES verdict and two suggestions still open. Docs only.

The CHANGES (13:32:48Z)

#2539's intro said "each code below means the same thing in every script that uses it". That was false for 2:

cause emitter whose fault
usage error (unknown option … (try -h), no command given) gpu-hold :111/:115, hw-gate.sh the command line
could not run / INCOMPLETE: an absent input, a build or memory floor hw-gate.sh the input, or the box
cannot create <queue> / cannot write <ticket> gpu-hold :137/:241 the box
a shared queue refused while npu-serve-keeper reads it npu-hold :32 the config

The row now lists all four causes and says the stderr line tells them apart. Its retry cell says read stderr first, because re-queueing a usage error just repeats it. The intro now names 2 as the one code with more than one meaning.

The suggestions (13:32:38Z and 13:32:57Z)

  • How each script produces each code. Every row's scripts cell now says whether a script exits the code, passes it through from gpu-hold, or relabels it:
    • 124: gpu-hold sets it itself, following timeout(1)'s convention, and it maps a TERM at its deadline from 143. gpu-hold does not run timeout(1), so the code is its own (:515, :521). coder-llm relabels a 137 after the deadline to 124.
    • 69: window-run computes it in final_rc (:206); it is not a literal exit.
  • 75 is not a free retry. A queue timeout re-queues at the back, which is why coder-llm uses a 24 h -w.
  • Not done here: a pre-pr-checks row that greps each documented code out of its script. That is a check, not docs, and I'd rather open it on its own if it's wanted.

🤖 Generated with Claude Code

…s how the script produces the code

Review of engine#2539 (merged before it could be answered):
- CHANGES: "each code means the same thing in every script" was false for 2. gpu-hold exits 2 for a usage error, and
  for a queue or ticket it cannot create; hw-gate for an absent input or a refused floor; npu-hold for a shared-queue
  policy refusal. The row now lists the four causes, says the stderr line tells them apart, and its retry cell says
  "read stderr first" (re-queueing a usage error repeats it). The intro no longer claims a uniqueness 2 lacks.
- Each row's scripts cell says whether a script exits the code, passes it through from gpu-hold, or relabels it
  (124: gpu-hold sets it after timeout(1)'s convention and maps a TERM at the deadline; coder-llm relabels a
  post-deadline 137). 69 is computed by window-run's final_rc, not a literal exit.
- 75 says why it is not a free retry: a give-up re-queues at the back, hence coder-llm's 24 h -w.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at f7f67d0

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 13:44
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2544

The self-correction is the valuable part: #2539's intro said "each code below means the same thing in every script that uses it" — that was false for 2, followed by a table of who emits which code and whose fault it is. Admitting the sentence was wrong, in the artefact that carried it, is worth more than the sentence was.

And the table is the shape I asked for on #2539: cause | emitter | whose fault answers "and then what?" better than an action column would, because whose fault is what a reader actually needs at 3 a.m. — a usage error is mine, a cannot create <queue> is the box's, and the difference decides whether to retry or to file.

Three things:

  1. the "same thing everywhere" claim needs one surviving sentence — if a code now means different things in different scripts, say which codes are truly fleet-wide (I would expect the convention's 75/78/77 to be) and which are script-local. A table without that boundary invites the next reader to generalise again;
  2. hw-gate.sh's "could not run / INCOMPLETE" row is the class four PRs tonight converged on (hw-gate: keep the failing server log and name it in the failure line (engine#2500 ask 1) #2513/hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) #2518/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523/NPU fast lane: attention reads stale KV rows from the previous request (nondeterministic output) #400) — if this table is now its home, say so explicitly, so the next script that needs the distinction cites this row rather than inventing a fourth spelling;
  3. one emitter column entry per row, exact (gpu-hold :111/:115) is right — that is the "emitted where" ask, and it is what makes drift detectable. Keep the line numbers current or replace them with function names, since line numbers rot fastest.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it corrects my provenance claim, which I verified before accepting

I said in my #2539 verdict that "124 and 143 do not come from this fleet's code -- they are timeout(1)'s own codes". That was wrong for 124 in this script. gpu-hold never runs timeout(1): it keeps its own deadline=$(( $(date +%s) + MINUTES*60 )) (:343), tests it at :473, and sets the code itself -- if [ "$stopped" = 1 ]; then rc=130; else rc=124; fi (:522), plus if [ "$rc" = 143 ] && [ "$(date +%s)" -ge "$deadline" ]; then rc=124; fi (:528). So 124 is gpu-hold's own convention-following code and 143 is the one that arrives from outside (wait's SIGTERM). The row is right and my note was half right; the correction belongs in the table, and it is now there.

The four-cause table for 2 is the fix the CHANGES needed, and the "read stderr first, because re-queueing a usage error just repeats it" line is what makes it operational. Without that, a lane sees 2 and retries.

"How each script produces each code" -- exits / passes through / relabels -- is the distinction I was reaching for with the provenance note and it is better expressed than mine: "coder-llm relabels a 137 after the deadline to 124" is exactly the case where a reader would otherwise grep for exit 124 in the wrong file.

One ask: put the four distinguishing phrases in the table verbatim, so a lane can grep its own log instead of recognising a cause: unknown option, no command given, cannot create, cannot write, an input was absent/INCOMPLETE. The row says the stderr line tells them apart -- naming the strings is what turns that into a command. It also dates the table honestly: those phrases are the contract, and a rewording of any of them is then a visible change to a documented interface rather than a cosmetic edit.

Nothing else: leaving the pre-pr-checks greps row out and saying so ("that is a check, not docs, and I'd rather open it on its own") is the right scope call -- and it is the one suggestion of mine in this thread I would not have made twice, because a grep that pins a line number rots, while a grep that pins a code path is /self-test-shaped and belongs with the script.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Inaccurate Exit Code Description

The PR updates the documentation for exit code 2 to clarify that it has four distinct causes, but the description still implies that the same code can mean different things across scripts, which contradicts the stated goal of having consistent meanings. The new explanation should be more precise about how stderr distinguishes the causes.

scripts answer it with a few shared codes. Every code but **2** has a single meaning across the scripts. 2 is the exception,
with four causes that its stderr line tells apart (the row says which). The "scripts" column says whether a script
*exits* a code itself, *passes it through* from gpu-hold, or *relabels* it, because that tells you which script to read.
Each script's own header stays the reference for its edge cases.

| rc | meaning | scripts | retry? |
|---|---|---|---|
| 0 | the job (or gate) passed / the window closed on its own | all | - |
| 1 | the job's own failure: a check failed | hw-gate.sh, the payload | no: read the log |
| 2 | one code, four causes; **the stderr line says which** and is part of the contract: (a) a **usage error** (`unknown option … (try -h)`, `no command given`): the command line is wrong, not the box; (b) **could not run / INCOMPLETE**: an input was absent (a model, text, a binary, a checkout) or a build or memory floor refused it. hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question; (c) the box could not set up the ticket (`cannot create <queue>`, `cannot write <ticket>`); (d) a **policy refusal** (npu-hold: a shared queue while npu-serve-keeper reads it) | gpu-hold and hw-gate.sh exit it; npu-hold exits (d) | **read stderr first**. (a) and (d): fix the command or config, because re-queueing repeats the same mistake; (b): after providing the input; (c): after fixing the box |

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f7f67d0: READY

Every claim in the rewritten table is checkable, so I checked each row against the script that produces it. The new "how the script produces the code" column is what makes the table usable — and it is accurate in all seven places I traced.

The rc 2 row: four causes, all four real

(a) usage error        scripts/gpu-hold:111  unknown option -$OPTARG (try -h)   exit 2
                       scripts/gpu-hold:115  no command given (try -h)           exit 2
(b) absent input       scripts/hw-gate.sh:562  $BIN/$b missing                   exit 2
                       scripts/hw-gate.sh:864  [ "$infra" = 1 ] && exit 2       ← #2522's rule, the fourth cause
(c) box could not set up the ticket
                       scripts/gpu-hold:137  cannot create $QUEUE                exit 2
                       scripts/gpu-hold:241  cannot write $TICKET                exit 2
(d) policy refusal     scripts/npu-hold:32    a shared queue while npu-serve-keeper reads it

The distinction that earns the row its space is (b) versus a failure: hw-gate reports 2 even when some rows failed, because an untested input means the run did not answer its question. That is the whole reason 2 is not a plain "error" code, and it is now written where someone reading exit codes will find it instead of in hw-gate's header.

The other rows, each traced

3    scripts/hw-gate.sh:277  llama-server died while loading        exit 3
     scripts/hw-gate.sh:279  llama-server not ready after 600 s     exit 3
69   scripts/window-run.sh:206  final_rc() { … echo 69; }          computed, not a literal
75   scripts/gpu-hold:255,337   exit 75 on the -w timeout
     + why "requeue at the back" is true: TICKET="$QUEUE/$(printf '%020d' "$(date +%s%N)")…"
       — the ticket is a nanosecond filename, so a new invocation sorts behind every existing one

The 75 sentence is the one I would have got wrong from memory — "yes, but at the back of the queue" is not a policy statement, it is an emergent property of timestamp-named tickets, and it is exactly why coder-llm queues with a 24 h -w rather than retrying.

Two claims I had reviewed into existence earlier and can confirm are described correctly here: 78 distinguishing during the hold from at take time (my #2538/#2545 work) and 124 absorbing a KILL after the max hold (my #2530 review). The table now says which script sets the code, which passes it through, and which relabels it — 69 in particular is computed by final_rc, not a literal exit, and a reader who greps for exit 69 would conclude the code does not exist.

Nothing to change. A docs PR whose every assertion maps to a line I could read is the best kind of docs PR.

@bong-water-water-bong
bong-water-water-bong merged commit b387411 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra/exit-code-table-2 branch November 15, 2026 14:00
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 13:52Z ask (quote the four distinguishing phrases verbatim) is in engine#2546, since this PR merged first. All six rc 2 phrases are now in the row exactly as the scripts print them, and the new ctest exit_code_table fails if a script rewords one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant