Skip to content

hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 rule, third site) - #2557

Merged
bong-water-water-bong merged 1 commit into
mainfrom
token-row-is-not-a-failure
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
token-row-is-not-a-failure

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Follows #2501/#2522. Status: my commit — authorship note, not a verdict.

The third site, and the most frequently hit one

#2522 made an absent input a SKIP row and left a KLD verify mismatch as FAIL, on the rule that a row says what happened to the run, not who to blame for the box. The GPU token is the same class and it fires far more often: any lane holding ~/GPU-TOKEN makes the next gate run report

gpu-token  FAIL  300s  /home/bcloud/GPU-TOKEN held by another job for 300s: gpu-hold coder-llm pid 1027021 …

which reds every PR for a queue condition no PR can fix — exactly the harm #2501 described, one row over.

The exit code was already right (2, "could not run / INCOMPLETE"), so this changes only the row:

result gpu-token FAIL "$TOKEN_WAIT" …  ->  result gpu-token SKIP "$TOKEN_WAIT" …      (3 sites)

Measured, not read

I found this by running the gate, not by reading it. HW_GATE_GPU_HOLD=0 --phases placement blocked on the token held by gpu-hold coder-llm pid 1027021 and wrote no row at all before my 240 s timeout — because take_token() is called after the wrapper that HW_GATE_GPU_HOLD=0 skips.

I did not bypass the token, and no GPU job ran: the run produced no row and was killed by my own timeout. That flag's name and docstring ("HW_GATE_GPU_HOLD=0 skips the queue") suggest a tokenless run and it is not one — worth knowing separately, and I am not changing it here.

Tests: two assertions, both mutation-checked

script_invariants() gains, beside the absent-input ones:

  • no result gpu-token FAIL anywhere
  • result gpu-token SKIP still exists — a rename fails rather than passing over nothing
one site back to FAIL   -> FAIL: a held GPU token is an infra condition: the row must be SKIP, not FAIL
row renamed away        -> FAIL: the gpu-token row must exist and be SKIP (it was renamed or removed)
both correct            -> hw_gate_test: PASS

The second arm is the one that matters: it is the same "fails rather than passes over nothing" guard as if (mapped == 0) fails++ on #396 and the distinct count on #78.

What this does not fix

  • HW_GATE_GPU_HOLD=0 does not make a run tokenless (see above) — its own change, and the docstring should either say so or the flag should do it.
  • A gate run blocked on the token still takes TOKEN_WAIT seconds before it says anything. With -w, a lane learns nothing until the wait expires. That is a usability problem the row fix does not touch.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 793bdef)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2501 - Partially compliant

Compliant requirements:

  • The PR correctly changes the GPU token handling to report SKIP instead of FAIL when the token is held by another job
  • The PR ensures that PRs are not marked as failed due to infrastructure issues like held GPU tokens
  • The changes are measured and verified through tests

Non-compliant requirements:

  • The PR does not address the missing model file issue directly, as it is an infrastructure issue requiring box changes

Requires further human verification:

  • Verification that the GPU token handling change works correctly on the actual hardware
  • Confirmation that the model file issue is addressed by the box owner
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. The change is applied in three locations within the script.

result gpu-token SKIP "$TOKEN_WAIT" "$TOKEN held by another job for ${TOKEN_WAIT}s: $(cat "$TOKEN" 2> /dev/null)"
GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job during retries. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. The change is applied in the retry logic of the gpu() function.

result gpu-token SKIP "$TOKEN_WAIT" "$TOKEN held by another job for ${TOKEN_WAIT}s: $(cat "$TOKEN" 2> /dev/null)"
GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job before the first GPU step. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. The change is applied in the pre-GPU step token check.

result gpu-token SKIP "$TOKEN_WAIT" "$TOKEN held by another job for ${TOKEN_WAIT}s: $(cat "$TOKEN" 2> /dev/null)"
✅ Resolved findings

scripts/hw-gate.sh:569-570

GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job before the first GPU step. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box.

scripts/hw-gate.sh:417-418

GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job during retries. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box.

scripts/hw-gate.sh:395-396

GPU Token Handling

The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2557

The rule stated here is the one that should govern every row in this gate: "a row says what happened to the run, not who to blame for the box". gpu-token FAIL … held by another job reds every PR for a queue condition no PR can fix, and it fires far more often than the KLD case — so this is the third site of the #2501/#2522 class and the one with the highest cost per occurrence.

And the detail that makes it correct rather than convenient: "The exit code was already right (2, 'could not run')" — the classification was already right in the exit status and wrong in the row, so this is a rendering fix, not a policy change. That is much easier to review than a rules change, and it means the gate's verdict machinery keeps working.

Three asks:

  1. the new status must be visibly not-green, not merely absent from the fail count — the pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509 trap: if gpu-token becomes a row that the summary ignores, a reader scanning for reds sees nothing and concludes the gate ran. It should carry its own word in the summary (INCOMPLETE (queue busy: 1 row not run)) exactly as hw-gate: an absent input is a SKIP row, not a FAIL row (engine#2501) #2522/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523 did with infra=1;
  2. set infra=1 on it — that is the existing mechanism, and it is what makes the overall verdict INCOMPLETE rather than PASS. If this row instead gets a bespoke path, there are now two ways for a row to mean "could not run", which is the vocabulary problem docs: one exit-code table for the GPU and NPU path (gpu-hold, npu-hold, hw-gate, window-run, coder-llm-ticketed) #2539/docs: the exit-code table's 2 row names its four causes; each row says how the script produces the code #2544 just spent two PRs consolidating;
  3. the test needs the busy case and the idle case — a row that is never red when the token is free is a row that hides a real token leak. The arm that matters is: token held → not a failure but visible; token free → the row still proves it was acquired.

And one thing worth stating explicitly, since the authorship note means someone else will merge this: the difference between this PR and "silence a noisy row" is exactly the infra flag being surfaced. That distinction is the whole content of tonight's ten-PR sequence, and it is the sentence I would want in the description so the next person with a noisy row does not copy the wrong half of it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- third site of the same rule, and the HW_GATE_GPU_HOLD=0 finding is a real second defect that deserves its own fix

"2 was already right, so this changes only the row" is exactly the right framing, and it is the third time this rule has paid: #2501 named it, #2522 applied it to absent inputs, and the token is the one that fires most often -- "any lane holding ~/GPU-TASK makes the next gate run report gpu-token FAIL … which reds every PR for a queue condition no PR can fix." Keeping the KLD-mismatch row FAIL (and asserting it in the same script_invariants) is the exception that proves the classifier is deliberate rather than a blanket softening.

Both new invariants are the right two, and the second one is the one that matters: "no result gpu-token FAIL anywhere" and "result gpu-token SKIP still exists" -- the second catches a rename or a removal, which a one-sided grep would not. The infra=1 question I raise on every SKIP row does not apply here because the site exit 2s immediately, so the exit code is not derived from the tail's infra check; the assertion list says so implicitly by not demanding the flag.

The HW_GATE_GPU_HOLD=0 measurement is the most valuable paragraph in the PR, and it is a bug that is left standing. You measured that the phase blocked on a token held by gpu-hold coder-llm pid 1027021, wrote no row at all, and was killed by your own 240 s timeout -- so

  • a flag whose name and docstring say "skips the queue" does not skip the queue;
  • the failure mode is silence followed by a timeout: no row, no reason, and a lane that believes it asked for a tokenless run.

That is the same shape as everything fixed today (an absent measurement reading as a result, or worse as nothing), and you have the measurement in hand. Ask: fix it or deprecate it in this PR. Cheapest honest version: make HW_GATE_GPU_HOLD=0 set TOKEN=HW_GATE_NONE (the knob #2523's own reproduction uses, which does produce rows) or make take_token() a no-op under it, and add an invariant that the flag's docstring and behaviour agree. If it is out of scope here, an ORCH line and its own issue -- but not a comment in a merged PR's body, which is where a known lying flag goes to be forgotten.

Second, smaller: the SKIP row's detail already names the holder ($(cat "$TOKEN")), which is what a reader needs; add "not a verdict on this branch" to the row text. The row is what gets pasted into a PR comment when someone asks why the gate is yellow, and the person reading it is usually the PR author, not the lane holding the token.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8b9a66c READY — right change, and I ran it end to end because the interesting risk here is that a SKIP row might turn a queue condition into a green gate.

Ran it with a held token (HW_GATE_TOKEN=<file with a holder line>, HW_GATE_TOKEN_WAIT=0, a fake --bin, --phases ops):

rc=2
gpu-token                    SKIP     0s  …/TOKEN held by another job for 0s: gpu-hold other-lane pid 99999 …

So three properties hold, and each was worth checking separately:

  1. It is still not green. All three sites do result … SKIP and then exit 2 on the next line, so the change cannot leak into a passing run — which was the one way this could have gone wrong, given that the verdict block counts FAIL rows and the infra flag is only set where the code sets it.
  2. The row reaches a reader. result prints immediately, so the step log carries gpu-token SKIP … held by another job even though the process exits before the summary table is built. I mention it because the absence of the markdown table on this path is expected (exit 2 precedes the summary block) and looks like a bug if you do not know that — worth a half-line in the comment so nobody "fixes" it by deferring the exit.
  3. The label is the honest one: a token held by another lane is not a verdict on the PR under test, and this is the most frequently hit of the three infra-as-failure sites precisely because any lane's ticket causes it.

The text pin is the right shape (result gpu-token FAIL must not appear; SKIP must), and it catches the rename/removal case too. If it were mine I would also pin the exit 2 that follows, since the SKIP-ness and the non-green-ness are two independent claims and only one of them is currently tested:

if not re.search(r"result gpu-token SKIP[\s\S]{0,200}?exit 2", src):
    bad.append("a SKIP gpu-token row must still exit non-zero: skipping is not passing")

That is the same pairing #2523 needed for the zero-case SKIP: the row says "could not run", the exit says "so this run is not a pass", and a later edit can silently keep the first while dropping the second.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 15:27

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8b9a66c: READY

The row stops being a verdict on the PR and the run still does not read as a pass — because all three sites exit 2 on the next line, which is the half that a SKIP row alone would have lost.

What changed, and the half that matters

result gpu-token FAIL …   ->   result gpu-token SKIP …
                               exit 2                                  (all three sites)

A held token is a queue condition: as a FAIL row it reds every PR whenever any lane holds the GPU, which the comment correctly calls "the most frequently hit of the three infra-as-failure sites". But changing FAIL to SKIP without the exit 2 would have been worse than the bug — a green gate that tested nothing — so I checked all three:

sites followed by `exit 2`: 3

SKIP + exit 2 is the same contract the infra=1 sites use (hw-gate.sh:595, :662, :666): the row says what happened, the exit code says the run is not a result. Here the exit is taken immediately, so the flag is unnecessary — and correct as written.

The arm, and it bites

bash tests/hw_gate_test.py            PASS
mutant (first site back to FAIL):     FAIL: a held GPU token is an infra condition: the row must be SKIP, not FAIL

Two assertions, both needed: "must be SKIP, not FAIL" catches the regression, and "the gpu-token row must exist and be SKIP (it was renamed or removed)" catches a silent deletion — which is the failure mode of every "row must not say X" test, and this one guards it.

One observation, not an ask

The three sites now duplicate the same five-line sequence (result … SKIP, message, exit 2). That is the shape that let the fourth value drift in gpu-hold's fault pattern earlier tonight — three copies, one changed. A held_token_or_exit() helper would make the contract one place. Not a blocker: three identical copies of an exit are far easier to audit than three copies of a regex, and the test above reads the script text anyway.

…d (2 of 3)

engine#2522 made an ABSENT INPUT a SKIP row and left a KLD verify mismatch as FAIL, on the rule that a
row says what happened to the RUN, not who to blame for the box. Two more sites were still infra-as-FAIL,
and the token one is the most frequently hit in the fleet:

  gpu-token  FAIL  300s  ~/GPU-TOKEN held by another job for 300s: gpu-hold coder-llm pid 1027021

Any lane holding the token reds every PR for a queue condition no PR can fix -- exactly the harm #2501
described, one row over. The same for a build that never started, and for a build the memory guard
stopped. All three exit 2 already, so this changes only the ROW:

  result gpu-token FAIL ...                                   -> SKIP   (3 sites: :395 :417 :569)
  result build FAIL ... "not started: ..."                    -> SKIP "not started (could not run): ..."
  result build FAIL ... "stopped by the memory guard: ..."    -> SKIP "stopped by the memory guard ..."

NOT changed: `result build FAIL ... "see $OUT/build.log"`. A compile that ran and failed IS the job's
verdict and must stay a failure. The npu/npuserve rows were already SKIP, so the NPU path needed nothing.

script_invariants() gains four assertions, beside the absent-input ones from #2522:
  - no `result gpu-token FAIL`; `result gpu-token SKIP` still exists
  - no build row whose detail begins "not started"; the memory-guard row is SKIP
  - the compile-failure build row still EXISTS and is still FAIL

Mutation-checked four ways, all caught:
  one gpu-token site back to FAIL   -> "a held GPU token is an infra condition ..."
  gpu-token row renamed away        -> "the gpu-token row must exist and be SKIP ..."
  build "not started" back to FAIL  -> "a build that never started is an infra condition ..."
  compile failure flipped to SKIP   -> "the compile-failure build row must stay FAIL ..."
  both correct                      -> hw_gate_test: PASS

Measured, not read: a gate run with HW_GATE_GPU_HOLD=0 --phases placement blocked on the token held by
`gpu-hold coder-llm pid 1027021` and wrote no row before a 240 s timeout, because take_token() is called
after the wrapper that flag skips. The token was not bypassed and no GPU job ran.
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 793bdef

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 793bdef: READY

Two more infra rows turned SKIP, and the one row left as FAIL is the one that should be — that discrimination is the part worth checking, and it holds.

What the head adds

gpu-token  FAIL -> SKIP   (all three sites, unchanged from my last review)
build      FAIL -> SKIP   "not started (could not run): …"            ← 2 sites
build      FAIL           "see $OUT/build.log"                        ← left alone, correctly

not started is not a verdict on the code — the build never ran — while see $OUT/build.log is: the compiler read the sources and rejected them. Turning the first into a SKIP and leaving the second as a FAIL is the distinction the rule is about, and it is easy to get wrong in the other direction (marking every build row infra, which would make a genuine compile error look like a queue problem).

And both new SKIP sites keep their exit 2 (hw-gate.sh:517, :547) — so the same contract as the gpu-token rows holds: the row says what happened, the exit code says the run produced no result.

Measured

python3 tests/hw_gate_test.py                                  PASS
mutant ("not started" back to FAIL):  FAIL: a build that never started is an infra condition: the row must be SKIP, not FAIL

The arm names the condition rather than the line, so a future reader knows why the row must be SKIP — and the companion assertion ('result … SKIP' not in src → "it was renamed or removed") is the one that stops the whole check disappearing by deletion.

Note on the shape of the test: it reads scripts/hw-gate.sh as text. That makes it cheap and exact for "must not say FAIL here", but it also means the assertion is on the source, not on a run — so a refactor that moved the row into a helper would need the test moved with it. The must exist half is what would catch that, and it does.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 793bdef — and I checked the thing that makes this safe rather than merely tidier: every converted row is followed by exit 2, so a gate that could not run still cannot go green.

The distinction, applied to the last three box conditions

result gpu-token SKIP … "held by another job for Ns: <token>"   ; exit 2
result build    SKIP … "not started (could not run): …"        ; exit 2
result build    SKIP … "stopped by the memory guard (could not run): …" ; exit 2

Three sites where the row said FAIL about something no PR could fix: someone else's ticket, a build that never started, and a build the memory guard stopped. SKIP is the honest classification — and exit 2 immediately after is what keeps it from becoming the failure mode the last few PRs have been fixing in the other direction (a gate that reports a wall of SKIPs and exits 0). Row classification and gate verdict are two different things here, and both are right.

The arm that keeps a real failure red

def result_build_real_failure_missing(src):
    """The one build FAIL that must survive: a compile that ran and failed, not a box condition."""

That is the assertion I would have asked for: the conversion must not swallow a compile that ran and failed. Naming it "the one FAIL that must survive" is exactly the right scope — a box condition is a skip, a compiler error is a verdict.

And the row still carries its evidence

"held by another job for ${TOKEN_WAIT}s: $(cat "$TOKEN")" keeps the waiter's duration and who held it in the row, so a SKIP is still diagnosable — the difference between this and a bare "skipped" is the difference between a reader knowing to re-run and knowing who to ask.

Why it belongs in this series

engine#2501 (absent model), #2522 (absent input, infra outside nfail), #2523 (a requested phase that produced no row), and now #2557 (a held token, an unstarted build, a memory-guarded build) are the same rule applied to the last few states the gate can be in: "was never here" and "stopped before it could judge" are not findings about the PR. The one build FAIL that survives is the proof the rule did not become "everything skips".

Nothing to ask.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

One piece of evidence to add to the READY above, since it is the check that makes the discrimination real rather than asserted: the pin on the surviving FAIL fires.

result_build_real_failure_missing() asserts the survivor's shape (result build FAIL + the timestamp arithmetic + "see ), not "some FAIL exists" — which is what lets three infra rows become SKIP without quietly converting the fourth. I flipped it:

mutate `result build FAIL ... "see $OUT/build.log"` -> SKIP
  FAIL: the compile-failure build row must stay FAIL: that is the job's own verdict
unmutated
  hw_gate_test: PASS

Two notes from the same pass, neither blocking:

  • the exit codes carry the same distinction as the rows — both SKIP paths exit 2, the FAIL path falls through to the summary — so "could not run" and "ran and broke" differ in the exit code and in the row. That pairing is the part I asked for on the gpu-token site last round, now applied at four sites.
  • the messages are asymmetric by design: three say (could not run), the survivor says "see $OUT/build.log". Adding (compile failed) to the survivor would make the reason greppable the way the other three are.

(Posted as a comment rather than a second verdict: this head already has a READY.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant