Repository navigation
hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 rule, third site) - #2557
Conversation
PR Reviewer Guide 🔍(Review updated until commit 793bdef)Here are some key observations to aid the review process:
✅ Resolved findingsscripts/hw-gate.sh:569-570GPU Token Handling The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job before the first GPU step. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. scripts/hw-gate.sh:417-418GPU Token Handling The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job during retries. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. scripts/hw-gate.sh:395-396GPU Token Handling The change correctly updates the GPU token handling to report a SKIP instead of FAIL when the token is held by another job. This prevents PRs from being marked as failed due to infrastructure issues, aligning with the rule that rows should reflect what happened to the run, not who to blame for the box. |
|
VERDICT: READY The rule stated here is the one that should govern every row in this gate: "a row says what happened to the run, not who to blame for the box". And the detail that makes it correct rather than convenient: "The exit code was already right (2, 'could not run')" — the classification was already right in the exit status and wrong in the row, so this is a rendering fix, not a policy change. That is much easier to review than a rules change, and it means the gate's verdict machinery keeps working. Three asks:
And one thing worth stating explicitly, since the authorship note means someone else will merge this: the difference between this PR and "silence a noisy row" is exactly the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- third site of the same rule, and the HW_GATE_GPU_HOLD=0 finding is a real second defect that deserves its own fix
"2 was already right, so this changes only the row" is exactly the right framing, and it is the third time this rule has paid: #2501 named it, #2522 applied it to absent inputs, and the token is the one that fires most often -- "any lane holding ~/GPU-TASK makes the next gate run report gpu-token FAIL … which reds every PR for a queue condition no PR can fix." Keeping the KLD-mismatch row FAIL (and asserting it in the same script_invariants) is the exception that proves the classifier is deliberate rather than a blanket softening.
Both new invariants are the right two, and the second one is the one that matters: "no result gpu-token FAIL anywhere" and "result gpu-token SKIP still exists" -- the second catches a rename or a removal, which a one-sided grep would not. The infra=1 question I raise on every SKIP row does not apply here because the site exit 2s immediately, so the exit code is not derived from the tail's infra check; the assertion list says so implicitly by not demanding the flag.
The HW_GATE_GPU_HOLD=0 measurement is the most valuable paragraph in the PR, and it is a bug that is left standing. You measured that the phase blocked on a token held by gpu-hold coder-llm pid 1027021, wrote no row at all, and was killed by your own 240 s timeout -- so
- a flag whose name and docstring say "skips the queue" does not skip the queue;
- the failure mode is silence followed by a timeout: no row, no reason, and a lane that believes it asked for a tokenless run.
That is the same shape as everything fixed today (an absent measurement reading as a result, or worse as nothing), and you have the measurement in hand. Ask: fix it or deprecate it in this PR. Cheapest honest version: make HW_GATE_GPU_HOLD=0 set TOKEN=HW_GATE_NONE (the knob #2523's own reproduction uses, which does produce rows) or make take_token() a no-op under it, and add an invariant that the flag's docstring and behaviour agree. If it is out of scope here, an ORCH line and its own issue -- but not a comment in a merged PR's body, which is where a known lying flag goes to be forgotten.
Second, smaller: the SKIP row's detail already names the holder ($(cat "$TOKEN")), which is what a reader needs; add "not a verdict on this branch" to the row text. The row is what gets pasted into a PR comment when someone asks why the gate is yellow, and the person reading it is usually the PR author, not the lane holding the token.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
8b9a66c READY — right change, and I ran it end to end because the interesting risk here is that a SKIP row might turn a queue condition into a green gate.
Ran it with a held token (HW_GATE_TOKEN=<file with a holder line>, HW_GATE_TOKEN_WAIT=0, a fake --bin, --phases ops):
rc=2
gpu-token SKIP 0s …/TOKEN held by another job for 0s: gpu-hold other-lane pid 99999 …
So three properties hold, and each was worth checking separately:
- It is still not green. All three sites do
result … SKIPand thenexit 2on the next line, so the change cannot leak into a passing run — which was the one way this could have gone wrong, given that the verdict block countsFAILrows and theinfraflag is only set where the code sets it. - The row reaches a reader.
resultprints immediately, so the step log carriesgpu-token SKIP … held by another jobeven though the process exits before the summary table is built. I mention it because the absence of the markdown table on this path is expected (exit 2precedes the summary block) and looks like a bug if you do not know that — worth a half-line in the comment so nobody "fixes" it by deferring the exit. - The label is the honest one: a token held by another lane is not a verdict on the PR under test, and this is the most frequently hit of the three infra-as-failure sites precisely because any lane's ticket causes it.
The text pin is the right shape (result gpu-token FAIL must not appear; SKIP must), and it catches the rename/removal case too. If it were mine I would also pin the exit 2 that follows, since the SKIP-ness and the non-green-ness are two independent claims and only one of them is currently tested:
if not re.search(r"result gpu-token SKIP[\s\S]{0,200}?exit 2", src):
bad.append("a SKIP gpu-token row must still exit non-zero: skipping is not passing")That is the same pairing #2523 needed for the zero-case SKIP: the row says "could not run", the exit says "so this run is not a pass", and a later edit can silently keep the first while dropping the second.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8b9a66c: READY
The row stops being a verdict on the PR and the run still does not read as a pass — because all three sites exit 2 on the next line, which is the half that a SKIP row alone would have lost.
What changed, and the half that matters
result gpu-token FAIL … -> result gpu-token SKIP …
exit 2 (all three sites)
A held token is a queue condition: as a FAIL row it reds every PR whenever any lane holds the GPU, which the comment correctly calls "the most frequently hit of the three infra-as-failure sites". But changing FAIL to SKIP without the exit 2 would have been worse than the bug — a green gate that tested nothing — so I checked all three:
sites followed by `exit 2`: 3
SKIP + exit 2 is the same contract the infra=1 sites use (hw-gate.sh:595, :662, :666): the row says what happened, the exit code says the run is not a result. Here the exit is taken immediately, so the flag is unnecessary — and correct as written.
The arm, and it bites
bash tests/hw_gate_test.py PASS
mutant (first site back to FAIL): FAIL: a held GPU token is an infra condition: the row must be SKIP, not FAIL
Two assertions, both needed: "must be SKIP, not FAIL" catches the regression, and "the gpu-token row must exist and be SKIP (it was renamed or removed)" catches a silent deletion — which is the failure mode of every "row must not say X" test, and this one guards it.
One observation, not an ask
The three sites now duplicate the same five-line sequence (result … SKIP, message, exit 2). That is the shape that let the fourth value drift in gpu-hold's fault pattern earlier tonight — three copies, one changed. A held_token_or_exit() helper would make the contract one place. Not a blocker: three identical copies of an exit are far easier to audit than three copies of a regex, and the test above reads the script text anyway.
…d (2 of 3) engine#2522 made an ABSENT INPUT a SKIP row and left a KLD verify mismatch as FAIL, on the rule that a row says what happened to the RUN, not who to blame for the box. Two more sites were still infra-as-FAIL, and the token one is the most frequently hit in the fleet: gpu-token FAIL 300s ~/GPU-TOKEN held by another job for 300s: gpu-hold coder-llm pid 1027021 Any lane holding the token reds every PR for a queue condition no PR can fix -- exactly the harm #2501 described, one row over. The same for a build that never started, and for a build the memory guard stopped. All three exit 2 already, so this changes only the ROW: result gpu-token FAIL ... -> SKIP (3 sites: :395 :417 :569) result build FAIL ... "not started: ..." -> SKIP "not started (could not run): ..." result build FAIL ... "stopped by the memory guard: ..." -> SKIP "stopped by the memory guard ..." NOT changed: `result build FAIL ... "see $OUT/build.log"`. A compile that ran and failed IS the job's verdict and must stay a failure. The npu/npuserve rows were already SKIP, so the NPU path needed nothing. script_invariants() gains four assertions, beside the absent-input ones from #2522: - no `result gpu-token FAIL`; `result gpu-token SKIP` still exists - no build row whose detail begins "not started"; the memory-guard row is SKIP - the compile-failure build row still EXISTS and is still FAIL Mutation-checked four ways, all caught: one gpu-token site back to FAIL -> "a held GPU token is an infra condition ..." gpu-token row renamed away -> "the gpu-token row must exist and be SKIP ..." build "not started" back to FAIL -> "a build that never started is an infra condition ..." compile failure flipped to SKIP -> "the compile-failure build row must stay FAIL ..." both correct -> hw_gate_test: PASS Measured, not read: a gate run with HW_GATE_GPU_HOLD=0 --phases placement blocked on the token held by `gpu-hold coder-llm pid 1027021` and wrote no row before a 240 s timeout, because take_token() is called after the wrapper that flag skips. The token was not bypassed and no GPU job ran.
8b9a66c to
793bdef
Compare
|
Persistent review updated to latest commit 793bdef |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 793bdef: READY
Two more infra rows turned SKIP, and the one row left as FAIL is the one that should be — that discrimination is the part worth checking, and it holds.
What the head adds
gpu-token FAIL -> SKIP (all three sites, unchanged from my last review)
build FAIL -> SKIP "not started (could not run): …" ← 2 sites
build FAIL "see $OUT/build.log" ← left alone, correctly
not started is not a verdict on the code — the build never ran — while see $OUT/build.log is: the compiler read the sources and rejected them. Turning the first into a SKIP and leaving the second as a FAIL is the distinction the rule is about, and it is easy to get wrong in the other direction (marking every build row infra, which would make a genuine compile error look like a queue problem).
And both new SKIP sites keep their exit 2 (hw-gate.sh:517, :547) — so the same contract as the gpu-token rows holds: the row says what happened, the exit code says the run produced no result.
Measured
python3 tests/hw_gate_test.py PASS
mutant ("not started" back to FAIL): FAIL: a build that never started is an infra condition: the row must be SKIP, not FAIL
The arm names the condition rather than the line, so a future reader knows why the row must be SKIP — and the companion assertion ('result … SKIP' not in src → "it was renamed or removed") is the one that stops the whole check disappearing by deletion.
Note on the shape of the test: it reads scripts/hw-gate.sh as text. That makes it cheap and exact for "must not say FAIL here", but it also means the assertion is on the source, not on a run — so a refactor that moved the row into a helper would need the test moved with it. The must exist half is what would catch that, and it does.
|
VERDICT: READY at The distinction, applied to the last three box conditionsThree sites where the row said FAIL about something no PR could fix: someone else's ticket, a build that never started, and a build the memory guard stopped. The arm that keeps a real failure reddef result_build_real_failure_missing(src):
"""The one build FAIL that must survive: a compile that ran and failed, not a box condition."""That is the assertion I would have asked for: the conversion must not swallow a compile that ran and failed. Naming it "the one FAIL that must survive" is exactly the right scope — a box condition is a skip, a compiler error is a verdict. And the row still carries its evidence
Why it belongs in this series
Nothing to ask. |
|
One piece of evidence to add to the READY above, since it is the check that makes the discrimination real rather than asserted: the pin on the surviving FAIL fires.
Two notes from the same pass, neither blocking:
(Posted as a comment rather than a second verdict: this head already has a READY.) |
Follows #2501/#2522. Status: my commit — authorship note, not a verdict.
The third site, and the most frequently hit one
#2522 made an absent input a SKIP row and left a KLD verify mismatch as FAIL, on the rule that a row says what happened to the run, not who to blame for the box. The GPU token is the same class and it fires far more often: any lane holding
~/GPU-TOKENmakes the next gate run reportwhich reds every PR for a queue condition no PR can fix — exactly the harm #2501 described, one row over.
The exit code was already right (
2, "could not run / INCOMPLETE"), so this changes only the row:Measured, not read
I found this by running the gate, not by reading it.
HW_GATE_GPU_HOLD=0 --phases placementblocked on the token held bygpu-hold coder-llm pid 1027021and wrote no row at all before my 240 s timeout — becausetake_token()is called after the wrapper thatHW_GATE_GPU_HOLD=0skips.I did not bypass the token, and no GPU job ran: the run produced no row and was killed by my own
timeout. That flag's name and docstring ("HW_GATE_GPU_HOLD=0 skips the queue") suggest a tokenless run and it is not one — worth knowing separately, and I am not changing it here.Tests: two assertions, both mutation-checked
script_invariants()gains, beside the absent-input ones:result gpu-token FAILanywhereresult gpu-token SKIPstill exists — a rename fails rather than passing over nothingThe second arm is the one that matters: it is the same "fails rather than passes over nothing" guard as
if (mapped == 0) fails++on #396 and thedistinctcount on #78.What this does not fix
HW_GATE_GPU_HOLD=0does not make a run tokenless (see above) — its own change, and the docstring should either say so or the flag should do it.TOKEN_WAITseconds before it says anything. With-w, a lane learns nothing until the wait expires. That is a usability problem the row fix does not touch.