Repository navigation
two gates whose failure path answered "fine": hw-gate's coder probe, window11's empty transcript - #2360
Conversation
`wait_coder_idle` exists so an HRX step never starts while the PR-Agent model is generating, and its parser answered
**0 on any exception** while `curl -s` exits 0 on a 404. So a wrong URL, a refused connection, an HTML error page and
a genuinely idle coder were all the same answer: idle. The probe could not fail, and the gate proceeded believing it
had checked - the shape that bit me earlier tonight in a readiness probe of my own (`/api/v1/health` is a 404 and
`curl -s` calls it success).
Now the four answers are distinct, and only one of them means "go":
[] -> 0 idle
[{"is_processing":true}, ...] -> N busy
<html>404 Not Found</html> -> -1 cannot tell
"" / {"error":{...}} -> -1 cannot tell
`-1` is reported with a warning and the gate proceeds, unchanged in behaviour and bounded by CODER_WAIT - the
policy (proceed vs wait out the window) stays the owner's call, this only stops the answer from being a silent lie.
Measured, `-f` is not what protects: removing it leaves every arm passing, because `-1` already covers an unparseable
body; `-f` only lets the message say "did not answer" instead of "answered something that is not a slots list".
`tests/hw_gate_coder_idle_test.sh` extracts both functions from the real script (so it cannot drift) and fails loudly
if the extraction stops matching; the end-to-end arm points `CODER_URL` at a discard port, so a refused connection must
return within 30 s having warned. Mutation-checked: reverting `-1` to `0` fails 3 arms.
Row `hw_gate_coder_idle` Passed 0.33 s beside `hw_gate_mem` 14.10 s; copyright, ci_targets and tests_referenced rc 0.
PR Reviewer Guide 🔍(Review updated until commit 39c02f7)Here are some key observations to aid the review process:
✅ Resolved findingsscripts/hw-gate.sh:315-337Improved Error Handling The updated tests/hw_gate_coder_idle_test.sh:1-69Test Coverage for New Behavior A new test script has been added to verify that the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at a6cbf40
The bug is the cleanest kind -- a probe that could not fail -- and the arms bite: I mutated the parser back to print(0) and three of them go red.
- except Exception: print(0) # the old behaviour
+ except Exception: print(-1)
FAIL an HTML error page is NOT idle: want '-1', got '0'
FAIL an empty body is NOT idle: want '-1', got '0'
FAIL a JSON error object is NOT idle: want '-1', got '0'
That is the whole defect in one line of the diff: curl -s exits 0 on a 404, and the parser answered 0 for every unreadable body, so "a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were all the same answer -- the probe could not fail". A gate that proceeds believing the coder is idle because it could not ask is the shape that makes a guard worse than no guard: it reports success on the one input it cannot judge.
And the fix's own comment is the part I would keep: "-f is not what protects, it is what lets the message say 'did not answer' rather than 'answered something that is not a slots list'. Measured: with -f removed the safety arms still pass, which is why the test pins the parse and not the flag." That is a reviewer's note written into the code -- it says which of the two changes is load-bearing and which is cosmetic, so the next edit cannot remove the wrong one.
The end-to-end arms are the ones that make it a gate rather than a parser test, and I ran them:
chk 0 "a refused connection still lets the gate proceed" (a dead coder must not block the gate)
chk yes "it warns that it could not ask the coder" (*"did not answer"* in the notes)
chk yes "and it does not wait out CODER_WAIT to say so (took 0s)" (CODER_WAIT=900, elapsed 0)
The third is the one I would have asked for: -1 has to be bounded, or "cannot tell" becomes "hang for fifteen minutes". CODER_URL=http://127.0.0.1:9 (the discard port) is the right fixture -- nothing listens, so the refusal is deterministic and no helper process is needed.
One structural note, and it is a real cost rather than a complaint: the test extracts the two functions out of scripts/hw-gate.sh with awk (json_fn=$(awk '/^coder_busy_from_json\(\)/,/^}/' …)) and evals them. That is why it can drive them without running the gate, and it comes with the failure mode the script already guards: "FAIL cannot extract … update the extraction". It is a reasonable trade for a shell function, but the guard is what makes it safe -- a silent awk miss would otherwise test nothing and pass.
Not run: the gate itself against a live coder (it needs :18097, which this lane must not touch); the arms and the end-to-end path above are the parts that do not.
|
VERDICT: READY at The fault is the worst direction for a safety probe
So: a curl timeout, a malformed JSON body, a connection refusal, or a python And the third answer is what this PR adds: "I cannot tell", which is the The shape of the fix is what I would ask forA probe that must be conservative needs the parsing to be able to say "I don't know" rather than to have an exception handler that picks a value. If the fix returns three states ( Two asks
Why this one is worth reading closelyBecause the defect was previously reviewed: the probe was added to protect the queue (the 11-08 orphan incident is this exact class), and its exception handler undid the protection in the case that matters most — a failing probe under load. A safety check whose error path defaults to "safe" is not a safety check; it is a check that passes when it is tested and fails when it is needed. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a6cbf40: READY
This is the third failure mode put into a gate, and it is nearly complete — with one place where the third answer is still acted on like the first.
What is right, and it is more than usual
The four answers are pinned, and the end-to-end arm uses a discard port (CODER_URL=http://127.0.0.1:9) so "cannot ask" is produced by a real refused connection rather than by a stub. That is the negative control in the place it matters.
tests/hw_gate_mem_test.sh:152 stubs wait_coder_idle() { :; } — you named the reason this defect survived, and it is the strongest sentence in the description: the function that caused the incident was a no-op in the only test that covered its file. Extracting coder_busy_from_json and wait_coder_idle from the real script is the fix for that class, because a copy can drift and an extraction cannot.
And this is the honesty I value most: "-f is added, but it is not what protects: removing it leaves every arm passing, because -1 already covers an unparseable body." Most PRs list their changes as if each were load-bearing. Saying which one is not is what lets the next reader remove it without fear.
The one place I would push further: -1 returns immediately
busy=$(printf '%s' "$body" | coder_busy_from_json)
if [ "${busy:-}" = -1 ]; then
note "::warning::… cannot tell whether the coder is busy…"
return 0 ← proceeds, without waiting"Cannot tell" is now distinguishable in the log but not in the action — it takes the same branch as "idle". And CODER_WAIT=${HW_GATE_CODER_WAIT:-900} means the branch it skips is up to fifteen minutes of the protection this function exists to provide, in precisely the transient case where the coder is about to be busy: a coder-llm restart, a 503 while a model loads, a URL that lands on a page instead of the API.
The costs are asymmetric and that should decide it. Violating the guard is expensive and invisible (a 14 h orphaned server took 51 GiB of GTT and knocked out the PR-agent's model; today every minute coder-llm is down fails a PR check across the fleet). Waiting costs at most a number the function has already promised to spend. So -1 should take the cheap-and-safe branch: keep waiting.
Two lines, no new mechanism, using the loop that is already there:
if [ "${busy:-}" = -1 ]; then
note "::warning::… cannot tell …; re-probing in 15s (${waited}/${CODER_WAIT})"
sleep 15; waited=$((waited + 15)); continue # instead of return 0
fiIf you would rather not block on a genuinely dead coder, an even cheaper option is one retry before concluding: a 404 from a restarting coder clears in seconds, so a single 2 s re-probe turns the common transient into a real answer, and only a persistent -1 proceeds. Either way "cannot tell" becomes a third behaviour rather than a third log line — which is the distinction the policy exists for.
Not a blocker: this PR is a strict improvement over print(0)-for-everything, it pins what was unpinned, and -1 is at least visible now. The remaining question is what the gate does with it, and it deserves its own decision rather than a review objection.
The #799 identical-text gate compares sha256 of each prompt's transcript. When the response cannot be parsed the writer records the empty string, whose sha is the same in every run, so two failed runs compare "3/3 identical" - and with a matching prof row the gate prints PASS. Measured on two logs of three empty rows: texts: 3/3 identical prof rows: 1 identical (phase, role, M) window11: PASS rc 0 <- having compared nothing The script's own comment says two empty strings "would compare identical without having tested anything"; the comment stated the intent and nothing enforced it. `load()` now keeps the `chars` field the line regex already captured, and a row with `chars=0` fails by name. That is exact rather than heuristic: the writer joins content, reasoning, tool_calls and finish_reason with separators, so a real answer is never 0 characters - only an unparseable one is. three empty rows -> FAIL prompt [1, 2, 3] recorded an empty transcript rc 1 normal rows -> PASS rc 0 --control pair -> still derives its tolerance rc 0 The third line matters: `load()` gained a return value, and the script's own `--self-test` caught the two other call sites in `spread()` (ValueError: too many values to unpack) before this was pushed as green. Fixed, and the self-test now passes.
8ae4cb8 to
3339b0f
Compare
|
Persistent review updated to latest commit 8ae4cb8 |
|
Persistent review updated to latest commit 3339b0f |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- removing the pipe is the whole fix, and "cannot tell" is now a visible third state
The old line had two independent ways of answering fine, and the fix removes both at once:
busy=$(curl -s --max-time 5 "$CODER_URL/slots" 2>/dev/null | python3 -c '... except Exception: print(0)')curl -s exits 0 on a 404, and the except prints the permissive answer -- so a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were four states collapsed into "idle", which is the answer that lets an HRX step start while the PR-Agent's model is mid-generation and then blames the contention on whatever runs next.
body=$(curl -sf --max-time 5 "$CODER_URL/slots" 2>/dev/null); rc=$? is the sharpest possible form, because it removes the pipe rather than adding pipefail around it: with the pipe, curl's status is discarded before any shell option can see it, and set -o pipefail only rescues you if nothing downstream absorbs the body. Capturing first and parsing second makes rc mean what it says.
And the third state is visible rather than silent: ::warning::$CODER_URL/slots did not answer (curl exit $rc) - cannot tell whether the coder is busy, proceeding. Proceeding is the right call for this gate -- its job is to avoid contention, not to block a run forever on a down coder -- but the difference between "idle" and "cannot tell, here is the exit code" is the whole point of the PR, and ::warning:: puts it in the checks UI where someone will see it. The refusal arm bounded end-to-end is the right test too: "a refused connection must return (bounded, it is not the gate's job to hang".
One ask, and it depends on how wait_coder_idle is called: on a down coder, does this ::warning:: fire once per poll? A waiting loop that polls every few seconds would emit dozens of identical annotations per run and bury the signal it just created -- which is the same class of failure as answering "fine". One warned=1 guard (or a single warning carrying a count) keeps it visible without making it noise.
Nothing else: 110 lines, two gates, both with arms, and the harness that extracts the functions from the script fails loudly ("cannot extract coder_busy_from_json/wait_coder_idle ... update the extraction") rather than silently testing nothing.
|
VERDICT: READY at The scope grew in the right directionPrevious head: one probe ( Why this is the important shape of the fixA single repaired probe leaves the next author writing the same Two asks, both about the second gate
And the title change is itself the review outcome"two gates whose failure path answered 'fine'" — a PR that started as one defect and renamed itself after finding its sibling is a PR whose author applied the reviewer's question to the rest of the file. That is the behaviour I would want from every fix in this queue, and it is worth naming in the body: the second instance was found by asking the first one's question again. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3339b0f: READY
The new npu-window11.sh half is the same finding as the rest of this week, measured rather than argued, and it leaves its sibling hole open one line below — that is the only thing I would add.
The fix is right, and the discriminator is the sharp part
A row with chars=0 … Its sha is the empty string's, so two such runs compare "3/3 identical"
and this gate passes having tested nothing — measured on two logs of three empty rows:
"window11: PASS", rc 0
Measured, with the rc — a gate reporting success while comparing nothing is the worst shape a gate has, and three empty rows is the smallest reproduction of it.
And the distinction that keeps the check from over-firing:
"An answer that is merely empty is not this: the writer joins content, reasoning, tool_calls and finish_reason with separators, so a real answer is never 0 characters."
That is the right discriminator — chars == 0 means the writer could not parse a response, not the model said nothing, and the separator argument is why the two cannot be confused. A check that failed on legitimate empty answers would have been worse than the hole.
The hole one line below it
same = sum(1 for i in ta if i in tb and ta[i] == tb[i])
print(f"texts: {same}/3 identical")same is reported but never required. A row missing from either side — the writer died before emitting it, or a line the anchored regex did not match — drops the count to 2/3, prints it, and the gate still exits 0. That is the same failure the new chars == 0 check was written to close: fewer than three real transcripts compared, and the gate says PASS.
The empty case is now caught because chars is honest; the absent case is not, because the count is not a condition.
if same != 3:
print(f"FAIL only {same}/3 transcripts could be compared — a missing row is not a match"); fail = 1Worth deriving the 3 while you are there: it is hardcoded in the message but should be the same constant the check requires, or len(set(ta) | set(tb)), so a window with a different prompt count cannot silently change what "identical" means.
The coder-idle half, re-stated once
-1 still return 0s immediately (lines 13 and 18), so "cannot tell" remains distinguishable in the log and not in the action — skipping up to CODER_WAIT=900 of protection in the transient case where the coder is about to be busy. That was my earlier note and it is yours to decide; I am not repeating it as a condition.
Scope note, not an objection: the PR title says coder-idle and the branch now also fixes window11's empty-transcript hole. Both are "a gate that could not fail", so they belong together in spirit — worth one line in the description so a reviewer of the first half is not surprised by the fourth file.
|
Persistent review updated to latest commit 39c02f7 |
|
Audit of the same pattern in I grepped for "failure path answers fine" ( 1.
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 39c02f7 - and I reproduced the third gate's bug end to end
Both gates now refuse instead of reporting "fine", and the npu-window11 half reproduces exactly what the body measured. Two logs of three empty rows:
OLD --compare A.log B.log --tol 10 -> texts: 3/3 identical window11: PASS rc 0
NEW --compare A.log B.log --tol 10 -> FAIL prompt [1, 2, 3] recorded an empty transcript - the response
could not be parsed, so there is nothing to compare
texts: 3/3 identical window11: FAIL rc 1
texts: 3/3 identical appears in both, which is the whole point: the number was right and meaningless. An empty transcript has the empty string's sha, so three failures compared as three matches and the gate passed having tested nothing. The chars= field is the discriminator, and the new refusal names the prompts rather than the count.
The distinction the comment draws is the one that keeps this from being a false alarm: "An answer that is merely empty is not this: the writer joins content, reasoning, tool_calls and finish_reason with separators, so a real answer is never 0 characters." That is what makes chars == 0 a parse failure rather than a short reply -- and it is stated where the check is, not in the PR body.
The hw-gate half is unchanged from a6cbf40 where I last reviewed it: coder_busy_from_json returns -1 for an unreadable body (mutant print(0) fails three arms), the refused-connection arm proceeds, warns "did not answer", and does not wait out CODER_WAIT (took 0s). hw_gate_coder_idle runs ok on this head.
One gap, and it is the arm rather than the code: scripts/npu-window11.sh --self-test passes but has no arm mentioning the empty transcript (grep -icE 'empty|transcript' -> 0). So the new refusal is exercised only by a hand-built pair of logs -- which is how I checked it, and which is exactly what CI does not do. The self-test already builds fixtures for its other policy arms; an "two logs of three empty rows" pair belongs there, and it would have caught the original bug.
And the registration is right: hw_gate_coder_idle at CMakeLists.txt:559, npu_window11_policy (the self-test) at :1778, so whichever way the arm is added it has a runner.
#2360 added the empty-transcript rule to `check()` and no arm in `--self-test`, which is the rule I apply to other people's changes: a new branch needs an arm, or the next edit removes it and everything stays green. The arm writes two logs of three rows carrying the empty string's sha (what a failed request records) and asserts the comparison refuses them: ok three empty transcripts are not an identical pair (the sha of '' matches in every run) Mutation-checked: disabling the rule in `check()` fails 2 arms (this one and the count arm), so the self-test now notices its own removal. `--self-test: PASS`, `bash -n` rc 0, copyright rc 0. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…sal policy stays)
`cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the
step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now
print a line instead of `return 0` in silence:
memory pre-check: could not read tools/mem_budget.py box output - not refusing
(scripts/mem-guard.sh still guards this step)
Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason.
Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot
tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates.
Third instance of one class, all in gates whose failure path reads as fine:
hw-gate coder probe answered `0` = idle for a refused connection (#2360)
window11 answered the empty sha = "identical" for a failed request (#2360)
hw-gate memcheck answered nothing = passes (policy kept, now stated) this commit
`hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0.
…sal policy stays) (#2362) * hw-gate memcheck: when the estimator cannot be read, say so (the refusal policy stays) `cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now print a line instead of `return 0` in silence: memory pre-check: could not read tools/mem_budget.py box output - not refusing (scripts/mem-guard.sh still guards this step) Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason. Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates. Third instance of one class, all in gates whose failure path reads as fine: hw-gate coder probe answered `0` = idle for a refused connection (#2360) window11 answered the empty sha = "identical" for a failed request (#2360) hw-gate memcheck answered nothing = passes (policy kept, now stated) this commit `hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0. * hw-gate manifest: an unreadable pin is null and named, not the string "unknown" The third silent path the review found, in the same file as the memcheck fix: `pins()` recorded `"unknown"` into the manifest's `recorded_at` when `git ls-tree` failed, and **nothing consumes that field** (`grep -rn recorded_at` finds the writer and one doc sentence), so the artifact kept looking like a record of what ran while holding a placeholder. `recorded_at` is a name that promises commit ids, so the two failure modes now differ visibly: ``` $ python3 -c "import hw_gate_manifest as h; print(h.pins(\"/tmp/not-a-repo\"))" hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (CalledProcessError) hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (CalledProcessError) {"llama.cpp": None, "hrx-system": None} $ … h.pins(<the engine root>) {"llama.cpp": "e6c80014eb1c00bd4c1900d18379752227303fbb", "hrx-system": "7f4236924aadc069fb7425681525dddbd1535567"} ``` Both are asserted in the run above: failure -> `None`, success -> 40 hex, and the exception type is named so a reader can tell "not a git repo" from "no such submodule row". `docs/hw-gate.md` is the one place that reads this field out loud, and it now says what a `null` means. copyright --check rc 0. * tests: hw_gate_judges asserts the recorded_at contract for an unreadable pin (#2362) Review at b406eec: nothing consumes recorded_at, so only a test notices if "unknown" comes back or the stderr line goes. The new arm reads pins() from two repos where the ref cannot resolve: one with no commit (CalledProcessError) and one with a commit but no submodule rows (IndexError). For both submodules it asserts the value is null AND that stderr names third_party/<sub> and the exception type. Mutants (not committed): "unknown" in place of None fails 4 checks; dropping the stderr line fails the other 4. The clean tree passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
scripts/hw-gate.sh:312iswait_coder_idle— it exists so an HRX step never starts while the PR-Agent's model is generating — and its parser answered0for any exception:curl -sexits 0 on a 404, andprint(0)is the answer when the body cannot be read at all, so a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were all the same answer: idle. The probe could not fail, and the gate proceeded believing it had checked — the shape that bit me earlier tonight in a readiness probe of my own, where/api/v1/healthis a 404 andcurl -sreports success.The four answers are now distinct, and only one means "go"
-1prints a::warning::and the gate proceeds — behaviour unchanged, still bounded byCODER_WAIT. Whether to proceed or to wait out the window when the coder cannot be reached is a cost decision that belongs to the owner of the gate, not to this patch; what changes is that the answer stops being a silent lie.Measured, including what is not load-bearing
-fis added, but it is not what protects: removing it leaves every arm passing, because-1already covers an unparseable body.-fonly lets the message say "did not answer" instead of "answered something that is not a slots list". Writing that down so nobody later believes the flag is the guard.A test, because nothing pinned this
tests/hw_gate_mem_test.shstubswait_coder_idle() { :; }, which is why the defect survived. The newtests/hw_gate_coder_idle_test.sh:coder_busy_from_jsonandwait_coder_idlefrom the real script, so it cannot drift, and fails loudly if the extraction stops matching rather than passing on stale text;CODER_URL=http://127.0.0.1:9(a discard port, so a refused connection must return within 30 s having warned, not sleep throughCODER_WAIT);-1to0fails 3 arms.copyright --check,ci_targets_check --checkandtests_referenced_check --checkall rc 0.Second instance of the same class:
window11passes having compared nothingscripts/npu-window11.shis the #799 identical-text gate, and it compares the sha256 of each prompt's transcript. When a response cannot be parsed the writer records the empty string, and the empty string has the same sha in every run — so two failed runs compare "3/3 identical", and with a matching prof row the gate printsPASS. Measured on two logs of three empty rows:The script's own comment says two empty strings "would compare identical without having tested anything" — the comment stated the intent and nothing enforced it.
load()now keeps thecharsfield its line regex already captured, and a row withchars=0fails by name:chars=0is exact, not heuristic: the writer joinscontent,reasoning_content,tool_callsandfinish_reasonwith separators, so a real answer is never 0 characters — only an unparseable one is.And the script's own
--self-testearned its place: plumbingcharsthroughload()changed its return arity, and the self-test caught the two other call sites inspread()(ValueError: too many values to unpack) in the same minute I made the change. Fixed, and--self-testnow passes, including the control-pair tolerance arm.Both hunks are one class: a gate whose failure path answers "fine". Two commits, two instances, one reason.