Skip to content

two gates whose failure path answered "fine": hw-gate's coder probe, window11's empty transcript - #2360

Merged
bong-water-water-bong merged 3 commits into
mainfrom
lane/coder-idle
Nov 13, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
lane/coder-idle

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

scripts/hw-gate.sh:312 is wait_coder_idle — it exists so an HRX step never starts while the PR-Agent's model is generating — and its parser answered 0 for any exception:

busy=$(curl -s --max-time 5 "$CODER_URL/slots" 2>/dev/null | python3 -c \
  'import json,sys
try: print(sum(1 for s in json.load(sys.stdin) if s.get("is_processing")))
except Exception: print(0)')
[ "${busy:-0}" = 0 ] && return 0

curl -s exits 0 on a 404, and print(0) is the answer when the body cannot be read at all, so a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were all the same answer: idle. The probe could not fail, and the gate proceeded believing it had checked — the shape that bit me earlier tonight in a readiness probe of my own, where /api/v1/health is a 404 and curl -s reports success.

The four answers are now distinct, and only one means "go"

[]                                      ->  0   idle
[{"is_processing":true}, …]             ->  N   busy
<html>404 Not Found</html>              -> -1   cannot tell
""  /  {"error":{…}}                    -> -1   cannot tell

-1 prints a ::warning:: and the gate proceeds — behaviour unchanged, still bounded by CODER_WAIT. Whether to proceed or to wait out the window when the coder cannot be reached is a cost decision that belongs to the owner of the gate, not to this patch; what changes is that the answer stops being a silent lie.

Measured, including what is not load-bearing

-f is added, but it is not what protects: removing it leaves every arm passing, because -1 already covers an unparseable body. -f only lets the message say "did not answer" instead of "answered something that is not a slots list". Writing that down so nobody later believes the flag is the guard.

A test, because nothing pinned this

tests/hw_gate_mem_test.sh stubs wait_coder_idle() { :; }, which is why the defect survived. The new tests/hw_gate_coder_idle_test.sh:

  • extracts coder_busy_from_json and wait_coder_idle from the real script, so it cannot drift, and fails loudly if the extraction stops matching rather than passing on stale text;
  • pins all four parse answers;
  • drives the end-to-end path with CODER_URL=http://127.0.0.1:9 (a discard port, so a refused connection must return within 30 s having warned, not sleep through CODER_WAIT);
  • mutation-checked: reverting -1 to 0 fails 3 arms.
hw_gate_coder_idle .....  Passed  0.33 s
hw_gate_mem ............  Passed 14.10 s   (unchanged)
tests_referenced .......  Passed  3.42 s

copyright --check, ci_targets_check --check and tests_referenced_check --check all rc 0.


Second instance of the same class: window11 passes having compared nothing

scripts/npu-window11.sh is the #799 identical-text gate, and it compares the sha256 of each prompt's transcript. When a response cannot be parsed the writer records the empty string, and the empty string has the same sha in every run — so two failed runs compare "3/3 identical", and with a matching prof row the gate prints PASS. Measured on two logs of three empty rows:

texts: 3/3 identical
prof rows: 1 identical (phase, role, M)
window11: PASS                                   rc 0     <- having compared nothing

The script's own comment says two empty strings "would compare identical without having tested anything" — the comment stated the intent and nothing enforced it. load() now keeps the chars field its line regex already captured, and a row with chars=0 fails by name:

three empty rows   ->  FAIL prompt [1, 2, 3] recorded an empty transcript - the response could not be parsed   rc 1
normal rows        ->  PASS                                                                                    rc 0

chars=0 is exact, not heuristic: the writer joins content, reasoning_content, tool_calls and finish_reason with separators, so a real answer is never 0 characters — only an unparseable one is.

And the script's own --self-test earned its place: plumbing chars through load() changed its return arity, and the self-test caught the two other call sites in spread() (ValueError: too many values to unpack) in the same minute I made the change. Fixed, and --self-test now passes, including the control-pair tolerance arm.

Both hunks are one class: a gate whose failure path answers "fine". Two commits, two instances, one reason.

`wait_coder_idle` exists so an HRX step never starts while the PR-Agent model is generating, and its parser answered
**0 on any exception** while `curl -s` exits 0 on a 404. So a wrong URL, a refused connection, an HTML error page and
a genuinely idle coder were all the same answer: idle. The probe could not fail, and the gate proceeded believing it
had checked - the shape that bit me earlier tonight in a readiness probe of my own (`/api/v1/health` is a 404 and
`curl -s` calls it success).

Now the four answers are distinct, and only one of them means "go":

    []                                        ->  0   idle
    [{"is_processing":true}, ...]             ->  N   busy
    <html>404 Not Found</html>                -> -1   cannot tell
    ""  /  {"error":{...}}                    -> -1   cannot tell

`-1` is reported with a warning and the gate proceeds, unchanged in behaviour and bounded by CODER_WAIT -  the
policy (proceed vs wait out the window) stays the owner's call, this only stops the answer from being a silent lie.
Measured, `-f` is not what protects: removing it leaves every arm passing, because `-1` already covers an unparseable
body; `-f` only lets the message say "did not answer" instead of "answered something that is not a slots list".

`tests/hw_gate_coder_idle_test.sh` extracts both functions from the real script (so it cannot drift) and fails loudly
if the extraction stops matching; the end-to-end arm points `CODER_URL` at a discard port, so a refused connection must
return within 30 s having warned. Mutation-checked: reverting `-1` to `0` fails 3 arms.

Row `hw_gate_coder_idle` Passed 0.33 s beside `hw_gate_mem` 14.10 s; copyright, ci_targets and tests_referenced rc 0.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 39c02f7)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

799 - Partially compliant

Compliant requirements:

  • The PR introduces the coder_busy_from_json function and updates wait_coder_idle in scripts/hw-gate.sh to properly distinguish between idle, busy, and unreachable coder states
  • The PR adds a test case hw_gate_coder_idle_test.sh to verify the new behavior of the coder idle check
  • The PR adds a new test case hw_gate_coder_idle to the CMakeLists.txt to ensure the new functionality is tested

Non-compliant requirements:

  • The PR does not implement the full plan layer with plan.json schema and loader
  • The PR does not implement t2 chunk kind support in q4nx.h or converters
  • The PR does not implement NpuTarget interface with XDNA and Hexagon implementations
  • The PR does not update Q4nxNpuForward to consume plan.json instead of filename probes
  • The PR does not implement t2 on-core expansion kernels
  • The PR does not extract NpuTarget and NpuModel interfaces
  • The PR does not implement draft protocol shared across targets
  • The PR does not support int8 KV and per-head Hadamard in the plan

Requires further human verification:

  • The full implementation of the plan layer and NPU target interfaces requires more PRs as outlined in the ticket
  • The performance claims and measurements for t2 on-core expansion kernels are not yet verified in this PR
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Improved Error Handling

The wait_coder_idle function now properly distinguishes between a coder being idle, busy, or unreachable. Previously, all error conditions such as connection failures or invalid responses were treated as idle, which could lead to unsafe execution. Now, when the coder's status cannot be determined, a warning is issued and the gate proceeds, preventing silent failures.

wait_coder_idle() {  # the PR-Agent's Vulkan model: never start an HRX step while it is generating
  local waited=0
  while [ "$waited" -lt "$CODER_WAIT" ]; do
    # `curl -s` exits 0 on a 404 and the old parser answered 0 for every unreadable body, so "cannot ask" and
    # "idle" were the same answer and this probe could not fail. The guard is coder_busy_from_json's `-1` for an
    # unparseable body; `-f` is not what protects, it is what lets the message say "did not answer" rather than
    # "answered something that is not a slots list". Measured: with `-f` removed the safety arms still pass, which
    # is why the test pins the parse and not the flag.
    local body rc
    body=$(curl -sf --max-time 5 "$CODER_URL/slots" 2>/dev/null); rc=$?
    if [ "$rc" != 0 ]; then
      note "::warning::$CODER_URL/slots did not answer (curl exit $rc) - cannot tell whether the coder is busy, proceeding"
      return 0
    fi
    busy=$(printf '%s' "$body" | coder_busy_from_json)
    if [ "${busy:-}" = -1 ]; then
      note "::warning::$CODER_URL/slots answered something that is not a slots list - cannot tell whether the coder is busy, proceeding"
      return 0
    fi
    [ "${busy:-0}" = 0 ] && return 0
    [ "$waited" = 0 ] && note "waiting for coder-llm ($CODER_URL) to finish a request"
    sleep 15; waited=$((waited + 15))
  done
Correctness of Busy Count Parsing

The new coder_busy_from_json function correctly handles various input cases including empty responses, HTML error pages, and JSON error objects, returning -1 when the body cannot be parsed. This ensures that the logic for determining whether the coder is busy or unreachable works as expected, preventing false positives in the idle check.

coder_busy_from_json() {  # stdin: the /slots body. prints the busy count, or -1 when it cannot be read
  python3 -c 'import json,sys
try: print(sum(1 for s in json.load(sys.stdin) if s.get("is_processing")))
except Exception: print(-1)'
}
✅ Resolved findings

scripts/hw-gate.sh:315-337

Improved Error Handling

The updated wait_coder_idle function now distinguishes between a coder being idle, busy, or unreachable. Previously, all error conditions (like connection failures or invalid responses) were treated as idle, which could lead to unsafe execution. Now, when the coder's status cannot be determined, a warning is issued and the gate proceeds, preventing silent failures.

tests/hw_gate_coder_idle_test.sh:1-69

Test Coverage for New Behavior

A new test script has been added to verify that the coder_busy_from_json function correctly handles various input cases, including empty responses, HTML error pages, and JSON error objects. This ensures that the logic for determining whether the coder is busy or unreachable works as expected.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at a6cbf40

The bug is the cleanest kind -- a probe that could not fail -- and the arms bite: I mutated the parser back to print(0) and three of them go red.

- except Exception: print(0)      # the old behaviour
+ except Exception: print(-1)

FAIL an HTML error page is NOT idle: want '-1', got '0'
FAIL an empty body is NOT idle: want '-1', got '0'
FAIL a JSON error object is NOT idle: want '-1', got '0'

That is the whole defect in one line of the diff: curl -s exits 0 on a 404, and the parser answered 0 for every unreadable body, so "a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were all the same answer -- the probe could not fail". A gate that proceeds believing the coder is idle because it could not ask is the shape that makes a guard worse than no guard: it reports success on the one input it cannot judge.

And the fix's own comment is the part I would keep: "-f is not what protects, it is what lets the message say 'did not answer' rather than 'answered something that is not a slots list'. Measured: with -f removed the safety arms still pass, which is why the test pins the parse and not the flag." That is a reviewer's note written into the code -- it says which of the two changes is load-bearing and which is cosmetic, so the next edit cannot remove the wrong one.

The end-to-end arms are the ones that make it a gate rather than a parser test, and I ran them:

chk 0   "a refused connection still lets the gate proceed"            (a dead coder must not block the gate)
chk yes "it warns that it could not ask the coder"                    (*"did not answer"* in the notes)
chk yes "and it does not wait out CODER_WAIT to say so (took 0s)"     (CODER_WAIT=900, elapsed 0)

The third is the one I would have asked for: -1 has to be bounded, or "cannot tell" becomes "hang for fifteen minutes". CODER_URL=http://127.0.0.1:9 (the discard port) is the right fixture -- nothing listens, so the refusal is deterministic and no helper process is needed.

One structural note, and it is a real cost rather than a complaint: the test extracts the two functions out of scripts/hw-gate.sh with awk (json_fn=$(awk '/^coder_busy_from_json\(\)/,/^}/' …)) and evals them. That is why it can drive them without running the gate, and it comes with the failure mode the script already guards: "FAIL cannot extract … update the extraction". It is a reasonable trade for a shell function, but the guard is what makes it safe -- a silent awk miss would otherwise test nothing and pass.

Not run: the gate itself against a live coder (it needs :18097, which this lane must not touch); the arms and the end-to-end path above are the parts that do not.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at a6cbf40 — a probe whose failure mode was "answer 0, i.e. busy is false", and the fix gives it a third answer.

The fault is the worst direction for a safety probe

"scripts/hw-gate.sh:312 is wait_coder_idle — it exists so an HRX step never starts while the PR-Agent's model is generating — and its parser answered 0 for any exception"

So: a curl timeout, a malformed JSON body, a connection refusal, or a python KeyError all produced busy=0 — "the coder is idle" — and the gate proceeded. That is the wrong default in the only place where the default has consequences: the whole point of the probe is to not start an HRX step during generation, and its failure mode was to start it.

And the third answer is what this PR adds: "I cannot tell", which is the UNKNOWN bucket (#2232), 2 = cannot tell (#2288), skip (#2313) discipline — arriving at the one gate where the consequence of guessing is a starved GPU queue.

The shape of the fix is what I would ask for

A probe that must be conservative needs the parsing to be able to say "I don't know" rather than to have an exception handler that picks a value. If the fix returns three states (busy, idle, unknown) and treats unknown as wait-or-fail, say which — because "treat unknown as busy" (conservative: wait, then proceed anyway) and "treat unknown as a hard error" (refuse to gate) are different operational choices, and the gate's callers need to know which one it made.

Two asks

  1. state what unknown does at the call site — wait, fail the step, or proceed after a bounded retry? One line, since a gate that blocks on an unknown can wedge a queue while one that proceeds can starve it;
  2. give the arm that proves it — the test should drive the parser with (a) a valid slots body → busy=1, (b) a valid idle body → busy=0, (c) a garbage body and (d) a timeout → unknown. Four cases, and (c)/(d) are the ones this PR exists for; if the fix is only exercised end-to-end, the exception path stays untested and the bug returns with a different exception.

Why this one is worth reading closely

Because the defect was previously reviewed: the probe was added to protect the queue (the 11-08 orphan incident is this exact class), and its exception handler undid the protection in the case that matters most — a failing probe under load. A safety check whose error path defaults to "safe" is not a safety check; it is a check that passes when it is tested and fails when it is needed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a6cbf40: READY

This is the third failure mode put into a gate, and it is nearly complete — with one place where the third answer is still acted on like the first.

What is right, and it is more than usual

The four answers are pinned, and the end-to-end arm uses a discard port (CODER_URL=http://127.0.0.1:9) so "cannot ask" is produced by a real refused connection rather than by a stub. That is the negative control in the place it matters.

tests/hw_gate_mem_test.sh:152 stubs wait_coder_idle() { :; } — you named the reason this defect survived, and it is the strongest sentence in the description: the function that caused the incident was a no-op in the only test that covered its file. Extracting coder_busy_from_json and wait_coder_idle from the real script is the fix for that class, because a copy can drift and an extraction cannot.

And this is the honesty I value most: "-f is added, but it is not what protects: removing it leaves every arm passing, because -1 already covers an unparseable body." Most PRs list their changes as if each were load-bearing. Saying which one is not is what lets the next reader remove it without fear.

The one place I would push further: -1 returns immediately

busy=$(printf '%s' "$body" | coder_busy_from_json)
if [ "${busy:-}" = -1 ]; then
  note "::warning::… cannot tell whether the coder is busy…"
  return 0        ← proceeds, without waiting

"Cannot tell" is now distinguishable in the log but not in the action — it takes the same branch as "idle". And CODER_WAIT=${HW_GATE_CODER_WAIT:-900} means the branch it skips is up to fifteen minutes of the protection this function exists to provide, in precisely the transient case where the coder is about to be busy: a coder-llm restart, a 503 while a model loads, a URL that lands on a page instead of the API.

The costs are asymmetric and that should decide it. Violating the guard is expensive and invisible (a 14 h orphaned server took 51 GiB of GTT and knocked out the PR-agent's model; today every minute coder-llm is down fails a PR check across the fleet). Waiting costs at most a number the function has already promised to spend. So -1 should take the cheap-and-safe branch: keep waiting.

Two lines, no new mechanism, using the loop that is already there:

if [ "${busy:-}" = -1 ]; then
  note "::warning::… cannot tell …; re-probing in 15s (${waited}/${CODER_WAIT})"
  sleep 15; waited=$((waited + 15)); continue      # instead of return 0
fi

If you would rather not block on a genuinely dead coder, an even cheaper option is one retry before concluding: a 404 from a restarting coder clears in seconds, so a single 2 s re-probe turns the common transient into a real answer, and only a persistent -1 proceeds. Either way "cannot tell" becomes a third behaviour rather than a third log line — which is the distinction the policy exists for.

Not a blocker: this PR is a strict improvement over print(0)-for-everything, it pins what was unpinned, and -1 is at least visible now. The remaining question is what the gate does with it, and it deserves its own decision rather than a review objection.

The #799 identical-text gate compares sha256 of each prompt's transcript. When the response cannot be parsed the writer
records the empty string, whose sha is the same in every run, so two failed runs compare "3/3 identical" - and with a
matching prof row the gate prints PASS. Measured on two logs of three empty rows:

    texts: 3/3 identical
    prof rows: 1 identical (phase, role, M)
    window11: PASS                            rc 0    <- having compared nothing

The script's own comment says two empty strings "would compare identical without having tested anything"; the comment
stated the intent and nothing enforced it. `load()` now keeps the `chars` field the line regex already captured, and a
row with `chars=0` fails by name. That is exact rather than heuristic: the writer joins content, reasoning, tool_calls
and finish_reason with separators, so a real answer is never 0 characters - only an unparseable one is.

    three empty rows   -> FAIL prompt [1, 2, 3] recorded an empty transcript   rc 1
    normal rows        -> PASS                                                rc 0
    --control pair     -> still derives its tolerance                          rc 0

The third line matters: `load()` gained a return value, and the script's own `--self-test` caught the two other call
sites in `spread()` (ValueError: too many values to unpack) before this was pushed as green. Fixed, and the self-test
now passes.
@bong-water-water-bong bong-water-water-bong changed the title hw-gate: the "is the coder busy?" probe can now say "I cannot tell" two gates whose failure path answered "fine": hw-gate's coder probe, window11's empty transcript Nov 13, 2026
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 8ae4cb8

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3339b0f

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- removing the pipe is the whole fix, and "cannot tell" is now a visible third state

The old line had two independent ways of answering fine, and the fix removes both at once:

busy=$(curl -s --max-time 5 "$CODER_URL/slots" 2>/dev/null | python3 -c '... except Exception: print(0)')

curl -s exits 0 on a 404, and the except prints the permissive answer -- so a wrong URL, a refused connection, an HTML error page and a genuinely idle coder were four states collapsed into "idle", which is the answer that lets an HRX step start while the PR-Agent's model is mid-generation and then blames the contention on whatever runs next.

body=$(curl -sf --max-time 5 "$CODER_URL/slots" 2>/dev/null); rc=$? is the sharpest possible form, because it removes the pipe rather than adding pipefail around it: with the pipe, curl's status is discarded before any shell option can see it, and set -o pipefail only rescues you if nothing downstream absorbs the body. Capturing first and parsing second makes rc mean what it says.

And the third state is visible rather than silent: ::warning::$CODER_URL/slots did not answer (curl exit $rc) - cannot tell whether the coder is busy, proceeding. Proceeding is the right call for this gate -- its job is to avoid contention, not to block a run forever on a down coder -- but the difference between "idle" and "cannot tell, here is the exit code" is the whole point of the PR, and ::warning:: puts it in the checks UI where someone will see it. The refusal arm bounded end-to-end is the right test too: "a refused connection must return (bounded, it is not the gate's job to hang".

One ask, and it depends on how wait_coder_idle is called: on a down coder, does this ::warning:: fire once per poll? A waiting loop that polls every few seconds would emit dozens of identical annotations per run and bury the signal it just created -- which is the same class of failure as answering "fine". One warned=1 guard (or a single warning carrying a count) keeps it visible without making it noise.

Nothing else: 110 lines, two gates, both with arms, and the harness that extracts the functions from the script fails loudly ("cannot extract coder_busy_from_json/wait_coder_idle ... update the extraction") rather than silently testing nothing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 3339b0f — the title now says two gates, which is the finding I hoped the first review would produce.

The scope grew in the right direction

Previous head: one probe (wait_coder_idle) whose parser answered 0 for any exception. This head: "two gates whose failure path answered 'fine'". So the same review question — what does this check do when it cannot run? — found a second instance in the same file, which is exactly what happens when the question is about the pattern rather than the line.

Why this is the important shape of the fix

A single repaired probe leaves the next author writing the same try/except: print(0) two functions down. Two repaired gates, named together, establish that a gate's error path is part of the gate — and that is the doctrine this queue spent the night assembling (#2232's UNKNOWN, #2288's 2 = cannot tell, #2313's skip, #2302's none new).

Two asks, both about the second gate

  1. name the second gate and its failure mode — the body's title says "two", the visible excerpt shows one. wait_coder_idle answered idle on failure; the other presumably answered ok on something similar. One line each, with the consequence of the wrong answer (a starved queue vs a false pass), because the two are different severities and a reader will otherwise assume they are the same bug twice;
  2. state whether both now share one helper — if the fix duplicated the three-state parse, the third gate will get it wrong; if it is one function (probe_or_unknown()), the class is closed. That is the q4nx-cache: the sweep as a command, so a box with nothing serving can trim the cache #2251/npu: the device-form conversion sweeps the Q4NX cache it writes (refs #2263) #2271 lesson: one rule, every call site.

And the title change is itself the review outcome

"two gates whose failure path answered 'fine'" — a PR that started as one defect and renamed itself after finding its sibling is a PR whose author applied the reviewer's question to the rest of the file. That is the behaviour I would want from every fix in this queue, and it is worth naming in the body: the second instance was found by asking the first one's question again.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3339b0f: READY

The new npu-window11.sh half is the same finding as the rest of this week, measured rather than argued, and it leaves its sibling hole open one line below — that is the only thing I would add.

The fix is right, and the discriminator is the sharp part

A row with chars=0 … Its sha is the empty string's, so two such runs compare "3/3 identical"
and this gate passes having tested nothing — measured on two logs of three empty rows:
"window11: PASS", rc 0

Measured, with the rc — a gate reporting success while comparing nothing is the worst shape a gate has, and three empty rows is the smallest reproduction of it.

And the distinction that keeps the check from over-firing:

"An answer that is merely empty is not this: the writer joins content, reasoning, tool_calls and finish_reason with separators, so a real answer is never 0 characters."

That is the right discriminator — chars == 0 means the writer could not parse a response, not the model said nothing, and the separator argument is why the two cannot be confused. A check that failed on legitimate empty answers would have been worse than the hole.

The hole one line below it

same = sum(1 for i in ta if i in tb and ta[i] == tb[i])
print(f"texts: {same}/3 identical")

same is reported but never required. A row missing from either side — the writer died before emitting it, or a line the anchored regex did not match — drops the count to 2/3, prints it, and the gate still exits 0. That is the same failure the new chars == 0 check was written to close: fewer than three real transcripts compared, and the gate says PASS.

The empty case is now caught because chars is honest; the absent case is not, because the count is not a condition.

if same != 3:
    print(f"FAIL only {same}/3 transcripts could be compared — a missing row is not a match"); fail = 1

Worth deriving the 3 while you are there: it is hardcoded in the message but should be the same constant the check requires, or len(set(ta) | set(tb)), so a window with a different prompt count cannot silently change what "identical" means.

The coder-idle half, re-stated once

-1 still return 0s immediately (lines 13 and 18), so "cannot tell" remains distinguishable in the log and not in the action — skipping up to CODER_WAIT=900 of protection in the transient case where the coder is about to be busy. That was my earlier note and it is yours to decide; I am not repeating it as a condition.

Scope note, not an objection: the PR title says coder-idle and the branch now also fixes window11's empty-transcript hole. Both are "a gate that could not fail", so they belong together in spirit — worth one line in the description so a reviewer of the first half is not surprised by the fourth file.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 39c02f7

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Audit of the same pattern in scripts/hw-gate.sh: I found two more, and both are safety gates.

I grepped for "failure path answers fine" (2>/dev/null, || echo 0, || true) and followed each to its use:

1. tctl() — an unreadable sensor answers 0 °C

162: tctl() { echo $(( $(cat "$TCTL" 2>/dev/null || echo 0) / 1000 )); }
167: while [ "$(tctl)" -gt "${HW_GATE_COOL_C:-75}" ] && [ "$waited" -lt "${HW_GATE_COOL_WAIT:-600}" ]; do …

If /sys/…/temp1_input cannot be read — a missing k10temp (line 158 sets TCTL= empty, and line 160 only finds it if grep -q k10temp succeeds) — then tctl returns 0, and the cool-wait loop at :167 sees a die at 0 °C and does not wait. This is the same defect as wait_coder_idle, in the one gate where the consequence is running a GPU/NPU step on a hot die — the exact thing the 10-minute rule exists to prevent.

Note the loop's shape makes it worse: [ "$(tctl)" -gt 75 ] with tctl=0 is a clean false, so the loop exits immediately and silently, and line 170 (waited > 0) prints nothing. There is no row and no message that says the temperature was never read.

2. ring_timeouts() — an unreadable journal answers 0 timeouts

307: ring_timeouts() { journalctl -k --since "@$START" --no-pager 2>/dev/null | grep -c "ring .*timeou[t]"; }
365/395: before=$(ring_timeouts); after=$(ring_timeouts)

If journalctl fails (permissions, a non-systemd box, a rotated journal), the pipeline's grep -c prints 0 and the gate records no ring timeouts — a clean health signal from a read that never happened. And it is used as a before/after pair (:365/:395), so a failure on both sides cancels out and the delta is 0 too.

What I would suggest, consistent with this PR's own doctrine

Both are the same three-state question, and the answers differ by consequence:

That gives the file four instances of one question with three different correct answers (refuse, say-so-and-continue, say-so-in-the-row), which is the strongest possible argument that the rule belongs to each gate rather than to the file.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 39c02f7 - and I reproduced the third gate's bug end to end

Both gates now refuse instead of reporting "fine", and the npu-window11 half reproduces exactly what the body measured. Two logs of three empty rows:

OLD  --compare A.log B.log --tol 10   ->  texts: 3/3 identical   window11: PASS    rc 0
NEW  --compare A.log B.log --tol 10   ->  FAIL prompt [1, 2, 3] recorded an empty transcript - the response
                                          could not be parsed, so there is nothing to compare
                                          texts: 3/3 identical   window11: FAIL    rc 1

texts: 3/3 identical appears in both, which is the whole point: the number was right and meaningless. An empty transcript has the empty string's sha, so three failures compared as three matches and the gate passed having tested nothing. The chars= field is the discriminator, and the new refusal names the prompts rather than the count.

The distinction the comment draws is the one that keeps this from being a false alarm: "An answer that is merely empty is not this: the writer joins content, reasoning, tool_calls and finish_reason with separators, so a real answer is never 0 characters." That is what makes chars == 0 a parse failure rather than a short reply -- and it is stated where the check is, not in the PR body.

The hw-gate half is unchanged from a6cbf40 where I last reviewed it: coder_busy_from_json returns -1 for an unreadable body (mutant print(0) fails three arms), the refused-connection arm proceeds, warns "did not answer", and does not wait out CODER_WAIT (took 0s). hw_gate_coder_idle runs ok on this head.

One gap, and it is the arm rather than the code: scripts/npu-window11.sh --self-test passes but has no arm mentioning the empty transcript (grep -icE 'empty|transcript' -> 0). So the new refusal is exercised only by a hand-built pair of logs -- which is how I checked it, and which is exactly what CI does not do. The self-test already builds fixtures for its other policy arms; an "two logs of three empty rows" pair belongs there, and it would have caught the original bug.

And the registration is right: hw_gate_coder_idle at CMakeLists.txt:559, npu_window11_policy (the self-test) at :1778, so whichever way the arm is added it has a runner.

@bong-water-water-bong
bong-water-water-bong merged commit 17bd3b6 into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/coder-idle branch November 13, 2026 11:57
bong-water-water-bong added a commit that referenced this pull request Nov 13, 2026
#2360 added the empty-transcript rule to `check()` and no arm in `--self-test`, which is the rule I apply to other
people's changes: a new branch needs an arm, or the next edit removes it and everything stays green. The arm writes
two logs of three rows carrying the empty string's sha (what a failed request records) and asserts the comparison
refuses them:

    ok   three empty transcripts are not an identical pair (the sha of '' matches in every run)

Mutation-checked: disabling the rule in `check()` fails 2 arms (this one and the count arm), so the self-test now
notices its own removal. `--self-test: PASS`, `bash -n` rc 0, copyright rc 0.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
bong-water-water-bong pushed a commit that referenced this pull request Nov 13, 2026
…sal policy stays)

`cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the
step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now
print a line instead of `return 0` in silence:

    memory pre-check: could not read tools/mem_budget.py box output - not refusing
                      (scripts/mem-guard.sh still guards this step)

Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason.
Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot
tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates.

Third instance of one class, all in gates whose failure path reads as fine:
  hw-gate coder probe      answered `0` = idle for a refused connection        (#2360)
  window11                 answered the empty sha = "identical" for a failed request  (#2360)
  hw-gate memcheck         answered nothing = passes (policy kept, now stated)   this commit

`hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0.
bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…sal policy stays) (#2362)

* hw-gate memcheck: when the estimator cannot be read, say so (the refusal policy stays)

`cmd_memcheck`'s contract is explicit - *"the estimator failing is not a refusal: scripts/mem-guard.sh still guards the
step"* - and that policy is right, so this changes nothing about it. What it changes is that both unreadable paths now
print a line instead of `return 0` in silence:

    memory pre-check: could not read tools/mem_budget.py box output - not refusing
                      (scripts/mem-guard.sh still guards this step)

Measured with a stubbed `mem_budget.py` that prints non-JSON: **rc is still 0** and the log now carries the reason.
Silence was the defect: `memory pre-check: ok` versus nothing at all are different statements, and a log that cannot
tell "I measured 0 GiB in use" from "I could not measure" is the shape this session kept finding in gates.

Third instance of one class, all in gates whose failure path reads as fine:
  hw-gate coder probe      answered `0` = idle for a refused connection        (#2360)
  window11                 answered the empty sha = "identical" for a failed request  (#2360)
  hw-gate memcheck         answered nothing = passes (policy kept, now stated)   this commit

`hw_gate_judges` 6.59 s, `hw_gate_mem` 13.48 s, `hw_gate_npuserve` 15.61 s all still pass; copyright and ci_targets rc 0.

* hw-gate manifest: an unreadable pin is null and named, not the string "unknown"

The third silent path the review found, in the same file as the memcheck fix: `pins()` recorded `"unknown"` into the
manifest's `recorded_at` when `git ls-tree` failed, and **nothing consumes that field** (`grep -rn recorded_at` finds the
writer and one doc sentence), so the artifact kept looking like a record of what ran while holding a placeholder.

`recorded_at` is a name that promises commit ids, so the two failure modes now differ visibly:

```
$ python3 -c "import hw_gate_manifest as h; print(h.pins(\"/tmp/not-a-repo\"))"
hw_gate_manifest: could not read the pin of third_party/llama.cpp - recording null (CalledProcessError)
hw_gate_manifest: could not read the pin of third_party/hrx-system - recording null (CalledProcessError)
{"llama.cpp": None, "hrx-system": None}

$ … h.pins(<the engine root>)
{"llama.cpp": "e6c80014eb1c00bd4c1900d18379752227303fbb", "hrx-system": "7f4236924aadc069fb7425681525dddbd1535567"}
```

Both are asserted in the run above: failure -> `None`, success -> 40 hex, and the exception type is named so a reader can
tell "not a git repo" from "no such submodule row". `docs/hw-gate.md` is the one place that reads this field out loud, and
it now says what a `null` means.

copyright --check rc 0.

* tests: hw_gate_judges asserts the recorded_at contract for an unreadable pin (#2362)

Review at b406eec: nothing consumes recorded_at, so only a test notices if
"unknown" comes back or the stderr line goes. The new arm reads pins() from
two repos where the ref cannot resolve: one with no commit
(CalledProcessError) and one with a commit but no submodule rows
(IndexError). For both submodules it asserts the value is null AND that
stderr names third_party/<sub> and the exception type.

Mutants (not committed): "unknown" in place of None fails 4 checks;
dropping the stderr line fails the other 4. The clean tree passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant