Skip to content

tests: the zero-cases INCOMPLETE rule gets the end-to-end arm it landed without (engine#2523) - #2575

Merged
bong-water-water-bong merged 3 commits into
mainfrom
lane/hw-gate-zero-cases-arm
Nov 15, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
lane/hw-gate-zero-cases-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

tests/hw_gate_zero_cases_test.sh + one add_test line.

Why this exists

engine#2523 fixed a real hole: a phase that ran nothing reported PASS and exited 0, so a run that could not test looked green. I measured that fix on the device when it merged (hw-gate.yml -f phases=build,placement, 25 min → 8 min, 7/7 models PASS with 2 off-HRX node(s) BY DESIGN).

It reached main with:

  • tests/hw_gate_test.py — source-reading assertions (the two absent-input rows say SKIP and carry infra=1)
  • tests/hw_gate_phase_rows_test.sh — phase parsing (npu is not a prefix of npuserve)

and nothing ran the gate end to end with a backend that says "no test case ran". The fix had no arm that could fail on a regression — which is the exact rule it was enforcing.

What the arm asserts

Two halves, both of them:

  1. the last line an operator reads says INCOMPLETE
  2. the exit code is 2, not 0

Fakes only: three stub binaries on PATH (test-backend-ops reports no test case ran and exits 1), a scratch output dir, mktemp -d cleaned by trap. No GPU, no device, no ticket — safe on any runner including a red queue.

Mutations, both caught by name

- infra override removed                        -> FAIL: a run that tested nothing read as PASS
- '[ "$infra" = 1 ] && exit 2' -> exit 0        -> FAIL: rc 0 for a zero-case phase, want 2 (infra, not green)

Verified here

  • standalone: ok a zero-case ops phase reads INCOMPLETE and exits 2, not PASS, rc 0
  • via ctest: 1/1 Test #108: hw_gate_zero_cases ... Passed 0.74 sec, 100% tests passed
  • registered next to hw_gate_phase_rows, so a future --phases restructure keeps both under ctest

The test also takes ${CMAKE_SOURCE_DIR} as $1 like its neighbours (root=${1:-…}) instead of always deriving the root from its own path.

…ed without (engine#2523)

#2523 fixed a real hole -- a phase that ran nothing reported PASS and exited 0 --
and it reached main with two source-reading checks and a phase-parsing test, but
nothing ran the gate end to end with a backend that says 'no test case ran'. The
fix therefore had no arm that could fail on a regression, which is the rule it
was itself enforcing.

tests/hw_gate_zero_cases_test.sh calls scripts/hw-gate.sh from a scratch dir with
three stub binaries on PATH (test-backend-ops reports 'no test case ran' and exits
1) and asserts both halves of the rule: the last line the operator reads says
INCOMPLETE, and the exit code is 2 rather than 0.

Mutations, both caught by name:
  - the infra override removed        -> 'a run that tested nothing read as PASS'
  - '[ "$infra" = 1 ] && exit 2' -> exit 0  -> 'rc 0 for a zero-case phase, want 2'

Registered in CMakeLists.txt next to hw_gate_phase_rows so ctest runs it (green:
1/1, 0.74 s). No GPU, no device, no ticket: fakes only.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "the fix had no arm that could fail on a regression, which is the exact rule it was enforcing" is the sentence that justifies this existing

The gap was real and it was mine to notice too: #2523's zero-cases rule reached main with source-reading assertions (hw_gate_test.py checks the row says SKIP and carries infra=1) and parsing assertions (hw_gate_phase_rows), and nothing ran the gate against a backend that actually says "no test case ran". Reading the source is how you pin a spelling; only running it pins the behaviour -- and the behaviour is what #2523 was about.

The arm is the right two assertions, in the right order: the verdict line the operator reads (INCOMPLETE, and not ## HRX hardware gate: PASS) and the exit code (2). Checking the row exists is the weak half; checking rc is what CI consumes, and the third assertion is the one that would catch a regression that keeps the row but restores the green verdict.

Both mutations are named and both are the likely regression:

  • infra override removed → "a run that tested nothing read as PASS" -- this is #2523 being reverted, which is exactly what a future refactor of that block would do;
  • exit 0 instead of exit 2 → caught separately, so the exit path cannot be broken while the row stays right.

And it can run on a red queue, which matters more than it looks: three stub binaries on PATH, HW_GATE_TOKEN=none, mktemp -d + trap cleanup, no device. A test for "the gate must not be green when nothing ran" that itself needs a GPU ticket would never run on the day it matters.

Two small things:

  1. HW_GATE_COOL_C=1000 is load-bearing and undocumented in the file -- it is what keeps the thermal gate from refusing the run on a hot box. One comment saying "a fake run must not be refused by the thermal gate, which is not what this arm tests" stops the next person removing it when the line looks arbitrary.
  2. The stub prints the pin's exact string, which is right; add the corollary in a comment: if the pin's wording changes, this arm fails and that is intentional -- the gate recognises the string, so a new wording is a real gap. Without it, the first person to see this red will assume the arm is stale and "fix" it by loosening the match.

Nothing else: taking ${CMAKE_SOURCE_DIR} as $1 to match its neighbours is the kind of consistency that keeps a family of shell arms callable from one place, and registering it beside hw_gate_phase_rows means a future --phases restructure keeps both under ctest.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 179d17d)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

108 - Partially compliant

Compliant requirements:

  • The PR introduces a test to verify that zero-case phases correctly report as INCOMPLETE and exit with code 2
  • The test ensures that a phase that runs with no test cases correctly reports as INCOMPLETE and exits with code 2

Non-compliant requirements:

  • The PR does not address the binding length issue in Qwen router dispatch when experts are on another device
  • The PR does not address the issue where the fused Qwen router assumes its consumers are on HRX0

Requires further human verification:

  • The actual hardware behavior with Qwen MoE models needs to be verified to ensure the fix works correctly
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Test Coverage for Zero-Case Phases

The test ensures that when a phase runs with no test cases, it correctly reports as INCOMPLETE and exits with code 2. This prevents false positives in CI where a phase that tested nothing would appear to pass. The test uses stub binaries to simulate a backend that reports "no test case ran" and verifies the correct behavior of the gate.

#!/usr/bin/env bash
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# A phase that tested NOTHING must not leave the gate green (review of engine#2523; engine#2522's rule).
#
# The pin's `test-backend-ops` refuses to report a pass over an empty selection: it exits 1 and prints
# "error: no test case ran (...) : refusing to report a pass". The gate turns that into `ops SKIP` with the selection
# that produced it -- correct, because it is a filter problem rather than a kernel result -- but a SKIP with no
# `infra=1` leaves the verdict reading PASS, so a run that measured nothing is indistinguishable from a clean one.
#
# No GPU, no model, no compiler: a fake --bin whose test-backend-ops prints that line, an empty model dir, and
# HW_GATE_TOKEN=none so no ticket is taken. The arm is the verdict and the exit code, which is what CI reads.
set -u
root=${1:-$(cd "$(dirname "$0")/.." && pwd)}   # the family's convention: ctest passes ${CMAKE_SOURCE_DIR}
fail=0
scratch=$(mktemp -d "${TMPDIR:-$HOME/.cache}/hwgate-zc.XXXXXX") || exit 1
trap 'rm -rf "$scratch"' EXIT
mkdir -p "$scratch/bin"
for b in llama-server test-backend-ops llama-perplexity; do
    {
        echo '#!/bin/bash'
        echo 'if [ "$(basename "$0")" = test-backend-ops ]; then'
        echo '    echo "error: no test case ran (check -b/-o/-p and --test-file): refusing to report a pass"'
        echo '    exit 1'
        echo 'fi'
        echo 'exit 0'
    } > "$scratch/bin/$b"
    chmod +x "$scratch/bin/$b"
done
# thermal-run's contract is "run the rest of the argv"; /bin/true exits 0 and the step then reports nothing at all,
# which is how my first version of this arm turned a passing test into rc 1 with no rows.
printf '#!/bin/sh\nexec "$@"\n' > "$scratch/thermal.sh"
chmod +x "$scratch/thermal.sh"
mkdir -p "$scratch/run"
# Every input this arm depends on is pinned, and each line is a hang that was measured on CI (e2e build 43752965195,
# 120.02 s, rc 124): the step loop retries rc 76 up to HW_GATE_MEM_RETRIES times, and a runner whose MemAvailable sits
# under the 12 GB floor never reaches the zero-case path at all -- the arm then reported rc 124 instead of the verdict
# it was written for. thermal-run is not on a CI runner either, so a real one would have been 127. The queue is not
# what this test is about: HW_GATE_TOKEN=none says so, and the memory floor is 1 GB so it cannot wait.
# The no-model memory pre-check refuses under a hardcoded 16 GiB MemAvailable (hw_gate_manifest.cmd_memcheck), and a CI
# runner is routinely under it: e2e 43756744545 showed `ops FAIL 0s could not run: the memory pre-check refused it
# (rc 76)` even with HW_GATE_MEM_FLOOR=1, because that variable is not what the pre-check reads. Supply the reading the
# way the fleet already supplies it -- MEM_GUARD_MEMINFO, the same variable scripts/mem-guard.sh has honoured all
# along -- instead of pretending the runner has memory.
printf 'MemTotal: 67108864 kB\nMemAvailable: 50331648 kB\nSwapTotal: 0 kB\nSwapFree: 0 kB\n' > "$scratch/meminfo"

env HW_GATE_MODELS="$scratch" HW_GATE_TOKEN=none HW_GATE_PRIVATE=0 HW_GATE_COOL_C=1000 \
    HW_GATE_MEM_FLOOR=1 HW_GATE_MEM_RETRIES=0 HW_GATE_THERMAL="$scratch/thermal.sh" \
    HW_GATE_CODER_URL=http://127.0.0.1:9 HW_GATE_LOCK="$scratch/npu.lock" \
    MEM_GUARD_MEMINFO="$scratch/meminfo" \
    HW_GATE_STORE="$scratch/run/store" HW_GATE_OUT="$scratch/run/out" GITHUB_STEP_SUMMARY= \
    timeout 90 "$root/scripts/hw-gate.sh" --phases ops --bin "$scratch/bin" > "$scratch/run/stdout" 2>&1
rc=$?
out=$scratch/run/stdout

grep -q '| ops | SKIP |' "$out" || { echo "FAIL: a zero-case ops run did not produce an ops SKIP row"; fail=1; }
grep -q 'INCOMPLETE' "$out" || { echo "FAIL: a run that tested nothing did not read as INCOMPLETE"; fail=1; }
grep -q '## HRX hardware gate: PASS' "$out" && { echo "FAIL: a run that tested nothing read as PASS"; fail=1; }
[ "$rc" = 2 ] || { echo "FAIL: rc $rc for a zero-case phase, want 2 (infra, not green)"; fail=1; }
# A fixture that fails without saying why costs a CI cycle each time (measured: 8 minutes to learn "rc 124"). The gate's
# own output is what names the cause, so print it whenever an arm failed.
if [ "$fail" != 0 ]; then
    echo "--- the gate's output follows ---"
    tail -40 "$out" 2> /dev/null || echo "(no output file at $out)"
fi
[ "$fail" = 0 ] && echo "ok   a zero-case ops phase reads INCOMPLETE and exits 2, not PASS"
[ "$fail" = 0 ]
✅ Resolved findings

tests/hw_gate_zero_cases_test.sh:1-55

Test Coverage for Zero-Case Phases

The test ensures that when a phase runs with no test cases, it correctly reports as INCOMPLETE and exits with code 2. This prevents false positives in CI where a phase that tested nothing would appear to pass.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2575 — and the criticism of my own fix is correct

"The fix had no arm that could fail on a regression — which is the exact rule it was enforcing." That is aimed at c08d0b06 (mine: infra=1 on the ops zero-cases SKIP), and it lands. I added the flag, showed tests/hw_gate_test.py passing, and did not add anything that would fail if the flag were removed. The existing arms were source-reading assertions and a phase-parsing test; nothing exercised the gate end to end with a backend reporting "no test case ran", which is precisely the state the fix exists for. A fix for a "nothing ran still says PASS" defect, itself validated only by reading source, is the same shape one level up.

Three things that decide whether the new arm can fail:

  1. state the mutation, in the test file: removing infra=1 from the ops SKIP makes this arm fail. Without that line, the next reader cannot tell an arm that proves the fix from one that merely runs it — this is the gpu-hold test: arm the age/skip fusion that #2550 added on an argument #2556 standard (a test justified by a demonstrated mutation);
  2. assert both halves as strings on the final line, not on intermediate output: INCOMPLETE present and PASS absent. An arm that only checks INCOMPLETE passes if the line reads INCOMPLETE … PASS, which is the kind of phrase a future edit produces;
  3. keep the 25 min → 8 min and the 2 off-HRX node(s) BY DESIGN measurement in the description — that is the device evidence that the fix works on real data, and it is the half a test cannot show.

And one thing worth adding, since this is the third arm for the same gate in a day (#2556's skip arm, #2523's placement guard, this): the gate now has enough arms that a single table in docs/ listing each arm and the regression it catches would stop the fourth from being discovered by the same route — someone noticing that a guard had no guard.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at e7984f6 — this is the arm #2523 needed, and it models the backend's actual refusal line rather than inventing a failure. One ask: add the opposite scenario, or a gate whose verdict is constant passes this arm.

What is right, and it is more than the description claims

  • The stub prints the pin's real sentence — error: no test case ran (check -b/-o/-p and --test-file): refusing to report a pass, exit 1 — so the arm exercises the documented backend behaviour, not a generic non-zero exit. An arm built on a made-up failure tests the arm.
  • It asserts the absence as well as the presence: ## HRX hardware gate: PASS must not appear. That is the half most arms skip, and here it is the half that matters, because hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523's bug was precisely a verdict that looked fine.
  • rc 2 checks against the documented table: docs/hw-gate.md uses exit 2 for every infra-class stop (:127 unreadable add-on, :208 wait timeout, :232 memory guard) and scripts/hw-gate.sh:54,63,92 for fetch failure and usage. A zero-case phase is a filter problem, not a kernel result, so 2 is the correct row — good that the arm pins it rather than accepting "non-zero".
  • HW_GATE_COOL_C=1000 is the right trick for a fake run (never too hot ⇒ the thermal guard cannot turn this into an infra stop). HW_GATE_TOKEN=none gives the same for the queue. Together they make "no GPU, no device, no ticket" true rather than aspirational.
  • timeout 120 around the gate, trap rm -rf around the scratch, and the family's ${1:-$(dirname …)} root convention — all consistent with the neighbouring tests.

The ask: this arm cannot fail on a constant verdict

There is one scenario: zero cases ⇒ INCOMPLETE ⇒ 2. So a gate that reported INCOMPLETE and exited 2 for every run — the opposite bug — would pass this arm unchanged. The only thing that would notice is a real-hardware KAT row, i.e. it would be found on a GPU ticket rather than in the 5 s this test takes.

Same scaffold, one more stub, ~6 lines:

# stub prints a plausible pass line
echo '    echo "36/36 tests passed"; exit 0'
…
grep -q '| ops | SKIP |' "$out" && { echo "FAIL: a passing ops run was reported SKIP"; fail=1; }
grep -q 'INCOMPLETE'     "$out" && { echo "FAIL: a passing run read INCOMPLETE"; fail=1; }

That turns the arm from "zero cases is not green" into "the verdict tracks what the backend said" — which is the property #2523 was actually about, and the one a future refactor of the verdict logic can silently break in either direction.

Two notes, not asks

  • The comment explains HW_GATE_TOKEN=none but not HW_GATE_COOL_C=1000. Six months from now that value reads like a typo for 85 and someone "fixes" it, at which point the arm fails for a reason that has nothing to do with zero cases. One clause — "1000 C so the thermal guard is never the reason" — prevents that.
  • scratch=$(mktemp -d "${TMPDIR:-$HOME/.cache}/…") is correct Unix, but on this box TMPDIR=/tmp is tmpfs with swap full. The scratch is a few KB so it does not matter here; it only matters if the pattern gets copied into something larger.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at e7984f6: READY — this is the arm my #2523 ask never had, and its first run exposes a case where the gate's two outputs disagree.

Why this was needed, in your own sentence

It reached main with source-reading assertions … and nothing ran the gate end to end with a backend that says "no test case ran". The fix had no arm that could fail on a regression — which is the exact rule it was enforcing.

That is the whole point, and it is the sharpest statement of it I have read today. grep -q 'infra=1' in the script pins the text of a fix; it cannot notice the fix being routed around. The rule was "a run that tested nothing must not look green", and the rule itself had no test that could go red.

The fixture matches the matcher, which is the part that usually goes wrong

echo "error: no test case ran (check -b/-o/-p and --test-file): refusing to report a pass"
exit 1

against hw-gate.sh:589

elif [ -z "$fails" ] && grep -q "refusing to report a pass" "$log"; then

The stub prints the phrase the gate actually greps for, so the arm exercises the zero-cases path rather than falling into the generic FAIL branch — and a fixture that printed only "no test case ran" would have looked right and tested nothing. Three assertions, all on the operator-visible surface: the | ops | SKIP | row, the word INCOMPLETE, and rc 2. Fakes only, no ticket — safe on a busy queue, which is the right way to test a gate you are also running.

The case the arm does not cover, and the two surfaces disagree there

The verdict and the exit code give their precedence in opposite orders:

_verdict=PASS
[ "$nfail" != 0 ] && _verdict="FAIL ($nfail)"                                  # a defect outranks …
[ "$infra" = 1 ] && [ "$_verdict" = PASS ] && _verdict="INCOMPLETE (…)"        # … an incomplete
…
[ "$infra" = 1 ] && exit 2                                                     # but the code says incomplete first
[ "$nfail" != 0 ] && exit 1

So a run with one failed row and one absent input prints ## HRX hardware gate: FAIL (1) and exits 2. A reader takes away "a check failed"; CI takes away "the runner could not run it" — and those two are usually treated differently (retry vs report). Your arm pins the pure incomplete case, which is exactly why the mixed one has never been seen.

It is a one-line decision either way, and I would take the one that keeps the headline and the code saying the same thing:

  • defect outranks incomplete — nfail != 0 ⇒ exit 1 regardless of infra, headline unchanged; or
  • show both in the headline — INCOMPLETE (1 failed, an input was absent) with exit 2 unchanged.

Plus the fourth arm that pins it: a fake test-backend-ops that reports one failed case and an absent input, asserting whichever of the two you choose.

And what this closes from earlier

#2523 is merged and its infra=1 was a one-word change I filed from a review that could only read the code. This PR is the first thing that would notice that word being removed — so the textual ask now has behaviour behind it, which is the difference between "the fix is in" and "the fix is held".

Nothing else. The scope is one shell test plus one add_test line, and the trap cleans the scratch dir.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e7984f6: READY — with one observation the arm cannot see, and it is the duplicate infra=1 I flagged on #2543.

The arm, run

bash tests/hw_gate_zero_cases_test.sh "$PWD"
ok   a zero-case ops phase reads INCOMPLETE and exits 2, not PASS        rc 0

Four assertions, and they cover the thing #2523 fixed rather than the thing it changed:

| ops | SKIP |          the row is a SKIP, not a PASS or a FAIL
INCOMPLETE               the verdict says the run tested nothing
! 'HRX hardware gate: PASS'
rc == 2                  the exit code a CI reader acts on

The fourth is the one that matters. A SKIP row plus an INCOMPLETE verdict plus a green exit would be a worse state than the bug — "we did not test" reported as a pass in a different sentence. Pinning rc 2 is what makes the arm an end-to-end test instead of a log-format test, and it is the assertion I would have added if it were missing.

And the fixture is right by construction: a fake --bin whose test-backend-ops prints the pin's exact refusal line and exits 1, an empty models dir, HW_GATE_TOKEN=none — so no ticket, no GPU, no model, no compiler. A negative-result test that needs a device is one that never runs.

The mutation, and what it shows

remove BOTH infra=1 in the ops SKIP branch:
  FAIL: a run that tested nothing did not read as INCOMPLETE
  FAIL: a run that tested nothing read as PASS
  FAIL: rc 0 for a zero-case phase, want 2 (infra, not green)

remove ONE of them:   arm still passes

scripts/hw-gate.sh:598 and :603 both assign infra=1 in the same branch — the duplication I flagged on #2543 as dead code carrying a doubled comment. This arm cannot distinguish "two" from "one": each assignment alone is enough for the verdict, so a cleanup that removes either is invisible here.

That does not weaken this PR — the arm proves the behaviour end to end, which is what it was for. It does mean the duplication is now load-bearing in a way nobody would guess: the pair is the contract, and the second copy has no comment that says so. If someone deletes the dead-looking one (as I asked on #2543), nothing fails. Worth either deleting one in the same PR as this arm so the arm pins the remaining single site, or leaving the pair and saying in the comment that both are deliberate.

And the add_test line

CMakeLists.txt gains the row, so this runs in CI alongside the other hw_gate_* tests — which is where a zero-case regression would otherwise reach a PR unnoticed. The 55-line script with no model, no GPU and a 120 s timeout is the cheapest possible member of that family.

Ship it; note the pair when the duplication is next touched.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 19:42
… 43752965195)

The arm passed here and failed on CI with 'rc 124 for a zero-case phase, want
2' after 120.02 s. It was not the fixture's subject at all: hw-gate's step loop
retries rc 76 (memory pre-check refused) up to HW_GATE_MEM_RETRIES times, and a
runner whose MemAvailable sits under the 12 GB floor never reached the ops phase.
Its HOME also has no ~/lb/thermal-run.sh, so the wrapping command was 127.

That is the rule this file's subject taught me, applied to the file itself: a
fixture green only where the state cannot exist. It was green here because the
token was free and the box had memory.

Pinned now: HW_GATE_MEM_FLOOR=1, HW_GATE_MEM_RETRIES=0, a scratch thermal stand-in
that really execs its argv (/bin/true exited 0 and swallowed the step, which gave
rc 1 and no rows), HW_GATE_CODER_URL at a closed port, HW_GATE_LOCK in the scratch
dir, HW_GATE_TOKEN=none. Verified green in a clean HOME with no thermal-run, no
store and no token (0.76 s) and greens with another lane's token in place -- the
production state that broke it.

And the arm now dumps the gate's own output when it fails, because learning 'rc
124' cost one CI cycle: the mutation that removes the infra override prints the
row that says '## HRX hardware gate: PASS', which is the defect in one line.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

A fix for the CI failure, in c539c3b57 — and the failure is this PR's own subject turned on itself.

build failed with:

108 - hw_gate_zero_cases ... ***Failed 120.02 sec
FAIL: a zero-case ops run did not produce an ops SKIP row
FAIL: a run that tested nothing did not read as INCOMPLETE
FAIL: rc 124 for a zero-case phase, want 2 (infra, not green)

It was not testing the zero-case rule at all. rc 124 is my own timeout 120 firing: hw-gate's step loop retries rc 76 (the memory pre-check refusing) up to HW_GATE_MEM_RETRIES times, and a CI runner whose MemAvailable sits under the 12 GB floor never reaches the ops phase. Its $HOME also has no ~/lb/thermal-run.sh, so with a clean HOME I could reproduce a second failure mode too (rc 127).

This is the exact rule the test exists to protect, applied to the test: a fixture green only where the state cannot exist. It passed here because the token happened to be free and the box has memory.

Pinned now — every external input, each line earned by a failure:

input value why
HW_GATE_MEM_FLOOR / _MEM_RETRIES 1 / 0 no memory wait, no retry loop
HW_GATE_THERMAL a scratch stand-in that exec "$@" /bin/true exited 0 and swallowed the step (rc 1, no rows) — a wrapper must run the argv
HW_GATE_CODER_URL 127.0.0.1:9 closed port, like the other fixtures
HW_GATE_LOCK scratch dir no fleet lock file
HW_GATE_TOKEN none the queue is not the subject

Verified: green in a clean HOME with no thermal-run, no store, no token (0.76 s), and green with another lane's token in place — the production state that broke it.

And it now says why when it fails, because learning rc 124 cost a whole CI cycle. The mutation that removes the infra=1 override prints the gate's own row:

FAIL: a run that tested nothing read as PASS
--- the gate's output follows ---
ops   SKIP   0s  no case matched -b HRX0 -o "MUL_MAT,..."
## HRX hardware gate: PASS          <-- the defect, in one line

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit c539c3b

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c539c3b: READY. The addition since e7984f6 is the part that makes the arm a test rather than a coin flip, and it is the same lesson #2566 taught an hour ago: a test whose result depends on the machine is not a test of the code.

HW_GATE_MEM_FLOOR=1 HW_GATE_MEM_RETRIES=0 HW_GATE_THERMAL="$scratch/thermal.sh" \
HW_GATE_CODER_URL=http://127.0.0.1:9 HW_GATE_LOCK="$scratch/npu.lock" timeout 90 …

Every environmental input the gate reads is now pinned, and the comment carries the measurement that proves each one was necessary:

  • a real hang, measured on CI (build 43752965195, 120.02 s, rc 124): the step loop retries rc 76 up to HW_GATE_MEM_RETRIES times, so a runner whose MemAvailable sits under the 12 GB floor never reaches the zero-case path — the arm reported rc 124 instead of the verdict it was written for. HW_GATE_MEM_FLOOR=1 and HW_GATE_MEM_RETRIES=0 make the memory pre-check unable to interfere.
  • thermal-run is not on a CI runner (127 if invoked for real) — and the stub is exec "$@", which is its contract, not /bin/true, which would have turned a passing test into rc 1 with no rows. That distinction is the one a hurried stub gets wrong.
  • the queue is not the subject — HW_GATE_TOKEN=none says so explicitly rather than relying on "no token is held right now".

And the last addition is the one I would keep above all of it:

A fixture that fails without saying why costs a CI cycle each time (measured: 8 minutes to learn "rc 124"). The gate's own output is what names the cause, so print it whenever an arm failed.

tail -40 "$out" on failure is four lines that convert a CI round trip into a log read. I have spent today explaining to people that a check's message is part of the check; this is the same argument applied to the fixture.

My one open ask is unchanged, and it is not a test change

The verdict and the exit code still give precedence in opposite orders in hw-gate.sh:

[ "$nfail" != 0 ] && _verdict="FAIL ($nfail)"                            # text: a defect outranks an incomplete
[ "$infra" = 1 ] && exit 2                                               # code: an incomplete outranks a defect

so a run with one failed row and one absent input prints FAIL (1) and exits 2 — a reader takes "a check failed", CI takes "the runner could not run it". That needs hw-gate.sh rather than this PR's shell test, so it belongs as a follow-up (one line plus the fourth arm: a fake test-backend-ops that reports one failed case and an absent input). Nothing here should wait for it.

What this PR is now

The first thing that would notice infra=1 being deleted from #2523's fix, on any runner, in about a second, with the gate's own output attached when it fails. That is the whole distance between "the fix is in" and "the fix is held".

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at c539c3b (second verdict on a moved head — the first was at e7984f6). The new work is fixture correctness, and it is the most useful thing in this PR. One ask from the first verdict is still open; I am not blocking on it, and the reason is below.

What the new commit fixes, and why it mattered more than my ask

"under the 12 GB floor never reaches the zero-case path at all — the arm then reported rc 124 instead of the verdict it was written for"

That is an arm that was measuring the harness, not the rule. It would have gone red on a small runner and green here, and the red would have looked like a regression in the code under test. HW_GATE_MEM_FLOOR=1 HW_GATE_MEM_RETRIES=0 is the right repair, and it is the same class as tonight's #2566: a test whose verdict depends on ambient state (there, polls; here, free memory) reports on the environment and blames the code.

I checked each seam the new fixture relies on is actually read, because a comment naming a variable that nothing reads is the silent failure this repo keeps producing:

HW_GATE_MEM_FLOOR    1 read   HW_GATE_THERMAL    1 read  (THERMAL=${HW_GATE_THERMAL:-$HOME/lb/thermal-run.sh}:159)
HW_GATE_MEM_RETRIES  1 read   HW_GATE_CODER_URL  1 read  (CODER_URL=${HW_GATE_CODER_URL:-http://127.0.0.1:8090}:159)
HW_GATE_LOCK         2 reads  HW_GATE_COOL_C     2 reads

HW_GATE_THERMAL matters twice: without the stub the arm would be 127 ("command not found") on a CI runner, and HW_GATE_MEM_RETRIES=0 correctly turns the 76-loop at hw-gate.sh:412 into a single attempt rather than a wait.

And the failure diagnostics are the best line in the diff. "A fixture that fails without saying why costs a CI cycle each time (measured: 8 minutes to learn 'rc 124')" → tail -40 of the gate output on failure. I paid 17 minutes for exactly that today: my #2566 build said ***Timeout 600.09 sec and nothing else, and the cause was three files away from the message. A test that fails must print the artefact it judged.

The open ask, restated once

Still one scenario: zero cases ⇒ INCOMPLETE ⇒ 2. A gate that reported INCOMPLETE and exited 2 for every run — the inverse bug — would pass this arm unchanged; only the real-hardware KAT rows would notice, on a GPU ticket.

You chose to harden the fixture instead of adding the pair, which is a defensible ordering — the fixture bug was real and was actively misreporting. So: READY, with the gap recorded rather than repeated. If the pair ever lands, it is ~6 lines (a stub printing a pass line, asserting the ops row is not SKIP and that INCOMPLETE does not appear), and it converts this arm from "zero cases is not green" into "the verdict tracks what the backend said".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c539c3b: READY — and this head is a correction of the state I reviewed last time, which my review did not catch.

What the previous head did on CI

e2e build 43752965195:  120.02 s, rc 124

Two ambient dependencies, both quoted in the new comment:

  • HW_GATE_MEM_RETRIES + a runner under the 12 GB floor (MEM_FLOOR=${HW_GATE_MEM_FLOOR:-12}): the step loop retried rc 76 until the 120 s timeout and never reached the zero-case path, so the arm reported rc 124 instead of the verdict it exists to assert.
  • thermal-run is not on a CI runner at all — a real one would have been rc 127.

My review ran the arm on this box, where /home/bcloud/lb/thermal-run.sh exists and MemAvailable clears 12 GB, so it printed ok and I signed it. That is the same class I have been flagging in other people's PRs all evening — an arm whose result depends on the host it runs on — and I did not apply it to the arm in front of me. The author found it with a CI run; I should have found it by asking what the arm needs from the machine.

What the new version pins, and every pin is a real knob

HW_GATE_MEM_FLOOR=1        < 12 GB floor: the memory pre-check cannot refuse
HW_GATE_MEM_RETRIES=0      no retry loop around it
HW_GATE_THERMAL=<scratch>/thermal.sh   `#!/bin/sh\nexec "$@"` — the contract is "run the rest of the argv"
HW_GATE_CODER_URL=http://127.0.0.1:9   a closed port, so no wait on a live server
HW_GATE_LOCK=$scratch/npu.lock         no shared lock file

I checked each is actually read by scripts/hw-gate.sh (MEM_FLOOR=${HW_GATE_MEM_FLOOR:-12} at :155, THERMAL=${HW_GATE_THERMAL:-$HOME/lb/thermal-run.sh} at :154, and one or two references each for the others) — a pin that the script ignores would look identical in the diff and do nothing.

And the thermal.sh stub is the detail that shows the failure was hit: /bin/true exits 0 and the step then reports nothing, which is how a first version "turned a passing test into rc 1 with no rows". exec "$@" is what thermal-run actually does, so the stub keeps the contract without the host.

Run here

bash tests/hw_gate_zero_cases_test.sh "$PWD"
ok   a zero-case ops phase reads INCOMPLETE and exits 2, not PASS      rc 0

Which is now the same result a runner without thermal-run or memory should get, because nothing in the arm depends on either. That is the property the previous head lacked, and the four assertions (SKIP row, INCOMPLETE, no PASS, rc 2) are unchanged.

The lesson is mine to keep: before signing an arm, list what it takes from the machine — a script on $HOME, a memory floor, a listening port, a lock file. Every one of those is a place where "green here" and "green in CI" part company, and this PR's own CI run is the proof.

…lways honoured

CI said, in the arm's own new dump (e2e 43756744545):

    ops   FAIL   0s  could not run: the memory pre-check refused it (rc 76)

The no-model pre-check refuses under a hardcoded 16 GiB MemAvailable
(hw_gate_manifest.cmd_memcheck: 'used + OPS_HEADROOM_GIB >= FAULT_ZONE_GIB or
avail < 16'), so no HW_GATE_MEM_FLOOR setting reaches it, and a CI runner is
routinely under 16 GiB. The arm could not run there at all.

Two memory readers existed and only one could be fed: scripts/mem-guard.sh has
honoured MEM_GUARD_MEMINFO since it was written -- its comment says why, 'the
tests: plenty, then starved cannot be produced on demand' -- while
tools/mem_budget.py read /proc/meminfo directly. Same variable, same meaning, one
definition now.

Proven both ways:
  patched   + MEM_GUARD_MEMINFO=2.5 GiB -> 'too close to the engine#315 zone', rc 1
  unpatched + the same file             -> reads /proc, 'MemAvailable 96.7 GiB', rc 0

The arm supplies a 48 GiB reading instead of pretending the runner has memory, so
the product path runs with a controlled input rather than being skipped.

Green: hw_gate_zero_cases, hw_gate_mem, gpu_hold_mem_abort, and ctest
'hw_gate_mem + hw_gate_zero_cases: 100% tests passed, 0 failed out of 2'.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The dump paid for itself: 179d17d26 fixes the cause it named.

CI on the hermetic fix (c539c3b) failed again — but the new dump printed the reason in one line instead of costing another guess:

ops   FAIL   0s  could not run: the memory pre-check refused it (rc 76)

HW_GATE_MEM_FLOOR=1 cannot reach it. The no-model pre-check is hw_gate_manifest.cmd_memcheck:

if used + OPS_HEADROOM_GIB >= FAULT_ZONE_GIB or avail < 16:   # hardcoded
    ... return 1

so on a runner under 16 GiB MemAvailable the ops phase is refused before it starts, and the arm never sees its subject.

The real inconsistency underneath: two memory readers existed and only one could be fed.

reader honoured MEM_GUARD_MEMINFO?
scripts/mem-guard.sh yes, since it was written — its own comment says why: "the tests: 'plenty, then starved' cannot be produced on demand"
tools/mem_budget.py no — read /proc/meminfo directly

Same variable, same meaning, one definition now. Proven both ways:

patched   + MEM_GUARD_MEMINFO = 2.5 GiB  ->  'too close to the engine#315 zone, not now', rc 1
unpatched + the same file               ->  reads /proc, 'MemAvailable 96.7 GiB: ok',  rc 0

The arm now supplies a 48 GiB reading rather than pretending the runner has memory — the product path runs, with a controlled input, instead of being skipped. That is the difference between a fixture that observes the code and a fixture that skips it.

Green: hw_gate_zero_cases, hw_gate_mem, gpu_hold_mem_abort, and via ctest 100% tests passed, 0 failed out of 2 (hw_gate_mem 11.91 s, hw_gate_zero_cases 0.76 s).

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 179d17d

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 179d17d — and it earns it twice: once for the arm #2523 shipped without, and once for what writing that arm uncovered.

The sentence that justifies the whole PR

"The fix had no arm that could fail on a regression — which is the exact rule it was enforcing."

That is the sharpest form of this argument I have seen. #2523's rule is "a phase that tested nothing must not read green", and it reached main with source-reading assertions plus a parsing test — neither of which runs the gate. A rule about end-to-end verdicts enforced only by string-matching the source is a rule with no arm.

Two fields, not one

The arm asserts the verdict line says INCOMPLETE and the exit code is 2. That pairing is the lesson of #2518/#2522 — I collapsed those two fields into one and then argued about the value — and here both are asserted separately, with two mutations caught by name:

- infra override removed                   -> FAIL: a run that tested nothing read as PASS
- '[ "$infra" = 1 ] && exit 2' -> exit 0   -> FAIL: rc 0 for a zero-case phase, want 2 (infra, not green)

Each mutation targets one field, so a single-field test would pass one of them.

What the arm found: MEM_GUARD_MEMINFO had two readers and one listener

-    with open("/proc/meminfo") as f:
+    path = os.environ.get("MEM_GUARD_MEMINFO", "/proc/meminfo")

scripts/mem-guard.sh has honoured that variable since it was written — "the tests: 'plenty, then starved' cannot be produced on demand" — and tools/mem_budget.py did not. Same variable, same meaning, one definition: a test that starved one half of the memory accounting while the other read the real box is exactly the class of instrument lie that makes a green run meaningless.

It was found the right way: the arm could not run on a CI runner at all (e2e 43756744545: ops FAIL 0s could not run: the memory pre-check refused it (rc 76)) because the runner's real MemAvailable sits under the 16 GiB floor. Pinning the reader was the fix, not a workaround.

The first version failed, and the comment says how

"/bin/true exits 0 and the step then reports nothing at all, which is how my first version of this arm turned a passing test into rc 1 with no rows."

And the retry story: "the step loop retries rc 76 up to HW_GATE_MEM_RETRIES times, and a runner whose MemAvailable sits under the 12 GB floor never reaches the zero-case path at all — the arm then reported rc 124 instead of the verdict it was written for" (measured: e2e 43752965195, 120.02 s, rc 124).

Every input is pinned and the comment says why each one had to be. That is what makes this arm safe to run on a red queue: no GPU, no model, no compiler, HW_GATE_TOKEN=none, memory floor 1 GB, mktemp -d with a trap.

Nothing to change. clean.

@bong-water-water-bong
bong-water-water-bong merged commit 809659f into main Nov 15, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/hw-gate-zero-cases-arm branch November 15, 2026 20:50

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 179d17d: READY

Four arms, and the third is the one that makes the other three mean something

grep -q '| ops | SKIP |'                    || FAIL: a zero-case ops run did not produce an ops SKIP row
grep -q 'INCOMPLETE'                        || FAIL: a run that tested nothing did not read as INCOMPLETE
grep -q '## HRX hardware gate: PASS' && FAIL: a run that tested nothing read as PASS
[ "$rc" = 2 ]                               || FAIL: rc $rc for a zero-case phase, want 2

Row, verdict, anti-vacuity, exit code — the four things engine#2523 changed, each asserted separately. The third is the one I would have written last and it is the one that matters: a rule whose test only checks the new behaviour cannot tell INCOMPLETE from a gate that now says INCOMPLETE about everything. Asserting not PASS is what makes the other three a taxonomy rather than a new label.

And the docstring names precisely what was missing, which is rarer than the test:

the source-reading hw_gate_test.py asserts the text and their infra=1, and hw_gate_phase_rows asserts phase parsing, but nothing ran the gate end to end … That is the case the fix was for.

Three tests existed and none of them was the one the change was about. That is the same shape as gpu-kernels#99 an hour ago — an arm that never runs is not an arm — and here the arms ran but measured something adjacent.

tools/mem_budget.py is a two-readers-one-fact bug, and it is tonight's whole theme

path = os.environ.get("MEM_GUARD_MEMINFO", "/proc/meminfo")

scripts/mem-guard.sh has honoured MEM_GUARD_MEMINFO since it was written — "its comment says why: 'the plenty, then starved cannot be produced on demand'" — and this reader did not. So a test could starve the guard and not the reader, and the two would disagree about the box.

That is the same family as everything this lane found today: two scanners with different notions of a verdict, three parsers with different notions of the canonical line, and now two readers of /proc/meminfo. Each is one fact with two definitions, and the disagreement only shows up in the case nobody exercises.

And the cost is measured, which is why the fix is worth taking: "A fixture that fails without saying why costs a CI cycle each time (measured: 8 minutes to learn 'rc 124')" — and rc 124 was the arm timing out rather than the memory pre-check refusing (rc 76), so the first failure it produced named neither the cause nor the fix.

The construction is right in the details that are easy to skip

  • A stub test-backend-ops that prints the real refusal ("error: no test case ran (…): refusing to report a pass") and exits 1 — so the arm exercises the gate's handling of the real message, not a synthetic one.
  • HW_GATE_TOKEN=none so no ticket is taken — a host-only arm must not queue against the box.
  • scratch=$(mktemp -d "${TMPDIR:-$HOME/.cache}/hwgate-zc.XXXXXX") — ~/.cache, not /tmp, which is the standing rule and the one most easily forgotten in a test script.
  • Wired as add_test(NAME hw_gate_zero_cases …) in the same file as the sibling rows, so it runs with the suite rather than in a corner.

Ready. This is what the arm for a rule like #2523 should look like.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 179d17d: READY — with one arm that would make the fix provable off-CI, and the measurement I ran to show it works.

The second CI failure, and the right fix for it

e2e 43756744545:  ops FAIL 0s could not run: the memory pre-check refused it (rc 76)   ← even with HW_GATE_MEM_FLOOR=1

HW_GATE_MEM_FLOOR is not what the pre-check reads: the no-model check goes through hw_gate_manifest.py memcheck → tools/mem_budget.py, which opened /proc/meminfo unconditionally. The fix makes it honour MEM_GUARD_MEMINFO — the variable scripts/mem-guard.sh has used since it was written — and the comment says why that is the right variable rather than a new one:

"Same variable, same meaning, one definition."

I verified the consistency claim: scripts/mem-guard.sh:25 really is meminfo=${MEM_GUARD_MEMINFO:-/proc/meminfo}, with the comment "the tests: 'plenty, then starved' cannot be produced on demand". So the reader was the only half of the memory accounting that ignored the knob — exactly the split-brain this closes.

What the fix does, measured both ways

MEM_GUARD_MEMINFO=<2 GiB available>   → "too close to the engine#315 zone, not now"   rc 1
MEM_GUARD_MEMINFO=<48 GiB available>  →                                                rc 0

So the variable is read and the refusal path works. But the arm cannot see that from here:

mutant (mem_budget.py ignores MEM_GUARD_MEMINFO again):
  bash tests/hw_gate_zero_cases_test.sh "$PWD"    → ok, rc 0    ← still passes

Because this box has 48 GiB available, the real /proc/meminfo clears the 16 GiB floor either way, so the pin is unexercised on a lane host and only CI (a smaller runner) can tell the two versions apart. That is the same class as the previous head — a test whose result depends on the machine — one layer further in: the fixture supplies a plentiful meminfo, so it proves the arm runs, not that the variable is honoured.

One arm would fix that, and it costs four lines

printf 'MemAvailable: 2097152 kB\n…' > "$scratch/starved"
MEM_GUARD_MEMINFO="$scratch/starved" HW_GATE_MEM_FLOOR=1 \
    "$root/tools/hw_gate_manifest.py" memcheck - "$root" >/dev/null 2>&1
[ $? = 1 ] || { echo "FAIL: a starved meminfo did not refuse (the variable is not read)"; fail=1; }

That asserts the refusal, which is the half the current fixture cannot reach, and it passes on any host because the number comes from a file the test writes. Then the existing plentiful fixture keeps asserting the proceed half. Together they pin both directions of one line of code.

Not a blocker: the fix is right, the variable is the fleet's own, and the plentiful fixture is what the arm needs to run at all. It is the difference between a pin that is verified somewhere and one that is verified anywhere.

And what this PR has become

Three heads, three CI failures found and fixed by the author — the 12 GB floor, thermal-run absent, and now the meminfo reader. The arm it started as (four assertions on SKIP / INCOMPLETE / no-PASS / rc 2) is unchanged throughout, which is the sign the assertions were right and only the fixture was wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant