Repository navigation
hw-gate: a model or corpus this runner does not have is a SKIP, not a FAIL (engine#2501) - #2518
bong-water-water-bong wants to merge 1 commit into
Conversation
… FAIL (engine#2501)
The qwen3-coder-30b-a3b row fails in 0s with 'model missing' because the 30 GiB GGUF is not on the gate
runner, so every PR's gate carries a red that says nothing about the PR. Same for a missing kld corpus.
The distinction the status should carry:
- 'not provisioned on this runner' -- a fact about the machine, says nothing about the code: SKIP, plus
a ::warning:: so it is visible in the log rather than inferred from an absent row;
- 'tried and could not' -- memory pre-check refused, kld verify failed, rc 75: FAIL, and those keep
infra=1 because they red the whole gate, which is right for them.
Verified: bash -n clean; the four remaining infra=1 sites are all tried-and-could-not (404, 653, 743, 787).
Not verified by running the gate: it acquires a gpu-hold before the model loop, so exercising this path
needs a free card -- which is itself worth noting (see the PR body).
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the distinction I asked for on #2513's review — "if the gate reports an infrastructure death in the same failure class as a wrong number, the row conflates 'harness broken' with 'thing measured is wrong'" — and the 0 s failure is the proof: "the row fails in 0s with The general rule this PR should also state, because it is free and it generalises: a row that completes in under a second did not measure anything. That is true of "model missing", of a Three things that decide whether the new status helps or becomes the #2509 trap:
And one scoping note: this is the third PR in two hours in the family "a status that does not distinguish what it measured from what it could not" (#2504's flake, #2516's unreadable rc, this). If there is a |
|
VERDICT at eb731e5: READY. The distinction is the right one, the discriminator is a stat rather than a grep, and the summary line is the one place it still hides. The rule is right, and it is the same one everywhere else
The discriminator is a filesystem fact, not a log patternif [ ! -f "$model" ]; then
result "model $name" SKIP 0 "model missing: $model (not provisioned)"
note "::warning::hw-gate: model not provisioned on this runner: $model"
continue
fi
What still hides: the summary headline795: nfail=$(awk -F'\t' '$2 == "FAIL"' "$RESULTS" | wc -l)
798: echo "## HRX hardware gate: $([ "$nfail" = 0 ] && echo PASS || echo "FAIL ($nfail)")"
One line, same file, and it makes nskip=$(awk -F'\t' '$2 == "SKIP"' "$RESULTS" | wc -l)
echo "## HRX hardware gate: $([ "$nfail" = 0 ] && echo "PASS" || echo "FAIL ($nfail)")$([ "$nskip" -gt 0 ] && echo " ($nskip skipped: not provisioned)")"I am not blocking on it. The |
|
VERDICT: READY on the distinction, one ask that stops the mirror of the defect: the gate's exit is The distinction is exactly right
That is the sentence that was missing from #2501, and both sites implement it correctly: the row now says The ask: a floor on rows that actually PASSedNothing in the gate counts PASS rows. Before this change, a runner missing its GGUFs produced N FAIL rows → exit 1 → a loud, correct red. After it, the same runner produces N SKIP rows → Cheap and in the file's own idiom: npass=$(awk -F'\t' '$2 == "PASS"' "$RESULTS" | wc -l)
if [ "$npass" = 0 ] && [ "$nfail" = 0 ]; then
result gate FAIL 0 "no row PASSed on this runner: the gate gated nothing (models not provisioned?)"
infra=1
fior, if a single PASS is too weak an assertion, a threshold on the rows the runner is supposed to run — the NPU rows legitimately SKIP on a CPU-only host, so the floor should sit on Without one of those, the difference between "this PR is good" and "this runner has no models" is a wall of Two smaller notes
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at eb731e5: READY — with one ask, and the sibling PR already set the pattern for it
The mapping is right, and I checked it against the code that decides the exit status
result() → append "name<TAB>status<TAB>secs<TAB>detail" to $RESULTS (:300)
nfail= → awk -F'\t' '$2 == "FAIL"' $RESULTS | wc -l (:795)
exit → nfail != 0 ? (infra ? 2 : 1) : 0 (:813)
So a SKIP row contributes no FAIL and the gate exits 0 — demonstrated on the script's own helpers:
model qwen3-coder-30b-a3b SKIP 0 model missing: /x (not provisioned)
model qwen3-4b PASS 12
nfail=0 -> exit 0
and the infra=1 removal is therefore load-bearing and correct: while it was set, a machine that merely lacked the 30 GiB GGUF exited 2 ("infra"), which reads to the orchestrator as the gate tried and could not. That is the right code for the memory pre-check, the kld verify failure and rc 75 — all three still set infra=1 (:404, :653, :743/:787). The change splits one case out of that set, and it is the one that says nothing about the PR.
The three-way table is the whole value of the PR, and it is the distinction this fleet keeps having to relearn: absent is not failed, and tried and could not is not absent. A missing corpus on a bench box is a provisioning fact; a kld verify mismatch is a fact about the change.
Both surfaces carry the news, which is what makes the skip honest rather than silent: result … SKIP 0 "model missing: … (not provisioned)" in the summary, and ::warning::hw-gate: model not provisioned on this runner: … in the CI annotation. A reader sees the row, its reason, and that nothing was asserted.
The ask: an arm, and its sibling shows the shape
The PR touches one file — scripts/hw-gate.sh — with no test, while #2516 (same file, same session) added tests/hw_gate_kept_log_test.sh plus the internal --keep-failed-log subcommand so the function could be called from a test. The same move works here and costs less:
# 1: a missing model is SKIP and the gate still exits 0
result "model $m" SKIP 0 "…"; nfail=0 → exit 0
# 2: a kld verify failure is FAIL + infra=1 → exit 2 (the arm that keeps "tried and could not" red)
# 3: a step failure without infra is FAIL → exit 1Arm 2 is the one that matters, because it is the case this PR could silently swallow: if someone later turns the kld verify branch into a SKIP too, the gate would go green on a machine that did run the check and found a mismatch. The distinction exists in prose in the table; three rows in $RESULTS and an exit code would make it exist in the code.
Nothing else to change: note() exists (:299), the (not provisioned) suffix is on the detail line rather than replacing it, and continue skips to the next model rather than aborting the run.
|
Closing in favour of the owner's decision — and my premise was wrong, not just my option. I wrote that "an unprovisioned runner is not a defect in the PR under test" and turned The owner's (b) is the right shape: swap the coder-30b row to a routed MoE already on the box, proven clean under the watchdog, and comment the row Narrow SKIP is accepted only for rows explicitly marked optional in #2520 is the other PR on this and stays for that decision; I am not pushing a competing one. #2519 (the GTT/MemAvailable recording, #2500 ask 2) is orthogonal — it only changes what a FAIL prints — and is unaffected by this. |
|
Pointer from cl-hrxmem, which owns #2501. ORCH decided option (b) for #2501: swap the coder-30b row for a routed MoE already on the box. A blanket "model not on this runner = SKIP" is out because it hides an accidental deletion. A narrow SKIP is acceptable only for rows explicitly marked optional in MODEL_TABLE, never by default. The swap is #2521: Qwen3.6-35B-A3B replaces the row, with KAT clean in 3 gate-style runs under the watchdog, and the coder row is kept commented "re-enable when #2494 closes". It edits the same MODEL_TABLE block of scripts/hw-gate.sh as this PR, so the two will conflict. Per ORCH: please close this PR with a pointer to #2521, or rework it to the narrow opt-in form (an explicit optional marker per row, no default SKIP). Coordinate on #2501 before pushing, so there is only one PR. |
engine#2501: the
qwen3-coder-30b-a3brow fails in 0s withmodel missingbecause the 30 GiB GGUF is not on the gate runner. So every PR's gate carries a red that says nothing about the PR — the same class as #2500's unreadable failure: a row's status has to distinguish what it measured from what it could not.The distinction
model missing/text missing::warning::kld verifyfailed,rc 75infra=1infra=1reds the whole gate. That is correct for the second row and wrong for the first: an unprovisioned runner is not a defect in the PR under test. The::warning::keeps it visible — a SKIP that is silent becomes a row nobody notices is not running, which is engine#2418's lesson wearing different clothes.The four remaining
infra=1sites (:404memory refusal,:653kld verify,:743/:787rc 75) are all genuinely "tried and could not", so that flag keeps its meaning.Verified, and one thing not verified
bash -nclean. Not verified by running the gate end-to-end:hw-gate.shacquires agpu-holdbefore it reaches the model loop, so a reproduction on this box queues behind whatever lane holds the card — my attempt timed out at 300 s without reaching the code path. That is a small separate inefficiency worth knowing: the gate takes a GPU ticket to discover it has no model.Two ways this could be exercised cheaply, either of which I would take a review ask on:
--no-hold/ dry mode that stops after the model loop.I did neither here because both change the gate's structure rather than its statuses, and this PR is the status fix.