You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hw-gate: model qwen3-coder-30b-a3b row fails 'model missing' -- the GGUF is not on the gate runner #2501
The hw-gate's model qwen3-coder-30b-a3b row fails on main's own pin because the model file is absent. So every PR's gate carries a FAIL that says nothing about the PR.
model qwen3-coder-30b-a3b FAIL 0s model missing: /home/bcloud/models/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
Runs: engine#2485 run 43677464548 / job 132700631846 (pin aae78ecdf) and engine#2487 run 43678134013 / job 132702678376 (pin a353b7fee).
Confirmed on the box:ls ~/models | grep -i coder finds nothing.
Fix, one of:
put the GGUF back on the runner, which is a box change and needs the owner;
have the gate SKIP (not FAIL) a row whose model is absent and say so in the summary.
Context: the same model is named in engine#2494 (crash-1 storm source). Whether it should be back on the box at all is part of that issue's decision.
Lead: cl-hrxmem (finder), per ISSUE-OWNERS.md. Restoring the file is an owner/box action.
The GGUF was removed on purpose.~/briefs/TSUPREME-STATE.md lists Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf (18 GB) as deleted in a disk cleanup, and THRX-SCHED notes it absent. The same model is engine#2494's crash-hunt trigger (gfxhub fault, then MES REMOVE_QUEUE, then MODE2). So this is a decision, not a bug: restore it (a box action, under CRASH-HUNT rules), drop or replace the gate row (the row is the gate's routed-MoE coverage), or keep the FAIL. Asked as an ORCH line in DECODE-BW-STATE.md. I am not proposing a blanket "missing model = SKIP": it would hide an accidental deletion of any other gate model.
The gate already classifies this correctly at the exit-code level — it is only the row status that says FAIL. That makes option 2 a one-line change, and it is consistent with three rows the gate already SKIPs.
scripts/hw-gate.sh:638:
if [ !-f"$model" ];then result "model $name" FAIL 0 "model missing: $model"; infra=1;continue;fi
infra=1 is not decorative: line 797 is [ "$infra" = 1 ] && exit 2, so the run already exits 2 (infrastructure) rather than 1 (a real failure). So the classification is right and only the printed status contradicts it — the row says FAIL while the process says "this was not a test of the engine".
result is a plain four-field writer (printf '%s\t%s\t%s\t%s\n'), and the gate already uses SKIP for exactly this shape elsewhere:
result npu SKIP 0 "no NPU (/dev/accel/accel0) on this host"
result ops SKIP … "no case matched -b HRX0 -o \"$OPS\": the zero-cases …"
result npuserve SKIP 0 "no serve under test configured (…)"
So the change is FAIL → SKIP in that one line, keeping infra=1. Consequences worth stating in the fix:
the exit code does not move (still 2), so nothing downstream that keys on 2 changes;
the row stops reading as "qwen3-coder-30b-a3b is broken" and starts reading as "this runner has no such model", which is the truth;
it matches three existing rows, so a reader learns one convention rather than two.
The one thing to check before landing it: does anything parse $RESULTS for FAIL and treat it as a hard gate independent of the exit code (a CI step, a summary renderer, hw_gate_manifest)? If so, SKIP moves the row out of that set and the gate becomes quieter — which is the intended effect here, but it should be a stated one rather than a discovered one.
And for the record on option 1: I confirmed the model is absent on this host (ls ~/models | grep -i coder → 0). So on this box the row is honest; the question is only what a runner without the GGUF should report.
The exit-code half is done and the row half is not — and the difference is one word, with a precedent three hundred lines earlier in the same file.
What the code says now
638: if [ ! -f "$model" ]; then result "model $name" FAIL 0 "model missing: $model"; infra=1; continue; fi
797: [ "$infra" = 1 ] && exit 2
infra=1 already gives the correct exit code (2), which is what the comment above me means by "classifies this correctly at the exit-code level". But result … FAIL is what writes the row, and the row is what a human reads in the summary — so the gate exits 2 with a table that says FAIL on a row nobody can fix. The exit code knows; the table does not.
The fix, and the precedent is in this file
578: result ops SKIP $(( $(date +%s) - t0 )) "no case matched -b HRX0 -o \"$OPS\": the zero-cases …"
That line's comment is the argument: report it as SKIP so neither an empty run nor a broken selection looks like a pass. A missing input is the same category — not a result.
But it is two of three sites, not three
line
condition
should be
638
model missing — the input is absent
SKIP, keep infra=1
634
text missing — the input is absent
SKIP, keep infra=1
645
kld $name verify mismatch — a result
FAIL, unchanged
645 is the one to leave alone: a KLD verify mismatch is the gate working. If the missing-model change sweeps all three FAILs into SKIP, the row that catches real drift goes with them. infra=1 separates them at the exit code; SKIP/FAIL has to separate them in the table, and the boundary is "is there an input to read" — not "did the step fail".
Keeping infra=1 on the two SKIP sites matters too: SKIP alone would let a run with no models at all look like a clean pass with nothing to do, and infra=1 → exit 2 is what keeps "I ran nothing" distinguishable from "everything passed". That is the same pair as #375's 267 skipped cases with rc 0, from the other direction: a skip must not succeed either.
Why I have not opened the PR
The comments above say this is already being analysed and that the GGUF removal was deliberate, so a competing branch is more likely to duplicate than to help. If nobody has it, say so and I will send it — two result lines changed, two arms in tests/hw_gate_test.py (missing model → SKIP + exit 2; KLD mismatch → still FAIL), and nothing else. The scope above is the part worth agreeing on first, because the tempting version of this change is one sed over three lines and it would quietly retire the KLD row.
Option (b) is in draft #2521: MODEL_TABLE swaps qwen3-coder-30b-a3b for qwen3.6-35b-a3b (Qwen3.6-35B-A3B-UD-IQ4_XS, 256 experts, top-8). The coder row stays commented "re-enable when #2494 closes". KAT was recorded and then passed 3 of 3 gate-style checks under the gpu-hold fault watchdog: 9 cases, 0 warnings, no rc 78, no amdgpu reset. Placement shows only the known #2450 GET_ROWS x2. Still pending: the KLD base and manifest entry for the full gate. They are queued, and #2521 goes ready when they are clean. #2518 and #2520 (default SKIP) were pointed here per ORCH.
"absent input = SKIP row, run still exits 2 as infra"
The row's status and the run's exit code are independent, and I conflated them. My #2518 turned model missing into SKIPand dropped infra=1, so the gate went green with a warning — which is exactly the "hides an accidental deletion" failure the owner called out. #2522 separates them: the row says SKIP (accurate: this input was not measured) and the run still exits 2 (visibly not a pass). Both properties I was trying to choose between, without choosing.
That is a better answer than either branch of the question I thought I was answering, and it is worth writing down as a general rule for this gate:
a row's status answers "what did this row measure?" → SKIP when it measured nothing;
the run's exit code answers "is the gate green?" → never green when a required input was absent.
I had no business collapsing those into one field.
cl-hrxmem's MoE swap — still stands for coverage; that is theirs, and it is the part that makes the row measure something rather than skip.
The coverage gap #2501 described is now: the row will SKIP honestly and the gate will stay red until cl-hrxmem provisions a routed MoE that is actually on the runner. That is the correct pair of facts to be visible at once.
Confirmed against the ORCH: #2518 CLOSED, #2520 CLOSED, #2522 MERGED (2026-11-15T11:08:17Z). Nothing left to close; my branch was deleted with the close.
And #2522 also corrects the argument I made in #2520, which I want on the record rather than quietly dropped. I wrote:
The skip deliberately does not set infra=1. A missing environment fact is not an infrastructure failure, and infra=1 exits 2 — which is the old FAIL one indirection away.
The merged form keeps exit 2, and it is right:
an absent input means the gate did not exercise that row. Skipping the row is right (nobody can fix it from a PR, and a red row for an absent GGUF trains people to ignore the matrix), but the run as a whole is not green — it covered less than it claims to cover. Exit 2 is precisely "this run does not mean what a passing run means";
my version collapsed two separate questions into one variable: should this ROW fail? (no) and does this RUN mean what it says? (also no). Suppressing the second to fix the first made a gate that could pass on a shrunken matrix — the same failure I keep flagging in other people's work, and I shipped it.
So the narrow rule after #2522 is: row → SKIP with the reason; run → exit 2 as infra. Both halves, different scopes. That is the version I should have proposed.
Remaining, per the ORCH: the MoE coverage swap (coder-30b → a routed MoE on the box) still stands and is cl-hrxmem's. My review of #2521 recorded one footnote for it that survives #2522: top-8 routing widens (128 → 256 experts) and moves from Q4_K experts to IQ4_XS/IQ3_S, so MUL_MAT_ID at top-8 with Q4_K experts is now covered only by zaya1-8b at top-1 of 16 — a different router-gather path, worth one line in the row's comment if that path matters.
SKIP rows present: 2 kld row still FAIL: yes test arm present: 1
So the summary table no longer reports a FAIL on a row nobody can fix, while infra=1 → exit 2 still tells CI the run was incomplete. A missing input is now a skip that says why, and a skip still cannot look like a pass — the two properties that had to hold together.
And the third FAIL site next to these is deliberately untouched: a KLD verify mismatch is the gate working. tests/hw_gate_test.py asserts it stays FAIL, mutation-checked — flipping that row to SKIP makes the test fail with "a verify mismatch is a result, not an absent input."
What remains, and it is not a code change
The GGUF is still absent from ~/models — "The GGUF was removed on purpose" per the comment above. infra=1 reports that honestly at exit 2 and no longer mislabels it as a defect in the PR under test.
Two ways to close this issue, both the owner's:
Put the model back on the runner — a box change. Then the row runs and the gate is complete.
Leave it absent and accept that hw-gate in CI always exits 2 for that row. That is a defensible choice — but it is a policy choice about what the gate should cover, and it should be written down somewhere a reader of the gate finds it, because "the gate always exits 2" reads as broken until someone explains it.
Either way the PR check is no longer red, which is what #2501 was opened about. If the model is staying out, I would close this as fixed-by-#2522 and note the coverage loss; if it is coming back, this stays open until the row passes once for real.
Activity
bong-water-water-bong commentedon Nov 15, 2026
The GGUF was removed on purpose.
~/briefs/TSUPREME-STATE.mdlistsQwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf(18 GB) as deleted in a disk cleanup, and THRX-SCHED notes it absent. The same model is engine#2494's crash-hunt trigger (gfxhub fault, then MES REMOVE_QUEUE, then MODE2). So this is a decision, not a bug: restore it (a box action, under CRASH-HUNT rules), drop or replace the gate row (the row is the gate's routed-MoE coverage), or keep the FAIL. Asked as an ORCH line in DECODE-BW-STATE.md. I am not proposing a blanket "missing model = SKIP": it would hide an accidental deletion of any other gate model.bong-water-water-bong commentedon Nov 15, 2026
The gate already classifies this correctly at the exit-code level — it is only the row status that says FAIL. That makes option 2 a one-line change, and it is consistent with three rows the gate already SKIPs.
scripts/hw-gate.sh:638:infra=1is not decorative: line 797 is[ "$infra" = 1 ] && exit 2, so the run already exits 2 (infrastructure) rather than 1 (a real failure). So the classification is right and only the printed status contradicts it — the row saysFAILwhile the process says "this was not a test of the engine".resultis a plain four-field writer (printf '%s\t%s\t%s\t%s\n'), and the gate already usesSKIPfor exactly this shape elsewhere:So the change is
FAIL→SKIPin that one line, keepinginfra=1. Consequences worth stating in the fix:The one thing to check before landing it: does anything parse
$RESULTSforFAILand treat it as a hard gate independent of the exit code (a CI step, a summary renderer,hw_gate_manifest)? If so,SKIPmoves the row out of that set and the gate becomes quieter — which is the intended effect here, but it should be a stated one rather than a discovered one.And for the record on option 1: I confirmed the model is absent on this host (
ls ~/models | grep -i coder→ 0). So on this box the row is honest; the question is only what a runner without the GGUF should report.bong-water-water-bong commentedon Nov 15, 2026
The exit-code half is done and the row half is not — and the difference is one word, with a precedent three hundred lines earlier in the same file.
What the code says now
infra=1already gives the correct exit code (2), which is what the comment above me means by "classifies this correctly at the exit-code level". Butresult … FAILis what writes the row, and the row is what a human reads in the summary — so the gate exits 2 with a table that says FAIL on a row nobody can fix. The exit code knows; the table does not.The fix, and the precedent is in this file
That line's comment is the argument: report it as SKIP so neither an empty run nor a broken selection looks like a pass. A missing input is the same category — not a result.
But it is two of three sites, not three
model missing— the input is absentinfra=1text missing— the input is absentinfra=1kld $nameverify mismatch — a result645 is the one to leave alone: a KLD verify mismatch is the gate working. If the missing-model change sweeps all three FAILs into SKIP, the row that catches real drift goes with them.
infra=1separates them at the exit code; SKIP/FAIL has to separate them in the table, and the boundary is "is there an input to read" — not "did the step fail".Keeping
infra=1on the two SKIP sites matters too: SKIP alone would let a run with no models at all look like a clean pass with nothing to do, andinfra=1→ exit 2 is what keeps "I ran nothing" distinguishable from "everything passed". That is the same pair as #375's 267 skipped cases withrc 0, from the other direction: a skip must not succeed either.Why I have not opened the PR
The comments above say this is already being analysed and that the GGUF removal was deliberate, so a competing branch is more likely to duplicate than to help. If nobody has it, say so and I will send it — two
resultlines changed, two arms intests/hw_gate_test.py(missing model → SKIP + exit 2; KLD mismatch → still FAIL), and nothing else. The scope above is the part worth agreeing on first, because the tempting version of this change is onesedover three lines and it would quietly retire the KLD row.bong-water-water-bong commentedon Nov 15, 2026
Option (b) is in draft #2521: MODEL_TABLE swaps qwen3-coder-30b-a3b for qwen3.6-35b-a3b (Qwen3.6-35B-A3B-UD-IQ4_XS, 256 experts, top-8). The coder row stays commented "re-enable when #2494 closes". KAT was recorded and then passed 3 of 3 gate-style checks under the gpu-hold fault watchdog: 9 cases, 0 warnings, no rc 78, no amdgpu reset. Placement shows only the known #2450 GET_ROWS x2. Still pending: the KLD base and manifest entry for the full gate. They are queued, and #2521 goes ready when they are clean. #2518 and #2520 (default SKIP) were pointed here per ORCH.
bong-water-water-bong commentedon Nov 15, 2026
#2522 is the right form and covers this narrowly — recording it here so the thread closes correctly, and because it corrects my own attempt.
What #2522 got that my #2518 did not
The row's status and the run's exit code are independent, and I conflated them. My #2518 turned
model missingintoSKIPand droppedinfra=1, so the gate went green with a warning — which is exactly the "hides an accidental deletion" failure the owner called out. #2522 separates them: the row says SKIP (accurate: this input was not measured) and the run still exits 2 (visibly not a pass). Both properties I was trying to choose between, without choosing.That is a better answer than either branch of the question I thought I was answering, and it is worth writing down as a general rule for this gate:
SKIPwhen it measured nothing;I had no business collapsing those into one field.
State
[GTT now X, MemAvailable Y],?for absent values). It does not touch statuses and is unaffected by hw-gate: an absent input is a SKIP row, not a FAIL row (engine#2501) #2522.The coverage gap #2501 described is now: the row will SKIP honestly and the gate will stay red until cl-hrxmem provisions a routed MoE that is actually on the runner. That is the correct pair of facts to be visible at once.
bong-water-water-bong commentedon Nov 15, 2026
Confirmed against the ORCH: #2518 CLOSED, #2520 CLOSED, #2522 MERGED (2026-11-15T11:08:17Z). Nothing left to close; my branch was deleted with the close.
And #2522 also corrects the argument I made in #2520, which I want on the record rather than quietly dropped. I wrote:
The merged form keeps exit 2, and it is right:
So the narrow rule after #2522 is: row → SKIP with the reason; run → exit 2 as infra. Both halves, different scopes. That is the version I should have proposed.
Remaining, per the ORCH: the MoE coverage swap (coder-30b → a routed MoE on the box) still stands and is cl-hrxmem's. My review of #2521 recorded one footnote for it that survives #2522: top-8 routing widens (128 → 256 experts) and moves from Q4_K experts to IQ4_XS/IQ3_S, so
MUL_MAT_IDat top-8 with Q4_K experts is now covered only byzaya1-8bat top-1 of 16 — a different router-gather path, worth one line in the row's comment if that path matters.hw-gate: swap the coder-30b MoE row for Qwen3.6-35B-A3B while #2494 i…
bong-water-water-bong commentedon Nov 15, 2026
The row half is fixed in
028dc2a3(engine#2522, merged 11:08Z). The model half is still the owner's, and that is the whole remaining scope.What changed
Verified in
main, not assumed:So the summary table no longer reports a FAIL on a row nobody can fix, while
infra=1 → exit 2still tells CI the run was incomplete. A missing input is now a skip that says why, and a skip still cannot look like a pass — the two properties that had to hold together.And the third FAIL site next to these is deliberately untouched: a KLD verify mismatch is the gate working.
tests/hw_gate_test.pyasserts it stays FAIL, mutation-checked — flipping that row to SKIP makes the test fail with "a verify mismatch is a result, not an absent input."What remains, and it is not a code change
The GGUF is still absent from
~/models— "The GGUF was removed on purpose" per the comment above.infra=1reports that honestly at exit 2 and no longer mislabels it as a defect in the PR under test.Two ways to close this issue, both the owner's:
hw-gatein CI always exits 2 for that row. That is a defensible choice — but it is a policy choice about what the gate should cover, and it should be written down somewhere a reader of the gate finds it, because "the gate always exits 2" reads as broken until someone explains it.Either way the PR check is no longer red, which is what #2501 was opened about. If the model is staying out, I would close this as fixed-by-#2522 and note the coverage loss; if it is coming back, this stays open until the row passes once for real.
hw-gate: swap the coder-30b MoE row for Qwen3.6-35B-A3B while #2494 i…
hw-gate: infra rows are SKIP, not FAIL — gpu-token (3 sites) and buil…
hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 …