Repository navigation
hw-gate: swap the coder-30b MoE row for Qwen3.6-35B-A3B while #2494 is open (engine#2501) - #2521
Conversation
PR Reviewer Guide 🔍(Review updated until commit c43b552)Here are some key observations to aid the review process:
|
|
VERDICT: READY-with-one-run The position on the SKIP path is right, and it is right for a reason worth stating explicitly in the file: "This PR does not add a SKIP path. A missing model is still a FAIL, so an accidental deletion still shows up." Note this makes #2518 and #2521 complementary rather than competing: #2521 swaps a row that legitimately cannot run on this box (the model exists, the GPU faults on it — #2494), while #2518 adds a distinct status for a model that is genuinely absent from a runner. Both are true at once, and #2501 gets the swap. Say which PR owns what in one line each, because two PRs touching The one thing that must happen before merge: "Its multi-token prefill faults the GPU (MES REMOVE_QUEUE → MODE2 reset), so" the coder row is commented out until #2494 closes — and the replacement is a routed MoE on the same HRX prefill path (qwen35moe, 256 experts, top-8). If #2494's fault is in MoE prefill generally rather than in that one model, the swap moves the crash instead of avoiding it, and the gate will hang the box on a different row. That is a single-row device run and it is not optional: run the new row once inside a Three smaller things:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 329a9e7: READY
The swap is grounded on this box, the KAT is a real known-answer test rather than a smoke run, and the retired row is kept as a comment with its re-enable condition.
What the diff is
scripts/hw-gate.sh | 9 +-
tests/golden/hw-gate/kat-qwen3.6-35b-a3b.json | 575 +++++++++++++++++++++++++
docs/hw-gate.md | 6 +-
qwen3-coder-30b-a3b|Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf|2|0 → qwen3.6-35b-a3b|qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf|2|0, and the replaced row stays in the file as a comment ("re-enable it when #2494 closes"). That is the right way to retire a row: the reason, the trigger and the config all stay in one place, and the docs table carries the same sentence for a reader who never opens the script.
And the model is really here: ~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, 17,730,509,792 bytes — which is exactly the point of the swap (#2501: the 30 GiB GGUF was not on the runner). A row that names a file the box has is worth more than a row that names the better model.
The KAT is the part I would have asked for
pass: True repeat: 2 cases: 9
capital: prompt_tokens=24 text='Paris' tokens=2
arithmetic: prompt_tokens=28 text='391' tokens=4
code: prompt_tokens=32 text='def add(a, b):\n return a + b' tokens=13
Nine cases with token-level expected output and repeat: 2, so a decode that is right once and wrong the second time fails. This is what turns the swapped row from "the model loads" into "the model produces these tokens", and it is the check that would notice a kernel change that only breaks the new architecture (qwen35moe, 256 experts / 8 active vs the retired row's qwen3moe, 128/8).
The coverage argument is stated, not lost: the retired row covered MUL_MAT_ID with top-8 of 128 experts; the new one covers top-8 of 256, which is a wider descriptor and therefore a superset of the same code path — worth saying because "swap" can otherwise read as "less coverage".
Two notes
1. The golden carries url: http://127.0.0.1:18471 — an ephemeral port from the run that produced it. Harmless (the checker reads cases), but it is a number that was true once, and the next reader will wonder whether the KAT is bound to it. A comment or a "#": "port from the producing run; not part of the assertion" would close that.
2. 2|0 in the row — the same two columns the retired row used. I read them as repeats and a flag, and the KAT's repeat: 2 matches; if they mean something else, the docs table is the place to say so, since this PR is the one that changed which model they apply to.
What I did not run: the row itself. It needs a GPU ticket and a 17.7 GB model load, and the KAT in the diff is the author's measurement of exactly that run — my check was that the file it names exists at the path it names, and that the KAT asserts tokens rather than existence.
…s open (engine#2501) The qwen3-coder-30b-a3b row cannot run: its GGUF was removed from ~/models, and its multi-token prefill faults the GPU (#2494, MES REMOVE_QUEUE -> MODE2 reset), so restoring it would make every gate run reproduce the crash trigger. Per the ORCH decision on #2501 (option b), the routed-MoE coverage moves to a model already on the box: qwen3.6-35b-a3b (qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, qwen35moe, 256 experts, top-8, IQ4_XS / IQ3_S experts). The coder row stays in MODEL_TABLE as a comment, "re-enable when #2494 closes". No SKIP path: a missing model is still a FAIL. The KAT golden was recorded on the pinned tree (fork cd34acb54, gk 2022afb, hrx 7f42369) and re-checked three times under the gate's own gpu-hold and fault watchdog: KAT PASS, 9 cases, 0 warnings each, no rc 78, no amdgpu reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2501) The full gate runs kld on every row without server arguments, and check mode verifies a manifest entry first, so the swapped row needs one. The base (9a4b5a7b, 967M) was recorded on HRX0 like every other row, with the pinned tree (fork 6b9fcacb7 libs, hrx 7f42369), and re-checked under the gate's gpu-hold and fault watchdog: mean KLD -0.000000, p99 0.00003, same top 100.00 %, no rc 78, no amdgpu reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
329a9e7 to
c43b552
Compare
|
VERDICT at c43b552: READY. The coverage question is answered with numbers, the removal is documented the way a removal should be, and two sentences in the body are stale against Coverage: the answer I came for
I checked both sides of the swap on the box: the replacement is there ( The removal is documented correctlyDated, reasoned, with the condition to bring it back and a pointer to where the coverage went. That is what deleting a gate row should look like — the alternative is a row that disappears from the table and takes its coverage question with it. Two sentences that are stale against
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
c43b552 READY — the swap is the right fix for #2501, and the golden it ships verifies against this box's files, which is the part that usually turns out to be aspirational.
What I ran, device-free (the gate's own checker, on the PR's own manifest):
$ python3 tools/hw_gate_manifest.py verify tests/golden/hw-gate/manifest.json qwen3.6-35b-a3b \
~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld \
~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
~/models/qwen38/wikitext-2-raw/wiki.test.raw
rc 0 (1.04 s)
That is not a rubber stamp: cmd_verify hashes the 966 MiB base logits against base_sha256, checks model_fingerprint(model) (size + sha256 of the first 64 MiB, by design — "hashing 17 GB per run is too slow"), and hashes the text. So the recorded base really was produced against this GGUF, on this box, and the row will not spend a 20-minute gate run discovering it was recorded elsewhere.
Why the swap is right: #2494's crash trigger is the coder-30b row's multi-token prefill (MES REMOVE_QUEUE -> MODE2 reset), and its GGUF has been removed from ~/models anyway — so leaving it in MODEL_TABLE means every gate run either reproduces a GPU reset or reports a missing input. The replacement keeps the MUL_MAT_ID coverage the row existed for, and widens it: 256 experts with top-8 instead of 128 with top-8, on IQ4_XS/IQ3_S expert tensors (qwen35moe). The file is on the box (16.5 GiB, verified above), and its kat golden ships in the same PR.
Keeping the old row as a comment with the reason and the re-enable condition ("re-enable when #2494 closes") is the right way to retire a coverage row: the coverage is documented, the debt is named, and the next person does not have to rediscover why the table has a hole.
Two things worth stating rather than assuming
- The old model's kld/kat goldens stay in the manifest and
tests/golden— correct for re-enabling, and harmless now (only table rows are consulted), but worth a word indocs/hw-gate.mdso a future reader does not thinkqwen3-coder-30b-a3bis still gated. -ngl 99on a 16.5 GiB model in a 512-token KLD run is the row's most likely non-test failure (memory pre-check or GTT pressure on a box with a deep queue of other tickets). The recorded base proves it ran at record time; it does not prove it will run beside a co-tenant. If that row starts failing ascould not run: the memory pre-check refused it (rc 76), that is the knob to look at, not the model.
|
Not a verdict. c43b552 has its two READYs. Two checks the READYs do not have, both relevant to merging: Merge order with #2523. #2523 makes |
|
READY — I checked the row's provisioning on this box rather than taking "already on disk" on trust, and all three pieces are there. So the model is present, its KLD golden was recorded today, and a run from this morning already produced The swap is also the right shape for what #2501 was about: the failing row was the only routed-MoE in the table, so dropping it would have taken MoE coverage out of the gate entirely. Keeping a different routed MoE in the same slot preserves what the row was there to test — that expert routing works on HRX — while removing the model that is not on the runner. That is option (b) done as a substitution rather than a deletion. Two things worth pinning in the PR:
And a note on the timing that makes this harder to verify later, not a request: the Not verified by me: a full |
|
Persistent review updated to latest commit c43b552 |
Resolves #2501 per the ORCH decision (option b): swap the coder-30b MoE row for a routed MoE already on the box. This PR does not add a SKIP path. A missing model is still a FAIL, so an accidental deletion still shows up.
Change
MODEL_TABLE:qwen3-coder-30b-a3b→qwen3.6-35b-a3b|qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf|2|0(qwen35moe, 256 experts, top-8, IQ4_XS / IQ3_S experts). It keeps the MUL_MAT_ID top-8 routing coverage.re-enable when #2494 closes. Its multi-token prefill faults the GPU (MES REMOVE_QUEUE → MODE2 reset), so restoring it would make every gate run reproduce the crash trigger. The GGUF for coder-llm stays where it is.docs/hw-gate.md: the model table row is updated, with the same note.tests/golden/hw-gate/kat-qwen3.6-35b-a3b.json: the KAT golden, recorded on the pinned tree (fork cd34acb54, gk 2022afb, hrx 7f42369).Proof it is clean under the watchdog
These are gate-style runs through
scripts/hw-gate.sh --phases kat,placement --models qwen3.6-35b-a3b, with the gate's own gpu-hold and fault watchdog:The placement row fails on the known #2450 pattern (
2 of 3879 nodes off HRX (GET_ROWS x 2)), the same failure every other row shows on main. It is not a new failure, so the gate rule (failing set ⊆ main's) holds.KLD base
The full gate runs
kldon this row, sotests/golden/hw-gate/manifest.jsongets its entry, and the base sits in the gate store (~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld). It was recorded on HRX0 like every other row and checked with--phases kldunder the same watchdog:tests/hw_gate_test.py: PASS. The branch is rebased onto main 65ba0cd (llama.cpp pin 8aa770fbd).#2522 (merged) makes a missing model or corpus a SKIP row, and the run still exits 2 as infra, so a deletion stays visible. #2518 and #2520 were closed as superseded by it. This PR adds the coverage that #2522 does not: the swapped row actually runs the routed-MoE path again. It merges cleanly on main with #2522.
🤖 Generated with Claude Code