Skip to content

hw-gate: swap the coder-30b MoE row for Qwen3.6-35B-A3B while #2494 is open (engine#2501) - #2521

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxmem/2501-gate-moe-row
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxmem/2501-gate-moe-row

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Resolves #2501 per the ORCH decision (option b): swap the coder-30b MoE row for a routed MoE already on the box. This PR does not add a SKIP path. A missing model is still a FAIL, so an accidental deletion still shows up.

Change

  • MODEL_TABLE: qwen3-coder-30b-a3b → qwen3.6-35b-a3b|qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf|2|0 (qwen35moe, 256 experts, top-8, IQ4_XS / IQ3_S experts). It keeps the MUL_MAT_ID top-8 routing coverage.
  • The coder row stays in the script as a comment: re-enable when #2494 closes. Its multi-token prefill faults the GPU (MES REMOVE_QUEUE → MODE2 reset), so restoring it would make every gate run reproduce the crash trigger. The GGUF for coder-llm stays where it is.
  • docs/hw-gate.md: the model table row is updated, with the same note.
  • tests/golden/hw-gate/kat-qwen3.6-35b-a3b.json: the KAT golden, recorded on the pinned tree (fork cd34acb54, gk 2022afb, hrx 7f42369).

Proof it is clean under the watchdog

These are gate-style runs through scripts/hw-gate.sh --phases kat,placement --models qwen3.6-35b-a3b, with the gate's own gpu-hold and fault watchdog:

run kat placement watchdog / reset
record 09:16Z REC, 9 cases, 0 warnings FAIL (#2450 pattern) none
check 1 10:00Z PASS, 9 cases, 0 warnings FAIL (#2450 pattern) none
check 2 10:17Z PASS, 9 cases, 0 warnings FAIL (#2450 pattern) none
check 3 10:31Z PASS, 9 cases, 0 warnings FAIL (#2450 pattern) none

The placement row fails on the known #2450 pattern (2 of 3879 nodes off HRX (GET_ROWS x 2)), the same failure every other row shows on main. It is not a new failure, so the gate rule (failing set ⊆ main's) holds.

KLD base

The full gate runs kld on this row, so tests/golden/hw-gate/manifest.json gets its entry, and the base sits in the gate store (~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld). It was recorded on HRX0 like every other row and checked with --phases kld under the same watchdog:

run kld watchdog / reset
record 11:04Z REC 9a4b5a7b, 967M none
check 11:30Z PASS: mean KLD -0.000000, p99 0.00003, same top 100.00 % none

tests/hw_gate_test.py: PASS. The branch is rebased onto main 65ba0cd (llama.cpp pin 8aa770fbd).

#2522 (merged) makes a missing model or corpus a SKIP row, and the run still exits 2 as infra, so a deletion stays visible. #2518 and #2520 were closed as superseded by it. This PR adds the coverage that #2522 does not: the swapped row actually runs the routed-MoE path again. It merges cleanly on main with #2522.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit c43b552)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2501 - Partially compliant

Compliant requirements:

Non-compliant requirements:

  • Does not implement a SKIP path for missing models (as per option b in the ticket description)

Requires further human verification:

  • Verification that the new model qwen3.6-35b-a3b is indeed present and functional on the gate runner
  • Confirmation that the KAT golden data accurately reflects the expected behavior for the new model

2494 - Partially compliant

Compliant requirements:

  • No requirements from this ticket are directly addressed by this PR

Non-compliant requirements:

Requires further human verification:

2450 - Partially compliant

Compliant requirements:

  • No requirements from this ticket are directly addressed by this PR

Non-compliant requirements:

Requires further human verification:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Model Row Swap

The PR swaps the qwen3-coder-30b-a3b model row for qwen3.6-35b-a3b in the MODEL_TABLE. While this resolves the immediate issue of the missing model file, it does not address the underlying placement issue (#2450) that affects all gate models. The swap is appropriate for the current gate run but doesn't fix the root cause of the placement failures.

# engine#2501: qwen3-coder-30b-a3b|Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf|2|0 -- re-enable when #2494 closes. Its
# multi-token prefill faults the GPU (MES REMOVE_QUEUE -> MODE2 reset), so every gate run would reproduce the crash
# trigger, and its GGUF was removed from ~/models. Its MoE coverage moved to the qwen3.6-35b-a3b row.
MODEL_TABLE="qwen3-0.6b|Qwen3-0.6B-Q4_K_M.gguf|3|1
zaya1-8b|zaya1-8b-Q4_K_M.gguf|2|1
qwen3.8-27b|qwen38/UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf|2|0
qwen3.6-35b-a3b|qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf|2|0
qwen3-4b-iq4xs|Qwen3-4B-IQ4_XS.gguf|2|0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY-with-one-run 2521

The position on the SKIP path is right, and it is right for a reason worth stating explicitly in the file: "This PR does not add a SKIP path. A missing model is still a FAIL, so an accidental deletion still shows up." Note this makes #2518 and #2521 complementary rather than competing: #2521 swaps a row that legitimately cannot run on this box (the model exists, the GPU faults on it — #2494), while #2518 adds a distinct status for a model that is genuinely absent from a runner. Both are true at once, and #2501 gets the swap. Say which PR owns what in one line each, because two PRs touching MODEL_TABLE/status semantics is exactly how #2484/#2490 collided two nights ago.

The one thing that must happen before merge: "Its multi-token prefill faults the GPU (MES REMOVE_QUEUE → MODE2 reset), so" the coder row is commented out until #2494 closes — and the replacement is a routed MoE on the same HRX prefill path (qwen35moe, 256 experts, top-8). If #2494's fault is in MoE prefill generally rather than in that one model, the swap moves the crash instead of avoiding it, and the gate will hang the box on a different row. That is a single-row device run and it is not optional: run the new row once inside a gpu-hold ticket and put the result in the description. A row swap accepted on the argument that a different model faults is a swap accepted on the least safe possible evidence.

Three smaller things:

  1. the commented-out row needs an owner and a trigger — # re-enable when #2494 closes is good practice, but a comment is invisible to any automated "which rows are disabled" query. If MODEL_TABLE supports a state field, use it; otherwise list disabled rows in docs/hw-gate.md beside the coverage table;
  2. coverage equivalence must be recorded, not asserted — the row existed for MUL_MAT_ID top-8 routing at 256 experts, and the replacement is 256 experts top-8; write that down in the doc, because the next person to trim the table needs to know what that row is for;
  3. IQ4_XS / IQ3_S experts is a different quantisation from the row it replaces — fine for placement/routing coverage, but say so, so nobody later reads gate timing from that row and compares it with the coder row's history.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 329a9e7: READY

The swap is grounded on this box, the KAT is a real known-answer test rather than a smoke run, and the retired row is kept as a comment with its re-enable condition.

What the diff is

scripts/hw-gate.sh                             |   9 +-
tests/golden/hw-gate/kat-qwen3.6-35b-a3b.json  | 575 +++++++++++++++++++++++++
docs/hw-gate.md                                |   6 +-

qwen3-coder-30b-a3b|Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf|2|0 → qwen3.6-35b-a3b|qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf|2|0, and the replaced row stays in the file as a comment ("re-enable it when #2494 closes"). That is the right way to retire a row: the reason, the trigger and the config all stay in one place, and the docs table carries the same sentence for a reader who never opens the script.

And the model is really here: ~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, 17,730,509,792 bytes — which is exactly the point of the swap (#2501: the 30 GiB GGUF was not on the runner). A row that names a file the box has is worth more than a row that names the better model.

The KAT is the part I would have asked for

pass: True   repeat: 2   cases: 9
capital:     prompt_tokens=24  text='Paris'                     tokens=2
arithmetic:  prompt_tokens=28  text='391'                       tokens=4
code:        prompt_tokens=32  text='def add(a, b):\n    return a + b'   tokens=13

Nine cases with token-level expected output and repeat: 2, so a decode that is right once and wrong the second time fails. This is what turns the swapped row from "the model loads" into "the model produces these tokens", and it is the check that would notice a kernel change that only breaks the new architecture (qwen35moe, 256 experts / 8 active vs the retired row's qwen3moe, 128/8).

The coverage argument is stated, not lost: the retired row covered MUL_MAT_ID with top-8 of 128 experts; the new one covers top-8 of 256, which is a wider descriptor and therefore a superset of the same code path — worth saying because "swap" can otherwise read as "less coverage".

Two notes

1. The golden carries url: http://127.0.0.1:18471 — an ephemeral port from the run that produced it. Harmless (the checker reads cases), but it is a number that was true once, and the next reader will wonder whether the KAT is bound to it. A comment or a "#": "port from the producing run; not part of the assertion" would close that.

2. 2|0 in the row — the same two columns the retired row used. I read them as repeats and a flag, and the KAT's repeat: 2 matches; if they mean something else, the docs table is the place to say so, since this PR is the one that changed which model they apply to.

What I did not run: the row itself. It needs a GPU ticket and a 17.7 GB model load, and the KAT in the diff is the author's measurement of exactly that run — my check was that the file it names exists at the path it names, and that the KAT asserts tokens rather than existence.

bong-water-water-bong and others added 2 commits November 15, 2026 07:30
…s open (engine#2501)

The qwen3-coder-30b-a3b row cannot run: its GGUF was removed from ~/models, and its
multi-token prefill faults the GPU (#2494, MES REMOVE_QUEUE -> MODE2 reset), so restoring
it would make every gate run reproduce the crash trigger. Per the ORCH decision on #2501
(option b), the routed-MoE coverage moves to a model already on the box:
qwen3.6-35b-a3b (qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, qwen35moe, 256 experts, top-8,
IQ4_XS / IQ3_S experts). The coder row stays in MODEL_TABLE as a comment, "re-enable when
#2494 closes". No SKIP path: a missing model is still a FAIL.

The KAT golden was recorded on the pinned tree (fork cd34acb54, gk 2022afb, hrx 7f42369)
and re-checked three times under the gate's own gpu-hold and fault watchdog: KAT PASS,
9 cases, 0 warnings each, no rc 78, no amdgpu reset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2501)

The full gate runs kld on every row without server arguments, and check mode verifies a
manifest entry first, so the swapped row needs one. The base (9a4b5a7b, 967M) was recorded
on HRX0 like every other row, with the pinned tree (fork 6b9fcacb7 libs, hrx 7f42369), and
re-checked under the gate's gpu-hold and fault watchdog: mean KLD -0.000000, p99 0.00003,
same top 100.00 %, no rc 78, no amdgpu reset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong marked this pull request as ready for review November 15, 2026 11:31
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c43b552: READY. The coverage question is answered with numbers, the removal is documented the way a removal should be, and two sentences in the body are stale against main as of this morning.

Coverage: the answer I came for

qwen3-coder-30b-a3b → qwen3.6-35b-a3b, 256 experts, top-8, IQ4_XS / IQ3_S experts — and you say it plainly: "It keeps the MUL_MAT_ID top-8 routing coverage." That is the question a row swap has to answer, because the tempting version of this PR is one that turns a red row green while silently testing less. Mixed IQ quant expert tensors also keep the IQ GET_ROWS paths in play, which the coder row was not covering as well.

I checked both sides of the swap on the box: the replacement is there (qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, 17.7 GiB) and the old one is genuinely gone (~/models/*coder-30b* matches nothing), so before this PR that row really was exit 2-ing on every run rather than failing for a reason.

The removal is documented correctly

# engine#2501: qwen3-coder-30b-a3b|… -- re-enable when #2494 closes. Its … trigger, and its GGUF was removed
#              from ~/models. Its MoE coverage moved to the qwen3.6-35b-a3b row.

Dated, reasoned, with the condition to bring it back and a pointer to where the coverage went. That is what deleting a gate row should look like — the alternative is a row that disappears from the table and takes its coverage question with it.

Two sentences that are stale against main

1. "A missing model is still a FAIL." #2522 merged this morning, and main now reads:

hw-gate.sh:652  if [ ! -f "$model" ]; then result "model $name" SKIP 0 "model missing (input absent): $model"; infra=1; continue; fi
hw-gate.sh:823  [ "$infra" = 1 ] && exit 2

So a missing model is a SKIP row plus infra=1, and the gate exits 2 — the box could not run it. The behaviour you want (an accidental deletion still shows up) is preserved, but by infra, not by FAIL, and the row's status now reads SKIP. Worth fixing the sentence, because the next person to read it will believe the row FAILs and will be surprised when a deletion produces an exit code instead of a red row.

2. The golden's pin. kat-qwen3.6-35b-a3b.json was recorded on fork cd34acb54, gk 2022afb, hrx 7f42369, while #2526 (open) pins the fork at 8aa770fbd. #397 is a DFlash-context change and should not move any number here — and if it does, this KAT is precisely the thing that catches it. Saying the recording pin in docs/hw-gate.md rather than only in the PR body makes a future mismatch attributable in one line instead of an afternoon.

And the honest note on what this does not fix

#2494 stays open and the coder row stays commented out, so the gate covers routed MoE through a different model rather than the one that faults the GPU. That is the right trade while a multi-token prefill can drive an MES REMOVE_QUEUE → MODE2 reset, and it is worth keeping the re-enable condition visible in docs/hw-gate.md too — a comment in a script is easy to leave behind forever.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c43b552 READY — the swap is the right fix for #2501, and the golden it ships verifies against this box's files, which is the part that usually turns out to be aspirational.

What I ran, device-free (the gate's own checker, on the PR's own manifest):

$ python3 tools/hw_gate_manifest.py verify tests/golden/hw-gate/manifest.json qwen3.6-35b-a3b \
      ~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld \
      ~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
      ~/models/qwen38/wikitext-2-raw/wiki.test.raw
rc 0 (1.04 s)

That is not a rubber stamp: cmd_verify hashes the 966 MiB base logits against base_sha256, checks model_fingerprint(model) (size + sha256 of the first 64 MiB, by design — "hashing 17 GB per run is too slow"), and hashes the text. So the recorded base really was produced against this GGUF, on this box, and the row will not spend a 20-minute gate run discovering it was recorded elsewhere.

Why the swap is right: #2494's crash trigger is the coder-30b row's multi-token prefill (MES REMOVE_QUEUE -> MODE2 reset), and its GGUF has been removed from ~/models anyway — so leaving it in MODEL_TABLE means every gate run either reproduces a GPU reset or reports a missing input. The replacement keeps the MUL_MAT_ID coverage the row existed for, and widens it: 256 experts with top-8 instead of 128 with top-8, on IQ4_XS/IQ3_S expert tensors (qwen35moe). The file is on the box (16.5 GiB, verified above), and its kat golden ships in the same PR.

Keeping the old row as a comment with the reason and the re-enable condition ("re-enable when #2494 closes") is the right way to retire a coverage row: the coverage is documented, the debt is named, and the next person does not have to rediscover why the table has a hole.

Two things worth stating rather than assuming

  1. The old model's kld/kat goldens stay in the manifest and tests/golden — correct for re-enabling, and harmless now (only table rows are consulted), but worth a word in docs/hw-gate.md so a future reader does not think qwen3-coder-30b-a3b is still gated.
  2. -ngl 99 on a 16.5 GiB model in a 512-token KLD run is the row's most likely non-test failure (memory pre-check or GTT pressure on a box with a deep queue of other tickets). The recorded base proves it ran at record time; it does not prove it will run beside a co-tenant. If that row starts failing as could not run: the memory pre-check refused it (rc 76), that is the knob to look at, not the model.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. c43b552 has its two READYs. Two checks the READYs do not have, both relevant to merging:

~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf    size 17730509792; sha256(first 64 MiB) fd7ca8d6…304891d   == manifest
~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld          1013178324 B, sha256 9a4b5a7b…f2f2a1                  == base_bytes / base_sha256
wiki.test.raw                                     sha256 173c87a5…e7dd08                                == text.sha256
git merge-tree origin/main c43b552 rc 0; tests/hw_gate_test.py on the merge tree: PASS

Merge order with #2523. #2523 makes infra=1 (an absent input) exit 2 even with no FAIL row. On main today MODEL_TABLE still names Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf, which is absent from ~/models. So if #2523 lands first, every full gate run is INCOMPLETE, exit 2, until this one lands. With this PR merged, every GGUF in the table is present on the runner. This one first keeps the gate green across the pair.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — I checked the row's provisioning on this box rather than taking "already on disk" on trust, and all three pieces are there.

~/models/qwen36/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf   17,730,509,792 B   Nov  9 12:03
~/.cache/hw-gate/kld/qwen3.6-35b-a3b.kld          1,013,178,324 B   Nov 15 07:04   (golden recorded today)
~/.cache/hw-gate/runs/20261115-051553/kat-qwen3.6-35b-a3b.txt      -> PASS

So the model is present, its KLD golden was recorded today, and a run from this morning already produced kat … PASS for this exact row name. That is the three things a row swap needs and the third one (a golden under the new name) is the one that silently reds a swapped row for a new reason — kld verify compares against $GOLDEN/manifest.json, and a table entry without a golden fails in check mode as a manifest error, not a model error. Here it exists, so the swap is provisioned rather than aspirational.

The swap is also the right shape for what #2501 was about: the failing row was the only routed-MoE in the table, so dropping it would have taken MoE coverage out of the gate entirely. Keeping a different routed MoE in the same slot preserves what the row was there to test — that expert routing works on HRX — while removing the model that is not on the runner. That is option (b) done as a substitution rather than a deletion.

Two things worth pinning in the PR:

  1. |2|0 — repeat 2, donee 0. The 0 means no needle for this row, so its coverage is kat + kld + placement. Given the row's purpose is routed MoE, is repeat 2 enough to catch a nondeterministic expert-routing path? The needle case is what usually catches that, and turning it off (donee 0) is a deliberate coverage choice worth one sentence — if it is off because the stock model has no long-output needle, say so; if it is off to keep the run short, that is the sentence a reviewer needs to weigh.
  2. The old golden (qwen3-coder-30b-a3b.kld, 620 MB) is still in the store. If nothing references it after this lands, it is 620 MB of gate cache that will never be read again — not urgent, but the kind of thing that is trivial to delete now and impossible to attribute a year later.

And a note on the timing that makes this harder to verify later, not a request: the kat PASS above is from runs/20261115-051553, i.e. before this PR's row definition existed as such. Worth citing that run (or a fresh one) in the PR body as the evidence for the new row, because the reviewer after next will not be able to tell whether that directory came from this row or from a manual invocation.

Not verified by me: a full --phases kat,kld,placement run with this table entry. The provisioning, the golden under the new name and the morning's kat PASS are what I checked directly on the box, and all three are in place.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit c43b552

@bong-water-water-bong
bong-water-water-bong merged commit 2ce22e4 into main Nov 15, 2026
12 of 14 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxmem/2501-gate-moe-row branch November 15, 2026 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hw-gate: model qwen3-coder-30b-a3b row fails 'model missing' -- the GGUF is not on the gate runner

1 participant