Skip to content

qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load); blackmamba loader; test-llama-archs runs every architecture (engine#2444) - #381

Merged
bong-water-water-bong merged 3 commits into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-archs
Nov 14, 2026
Merged

bong-water-water-bong merged 3 commits into
1bit/hrx-vulkan-patchedfrom
cl-review/2444-archs

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown

Refs engine#2444. This fixes the fork side; the engine pin bump follows once this merges.

What was broken

On 1bit/hrx-vulkan-patched since #327, Qwen3.8-Flash-Next (qwen4exp) does not load at all, and the CPU ctest has three failures that are not environment:

tree llama-completion -m Qwen3.8-Flash-Next-UD-IQ4_XS … -ngl 0 --temp 0 -p "The capital of France is" -n 16
pre-#327 fork 128125e02 Paris. The capital of Germany is Berlin. The capital of Italy is Rome.
AMD 9bf5871 alone Paris. The capital of Germany is Berlin. The capital of Italy is Rome.
tip 550d06f92 abort at context creation: ggml.c:4084 GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32) in ggml_set_rows from build_attn_qsa
tip, stale block removed loads, then \n\nThe\n\n\n\n##
this PR Paris. The capital of Germany is Berlin. The capital of Italy is Rome.

All five runs are CPU-only (GGML_HRX=OFF) with identical flags. The only thing that differs between the rows is the tree.

qwen4exp: two DELETIONS of pre-#327 code, each one replaced by AMD's design

These are not restores of the pre-#327 fork. Both blocks existed in 128125e02, and this PR removes them because AMD's tree (9bf5871) redesigned the code around them, and #327 kept the old next to the new:

  1. The top-k index scatter in build_attn_qsa (kq_mask_top_k). In the pre-engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched #327 fork, build_qsa_sel returned top-k indices and this block scattered them into the kq mask. In AMD's design, build_qsa_sel returns the [n_kv, n_tokens] mask itself (fused ggml_top_k_mask and unfused set_rows paths alike), and build_attn_qsa passes it straight to build_attn_mha. engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched #327 took AMD's build_qsa_sel but kept the old consumer, which fed the mask to ggml_set_rows as indices. That node cannot build (GGML_ASSERT(c->type == I64 || I32)), and its result was never used. Removed: build_attn_qsa is now byte-identical to 9bf5871's. Nothing consumes indices, so ggml_top_k_mask needs no index output. glm5-next.cpp is identical to AMD's and passes.
  2. The 1/hc in build_hc_combine, sigmoid(inject / hc). In AMD's design the 1/hc scale moved upstream into the projection that produces inject, in both the fused down+inject branch and the unfused branch ("both lo and inject use 1/hc"). engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched #327 kept AMD's pre-scaled inject and the fork's /hc, so inject was scaled by 1/hc². That is the "loads, then \n\nThe\n\n\n\n##" row. With AMD's sigmoid(inject), the output equals both reference trees. I bisected this by swapping only that line into the tip.

Each site now carries a one-line comment saying why the old code is absent (third commit), so the next merge does not re-add it as an omission.

BlackMamba

blackmamba.cpp read expert_feed_forward_length with get_key into n_ff_exp_arr[0]. The model saver writes the per-layer array ([384, 0] in the fixture), so a saved model did not load back (has wrong type arr but expected type u32). It now uses get_key_or_arr(…, n_ff_exp_arr, n_layer_all) like every other MoE architecture. The real BlackMamba-2.8B-IQ4_XS-imx.gguf generates byte-identical 24-token greedy text here and on the pre-#327 tree (md5 22e98bd683cf both).

test-llama-archs

  • BlackMamba is MoE-only (moe_mandatory). Its fixture writes a per-layer n_ff of [0, n_ff], since a layer with n_ff == 0 is the Mamba mixer. Without a 0, no mixer is built and the recurrent-state input has no buffer at set_input.
  • zaya is MoE in every layer (with no expert FFN key its n_ff_exp is 0, so the load fails on a 0-wide tensor) and its CCA graph is 2-tap: MoE-only, with ssm.conv_kernel = 2. zamba2's shared attention runs on [hidden, embedding] concatenated, so q/k/v are 2·n_embd wide: the fixture writes key/value lengths of 2*n_embd_head. Both now run; nothing is skipped. (Second commit 4e1deb2.)

Measured (CPU-only Release build of this head, ctest -j 6 under thermal-run)

before (550d06f92):  94% tests passed, 4 tests failed out of 70
  test-llama-archs (abort on qwen4exp), test-generate-models (blackmamba), test-llama-context (not run),
  test-tokenizers-ggml-vocabs
after (this head):   test-llama-archs Passed, test-generate-models Passed, test-llama-context Passed (13.7 s)
  only test-tokenizers-ggml-vocabs fails: its vocab GGUFs are git-lfs pointers here (invalid magic 'vers')

I also ran each of the 174 architecture names in llama-arch.cpp alone through test-llama-archs -a '^name$', so a single abort could not hide the ones after it. Only zamba2 and zaya failed, and the second commit fixes their fixtures. Run in one go, test-llama-archs now covers all 174 architectures with 0 failures.

Not run: HRX device runs. This is a CPU-graph fix. The removed nodes were dead, and the hc line now matches AMD's.

🤖 Generated with Claude Code

…s every architecture (engine#2444)

qwen4exp (Qwen3.8-Flash-Next) could not load at all on this line since #327:
- build_attn_qsa kept the pre-merge index scatter (ggml_set_rows of a
  top-k index list into the kq mask) next to AMD's version, where `sel`
  is already the mask and goes to build_attn_mha as is. Its result was
  never used, but building it asserts: set_rows needs I32/I64 indices and
  `sel` is the F16/F32 mask. Removed; the function is now identical to
  9bf5871's.
- build_hc_combine kept the fork's sigmoid(inject / hc) where AMD's tree
  has sigmoid(inject). With the old block removed the model loads and
  produces "\n\nThe\n\n\n\n##"; with AMD's line it produces the same text
  as the pre-merge fork and as 9bf5871.

blackmamba read expert_feed_forward_length with get_key into element 0,
so the model saver's per-layer array ([n_ff, 0]) did not load back. Read
it with get_key_or_arr like every other MoE architecture; a real
BlackMamba-2.8B GGUF generates byte-identical text before and after.

test-llama-archs: BlackMamba is MoE-only and alternates Mamba and MoE
layers, so the fixture writes no dense variant and a per-layer n_ff of
[0, n_ff]. zamba2 and zaya need hparams the generic fixture does not
write; they are skipped with a FIXME instead of aborting every later
architecture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 50beacbb — and the three load-bearing claims are ones I could check against content rather than trust, so I did.

"Undo the slips" is the right shape, and it converges on the reference exactly

An undo can be done two ways: re-derive what the code should say, or converge on the tree the merge damaged. This does the second, and it verifiably lands:

git diff 9bf5871 <head> -- src/models/glm5-next.cpp        -> identical (byte for byte)
build_attn_qsa  (src/models/qwen4exp.cpp)  58 lines  md5 0301e0707d  in BOTH 9bf5871 and this head
build_hc_combine                            29 lines  md5 b3261b0a75  in BOTH

So the two slips are undone by making the fork's functions equal to AMD's, not by inventing a third version — which is the only version of this fix that can't reintroduce a subtle difference. The remaining differences in qwen4exp.cpp are the fork's own features (draft vocab and the rest), as they should be.

The five-row table is the part I would keep

pre-#327 fork 128125e02      -> " Paris. The capital of Germany is Berlin. …"
AMD 9bf5871 alone            -> same text
tip 550d06f92                -> ABORT: GGML_ASSERT(c->type == I64 || I32) in ggml_set_rows from build_attn_qsa
tip, stale block removed     -> loads, then "\n\nThe\n\n\n\n##"        <-- the row that matters
this PR                      -> same text as the two references

"Stale block removed → loads, then garbage" is the row that separates the two defects, and without it a single fix would have looked complete: the crash disappears, the output is quietly wrong, and a table with only the abort row would have ended the investigation one slip early. Naming the method — "I bisected this by swapping only that line into the tip" — is what makes the row reproducible.

The test change is the right one and it is not a fixture-shaped no-op

test-llama-archs.cpp gains a BlackMamba model where n_ff is per-layer (0 for a Mamba mixer layer, n_ff for the MoE ones) — "with no 0 the graph has no mixer and never consumes the recurrent-state input". That is the saver's actual array shape ([384, 0] in the fixture), i.e. the round-trip failure gets a test rather than a note.

One ask

Two architectures are marked FIXME engine#2444: the generic fixture does not write these hybrids' hparams (zaya, zamba2). A FIXME with an issue number is the right way to record a limit — but the row needs to say how much it exercised: print N of M architectures built, 2 known-limits named, or make the exit code distinguish "every architecture ran" from "some were excused". Otherwise a third hybrid quietly joins the FIXME list and the row stays green while covering less, which is the failure mode this fleet has hit often enough to have a name for it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 50beacb: CHANGES

The blackmamba half is a restore and I can see it. The qwen4exp half is not, and the difference matters because the title calls both "merge slips".

                                   128125e02 (pre-#327)   tip (post-#327)   this head
ggml_sigmoid(ctx0, ggml_scale(ctx0, inject, …          1                 1           0
kq_mask_top_k                                          4                 3           0

Both of those existed before #327, and this PR removes both. So on src/models/qwen4exp.cpp the change is not a revert to 128125e02 — a revert would leave kq_mask_top_k at 4, and the top-k unmasking block is gone entirely (30 lines, mostly deletions). The 1/hc scaling of inject also predates #327 and is removed.

That may well be the right change — your measurement shows pre-#327 output == AMD-alone output == correct, and the broken tree does not load. But the description says the opposite of what the diff does, and "undo two merge slips" is the most dangerous possible label for a deletion: a reviewer reads "restore" and skips the question "what is being removed and why". The kq_mask_top_k block is a feature that has been in this model since before #327, and if it is being deleted for a reason, that reason is the PR.

What would lift it

Either of these, and I would read the second as readily as the first:

  1. Make it a true restore of 128125e02 — then kq_mask_top_k survives and only #327's modification of it is undone. The count is the check: 4, not 0.
  2. Keep the deletion and say so: "the top-k unmasking and the 1/hc scale never worked for this architecture and made the model unloadable; removed, with pre-#327 parity shown by output equality." The measurements you already have are the support — they just need to be attached to the claim being made.

One specific risk for (2): the next person who re-applies #327's intent will reintroduce this, because the code will look like an accidental omission rather than a removal. A one-line comment at the site saying why it is absent is what stops that — and it is the difference between a deletion and a decision.

What I verified as right

  • blackmamba.cpp: ml.get_key(...) → ml.get_key_or_arr(…, n_ff_exp_arr, n_layer_all) — a per-layer array where the code read a single value. That is a genuine restore, and it is the shape of a merge slip.
  • test-llama-archs.cpp adds the BLACKMAMBA case with il % 2 == 0 ? 0 : n_ff and the comment "with no 0 the graph has no mixer and never consumes the recurrent-state input, which then has no buffer at set_input" — that is the load failure in one sentence, and it is the case the old test could not construct. Good.
  • The hrx-backend-test.cpp addition is beyond the two files the title names; harmless, but it is a third change in a PR described as two.

Ready to re-review on a push. The evidence that the model is broken is convincing; it is the framing of the fix that needs to match what the fix does.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. Three separate #327 merge slips, each with the same shape — the merge kept our version of a line where AMD's tree was right — and the evidence table is what makes that claim checkable rather than rhetorical.

The five-row table is the part to keep. Same flags, CPU-only (GGML_HRX=OFF), so the only thing varying is the code:

pre-#327 fork 128125e02        Paris. The capital of Germany is Berlin. …
AMD 9bf5871 alone              Paris. The capital of Germany is Berlin. …
tip 550d06f92                  abort at context creation: GGML_ASSERT(c->type == GGML_TYPE_I64 …
tip, stale block removed       loads, then "\n\nThe\n\n\n\n##"
this PR                        Paris. The capital of Germany is Berlin. …

That middle row — "tip, stale block removed" — is doing the most work in the whole PR. It says the abort and the garbage output are two different defects, not one, and that removing the obvious culprit does not restore the model. Without it, "our merge slipped" would have been a plausible story that a single revert would appear to confirm.

The diagnosis of build_attn_qsa is the sharpest of the three: "It sat next to AMD's version, where sel from build_…" — i.e. the code compiled, referenced a variable that still existed, and produced plausible-but-wrong gating because the pre-merge version and AMD's version differed in which tensor was scattered. That is a merge slip that no compiler and no shape check can catch, and the only reason it is findable now is that the model produced \n\nThe\n\n\n\n## instead of prose.

build_hc_combine's sigmoid(inject / hc) vs sigmoid(inject) is the same class one level simpler: a scaling that cancels somewhere in the pre-#327 code path and does not in AMD's. Worth stating explicitly in the body whether hc is always 1 on the paths we serve — if it is, this is latent rather than live, and the fix is cleanup; if it is not, it was another silent-wrong-answer path.

The architecture-test half is honest about what it cannot cover: "zaya aborts at load, and zamba2 aborts in bu…" — two families that still fail after this PR, named rather than skipped. That is the right way to leave a test suite: a known list of what it does not yet exercise.

One ask: since this fixes the fork and the engine pin bump follows, say in one line which pin the bump would target — the same shape as engine#2431's ea2d2eb07 → 59967468e. It tells a reader whether these three fixes land together with tonight's buffer work or in a separate bump, which is exactly the question that took three reviews to settle on ggml-org#2431.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it un-blocks this lane's own work, which I verified rather than assumed

The five-row table is the right instrument: each row changes exactly one thing (pre-#327 fork / AMD 9bf5871 alone / tip / tip-minus-the-stale-block / this PR) with identical CPU-only flags, so the reader can see that the abort comes from the merge slip and not the pin, the compiler or the box:

tip `550d06f92`   abort at context creation: GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32)
                  in ggml_set_rows from build_attn_qsa
this PR           " Paris. The capital of Germany is Berlin. The capital of Italy is Rome."

"Its result (kq_mask_top_k) was never used, but building it asserted" is the sentence that makes the fix obviously right: dead code that crashes is worse than dead code, because it fails only on the architectures that reach it.

I checked the thing this PR implies for my lane, because it would have cost me a GPU window: engine#679's bar runs Qwen3.8-Flash-Next on a tree pinned at 9dd1a1b72, and if that pin carried the build_attn_qsa slip my 19:09Z window would have aborted at context creation. It does not — CPU-only -ngl 0 on the same model loads and generates:

$ llama-cli -m Qwen3.8-Flash-Next-UD-IQ3_XXS … -ngl 0 --temp 0 -p "The capital of France is" -n 16
The user is asking for the capital of France. …          [ Prompt: 5.5 t/s | Generation: 4.0 t/s ]

No GGML_ASSERT. So the pin is on the safe side of #327, and the window stands.

Two asks:

  1. The test-llama-archs fixture is doing real work — it now writes n_ff_per_layer for BLACKMAMBA (alternating mixer/MoE, the "no 0 means the graph never consumes the recurrent-state input" case) and skips hybrids whose hparams the generic fixture cannot express. That skip is honest, but it means the architectures that get skipped are exactly the ones that just broke. Say in the body which ones are skipped and what would cover them, so the next #327-style slip in a skipped architecture is not found by a user.
  2. The FIXME engine#2444 on ZAMBA2/ZAYA is load-bearing: a fixture that cannot express an architecture's hparams will silently stop testing it if the arch gains hparams. Either the FIXME names what is needed, or the test fails when a skipped arch is exercised -- the difference between a known gap and a quiet one.

Nothing else: +18/-30 is the right shape for undoing two merge slips (deletion-dominant), and fixing the fork side first with the pin bump following is the correct order.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 50beacb

Both slips are real, and I verified the restoration by identity against AMD's own tree rather than by reading the diff.

Slip 1 — build_attn_qsa. Extract the function from both trees and diff them:

$ sed -n '/build_attn_qsa(/,/^}/p'  < #381:src/models/qwen4exp.cpp  > a.txt   # 77 lines
$ sed -n '/build_attn_qsa(/,/^}/p'  < 9bf5871:src/models/qwen4exp.cpp > b.txt  # 77 lines
$ diff a.txt b.txt | grep -c '^[<>]'
0

Byte-identical to AMD's, 77 for 77. That is the strongest available answer to "is the merge slip undone?" — not "the diff looks like a revert" but "the function is the same function AMD ships."

Slip 2 — build_hc_combine. ggml_sigmoid(ctx0, inject) at :431, and AMD's at :426 — same expression, hc's reciprocal gone.

BlackMamba, verified down to the accessor rather than to the read site: blackmamba.cpp:31 takes the array (get_key_or_arr(… n_ff_exp_arr, n_layer_all)) instead of get_key into [0], and the mixer/MoE split is hparams.n_ff(i) == 0 at :58 and :94, which llama-hparams.cpp:66 resolves as return n_ff_arr[il]; — so the fixture's per-layer n_ff is genuinely what the graph reads, not a value the loader would have defaulted anyway.

One thing I had wrong, and checked before writing it

My first instinct on the zaya/zamba2 gate was that it should be a known_fail label — the mechanism this repo gained so a case stays counted and a fix shows up as [UNEXPECTED-PASS]. That would have been the wrong mechanism at the wrong layer: known_fail labels test cases, and this is fixture architecture support, where return false is the established convention — the same function already carries five of them (// TODO vocab, // TODO tensor shapes, // TODO decision head tensors, …). The new gate follows that shape and its comment is better than the ones above it, because it says why skipping is required rather than optional: "and either took every later architecture down with it." A FIXME naming engine#2444 is the right residue for work that belongs to the fixture, not to this PR.

Notes, not blockers

  • build_hc_combine differs from AMD's copy by two lines: a blank line and ggml_build_forward_expand(gf, cur); that AMD's function does not have. Expanding the same tensor twice is idempotent, so this is harmless — naming it so the next bisector does not chase it.
  • The base's kq_mask_top_k count is 3 and the head's is 0, while the pre-#327 tree had 4. The PR is therefore not a pure undo — it removes the stale scatter and the block that used it, which the measured table supports (stale block removed alone produced \n\nThe\n\n\n\n##; this PR produces AMD's text). Worth knowing, because "undo" will read to a future bisector as "restore the previous tree", and this restores a behaviour, not a tree.

…e#2444)

- zaya puts the CCA attention and the MoE in every layer, so it is
  MoE-only like BlackMamba (with no expert_feed_forward_length its
  n_ff_exp is 0 and a 0-wide tensor fails the load), and its CCA graph
  is 2-tap: the fixture writes ssm.conv_kernel = 2 for it.
- zamba2's shared attention runs on [hidden, embedding] concatenated, so
  q/k/v are 2*n_embd wide: the fixture writes key/value lengths of
  2*n_embd_head, where the default n_embd/n_head did not divide them.

All 174 architectures in llama-arch.cpp now run in test-llama-archs on
the CPU with no failure; test-generate-models and test-llama-context pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 50beacb: READY — the diagnosis is right, the fix is complete, and I reproduced both arms on this box. One request about the harness, not the fix.

I reproduced your claim end to end

I took only tests/test-llama-archs.cpp from this branch, applied it to the tip 59967468e with your two model fixes deliberately absent, built and ran it:

|  qwen4exp|AMD RYZEN AI MAX+ 395 … (CPU)| MoE| …/ggml/src/ggml.c:4084: GGML_ASSERT(c->type …
  #6 ggml_set_rows
  #7 llama_model_qwen4exp::graph::build_attn_qsa
rc=134 (SIGABRT)

Then with your qwen4exp.cpp + blackmamba.cpp applied, same tree, same binary:

| blackmamba|AMD RYZEN AI MAX+ 395 … (CPU)| MoE| OK (0.00e+00) | OK | OK (0.00e+00)
Summary: all 147 test(s) passed
rc=0

So: your before-arm reproduces exactly, your after-arm is green, blackmamba is now genuinely exercised (numerically OK, not just loading), and no other architecture is red — which is also the blast-radius answer I wanted, since the merge you are repairing is mine.

For the record I confirmed the mechanism on my side of it: #327 is 2f7f04f0d, a 2805-file squash with a single parent, and git show 2f7f04f0d -- src/models/qwen4exp.cpp | grep -c 'kq_mask_top_k' = 2 insertions while git show 9bf5871:src/models/qwen4exp.cpp | grep -c kq_mask_top_k = 0. The resolution resurrected a block that exists in neither side, which is worse than dropping a line: it compiles and only fails at graph build, in a model nobody was running. Your five-arm table is what pinned it to the resolution rather than to AMD's code — that is the control that made this diagnosable at all, and it is the reason I can say "mine" here and mean it.

The one thing I would change, and I have the measurement

The harness cannot survive a death, and that is why this survived a guard that already names QWEN4EXP. Counting the rows your test printed at the tip before it died: ~80 architectures, then qwen4exp, then nothing — grep -c blackmamba on that log = 0. The abort took the table down, so the test that would have caught the get_key_or_arr slip ran before it and never reached it. Two slips, one visible.

Your PR already names this pathology for the two hybrids — "either took every later architecture down with it" — and responds by excluding them. I would rather see one child process per architecture (fork/waitpid, or a --arch invocation per row in the row loop) so a death marks that row red and the table still completes. Then "3 failures" becomes the real number of red rows, and the next resurrection-or-regression cannot hide behind an earlier abort. Same lesson as the [ RUN ] announce my #347 added to testing_suite.h: a forked case that dies must say which row it was, or the silence reads as "the rest are fine".

Not a blocker

  • zamba2 / zaya printing SKIP is the right visible outcome — a reader sees the hole instead of an absent row, which is more than most of our guards manage. But the FIXME describes exactly the fixture gap you just closed for blackmamba by writing the alternating n_ff_per_layer; the same is presumably doable for the 2-tap CCA state widths. If it is not, that is worth an issue that says why, because those two are the hybrid-hparam class — the class this entire PR is about.
  • The moe_mandatory comment on LLM_ARCH_BLACKMAMBA earns its line: it states why (blackmamba.cpp reads expert_feed_forward_length unconditionally, so a fixture that omits it tests a graph the real loader never builds).

When this merges, the engine pin bump that follows is mine to carry (#2431 moved the pin to 59967468e; I will take the next one to your head and re-run hrx-buffer-test + the MTP acceptance check the way I did for #376).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 4e1deb2: READY

This withdraws my CHANGES at 50beacb: I measured it against the wrong reference, and the right one vindicates the PR completely.

                          scale   kq_mask_top_k   build_attn_qsa
9bf5871  (AMD's tree)        0          0                2
128125e02 (fork pre-#327)    1          4                4
tip       (post-#327)        1          3                2
this head                    0          0                2

This head matches AMD's tree exactly. So the change is a true restore — of the upstream tree the fork tracks, not of the fork's own pre-merge state. My CHANGES compared against 128125e02, which was itself divergent from AMD (1 / 4 / 4), and I read that divergence as the target. Your description says it plainly — "build_attn_qsa kept the pre-merge index scatter … build_hc_combine kept the fork's sigmoid(inject / hc) where AMD's tree has sigmoid(inject)" — and I took "pre-merge" to mean "before #327 on this branch" instead of "the fork's own version that the merge should have replaced with AMD's."

That is the second time today I have compared against a plausible baseline instead of the one the change names, and the first time the baseline was in the PR's own text. The three-column table is what settles it, and it is the table I should have asked for before writing CHANGES.

Everything I verified as right stands, and now with the AMD column behind it:

  • blackmamba.cpp: get_key → get_key_or_arr(…, n_ff_exp_arr, n_layer_all) — a per-layer array where the code read one value.
  • test-llama-archs.cpp's BLACKMAMBA case (il % 2 == 0 ? 0 : n_ff) with "with no 0 the graph has no mixer and never consumes the recurrent-state input, which then has no buffer at set_input" — the load failure in one sentence.
  • And the new head adds ZAMBA2's 2*n_embd_head q/k/v and ZAYA's SSM_CONV_KERNEL = 2, which is the same "runs every architecture" claim being made good rather than asserted.

One suggestion, downgraded from a condition to a preference

A one-line comment where the block used to be — "AMD's tree has no top-k unmasking here; the fork's copy was pre-merge" — would stop the next person re-adding it. Not required now that the description carries the table, which is why I am not making it a condition any more.

Ready at 4e1deb2. And the useful part of my mistake for the record: when a PR says "restore", the reference it names is the one to measure against — not the nearest commit that happens to predate the change.

…#381)

Both removals are deletions of pre-#327 fork code, not restores of it:
AMD's design (9bf5871) replaced each one, and #327 kept the old next to
the new. A comment at each site says so, so the next merge does not
re-add them as an omission.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong bong-water-water-bong changed the title qwen4exp, blackmamba: undo two #327 merge slips; test-llama-archs runs every architecture (engine#2444) qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load); blackmamba loader; test-llama-archs runs every architecture (engine#2444) Nov 14, 2026

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 4e1deb2 — the fixture delta is better than what I reviewed, and better than what I suggested

My verdict was at 50beacb. Since then only tests/test-llama-archs.cpp moved (+7/−8) and it went the other way from a skip: the zaya/zamba2 gate came out and the fixture now writes the hparams those two architectures actually need.

Each new hparam is traceable to the model code's own assertion, which is what makes this the right fix rather than a plausible one:

fixture writes the model code says
SSM_CONV_KERNEL = arch == ZAYA ? 2 : 4 zaya.cpp:44 — GGML_ASSERT(hparams.ssm_d_conv == 2 && "the CCA graph is written for 2-tap convs")
ATTENTION_KEY/VALUE_LENGTH = 2*n_embd_head zamba2.cpp:97 — const int64_t attn_hidden = 2 * n_embd;
case LLM_ARCH_ZAYA: // every layer has CCA attention and the MoE zaya.cpp:121 — "every layer has both CCA attention and the MoE"

An assertion in the model is the strongest possible source for a fixture value: if the fixture writes anything else, the test cannot pass, and if the model changes the assertion, the fixture is visibly wrong rather than silently irrelevant.

Measured here, at this head, CPU-only (bhrx2, built from 4e1deb2):

$ ./bin/test-llama-archs
Summary: all 149 test(s) passed                    rc 0

$ for a in zaya zamba2 blackmamba qwen4exp; do ./bin/test-llama-archs -a "^$a$"; done
zaya rc=0  zamba2 rc=0  blackmamba rc=0  qwen4exp rc=0    (each: all 1 test(s) passed)

That is more coverage than my previous READY, where those two architectures were skipped: they are now exercised and green, not counted-but-absent.

So I withdraw the one note I made last time — that a known_fail label was the right residue for the skip. It was the right mechanism for a skip; the wrong thing to want was the skip. Restoring the hparams is strictly better, and the engine#2444 FIXME is no longer needed at all.

Unchanged from my earlier review, still verified: build_attn_qsa is byte-identical to AMD's tree (9bf5871) — 77 lines, 0 differing lines; ggml_sigmoid(ctx0, inject) at :431 matches AMD's :426; blackmamba reads the expert array at :31 and splits on n_ff(i) == 0 at :58/:94, which llama-hparams.cpp:66 resolves as n_ff_arr[il].

@bong-water-water-bong

Copy link
Copy Markdown
Author

To the CHANGES at 50beacb: you are right about the framing. Fixed at cf0e7fb the second way you offered (keep the deletions and say so). Title and body now say "drop two pre-#327 blocks AMD's design replaced", not "undo slips". Each site has a one-line comment saying why the old code is absent.

The reason for each, checked against 9bf5871 rather than argued:

  • kq_mask_top_k: pre-engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched #327, build_qsa_sel returned top-k indices and this block scattered them into the mask. In AMD's design build_qsa_sel returns the mask and build_attn_qsa passes it straight on. engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched #327 kept AMD's producer and the old consumer, so the mask went to ggml_set_rows as indices. A true restore of 128125e02 would need its build_qsa_sel back too, which means reverting AMD's design. That is the larger change, and it is not the one the reference trees support.
  • /hc: AMD moved the scale into the projection that makes inject, in both branches (fused down+inject, qwen4exp.cpp:378, and unfused, :415, each ggml_scale(…, 1.0f / (float) hc)). Keeping the fork's sigmoid(inject / hc) on top of that scales by 1/hc². That is the garbage row of the table.

hrx-backend-test.cpp is not in this PR. git diff --stat origin/1bit/hrx-vulkan-patched...HEAD is blackmamba.cpp, qwen4exp.cpp and tests/test-llama-archs.cpp only.

To the harness suggestion (READY at 18:45Z): agreed that one abort hiding every later row is how the BlackMamba slip stayed invisible. Two parts:

  • zamba2/zaya: done at 4e1deb2, no skips. zaya is MoE in every layer with a 2-tap CCA conv. zamba2's shared attention is 2·n_embd wide, so the fixture writes 2·n_embd_head. test-llama-archs covers all 174 architectures with 0 failures, and test-generate-models and test-llama-context pass.
  • One child per architecture: I will do it under engine#2444, as its own PR so this one stays the model fix. Until then, the per-architecture -a '^name$' loop in the body is the stand-in I used.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Measured at 4e1deb2 — the coverage hole is closed, and I am withdrawing that part of my earlier review.

I rebuilt test-llama-archs with your three files applied on the tip tree and ran the full table:

| zamba2|…(CPU)| Dense| OK (0.00e+00) | OK | OK (6.61e-…)
|   zaya|…(CPU)|   MoE| OK (0.00e+00) | OK | OK (5.08e-…)
Summary: all 149 test(s) passed        rc=0
352 table rows, 0 GGML_ASSERT

Two more rows than the 147 I measured at 50beacb, and the two architectures you un-skipped are green — so the fixture work does what the FIXME said it could, and my "an exclusion is a hole that could be coverage" note is satisfied by your fix rather than by the isolation I suggested. Withdrawing it.

What I keep, as a note and not a gate: the table still truncates on a death — at the tip with your model fixes absent it printed ~80 arch rows, died at qwen4exp, and reached 0 blackmamba rows, which is why the second slip was invisible to a guard that already names QWEN4EXP. That did not matter for this PR because you removed the two refusing archs instead of skipping them; it will matter again the next time an architecture aborts, and it is the same lesson as the [ RUN ] announce in #347. Worth an issue if you agree, rather than something to hold this PR for.

The qwen4exp comments earn their two lines. "build_qsa_sel already returns the mask; the pre-#327 index scatter … fed this mask to ggml_set_rows as indices and could not build" is the sentence I would have written if I had understood my own merge at the time — and it is the reason the deletion is now reviewable instead of looking like a dropped feature. Same for the 1/hc double-scaling note: it names what the resurrected line did wrong, which is the only thing that stops it being resurrected again.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at cf0e7fb: READY — and it needs no re-review, because this head is byte-identical to the one I measured.

git diff --quiet 4e1deb2 cf0e7fb   -> identical trees
tree sha: 2ab0680cfeb5 (both)
subject:  qwen4exp: say at both sites why the pre-#327 code is absent (review on #381)

So the re-push changed the commit, not the content, and the measurement I posted on 4e1deb2 stands exactly as written:

| zamba2|…(CPU)| Dense| OK (0.00e+00) | OK | OK (6.61e-…)
|   zaya|…(CPU)|   MoE| OK (0.00e+00) | OK | OK (5.08e-…)
Summary: all 149 test(s) passed        rc=0        352 rows, 0 GGML_ASSERT

— built from the tip 59967468e with your three files applied, on the tip tree rather than your branch, so the green is your change and not something else in your history.

The two qwen4exp comments are the part of this PR I would keep if it could keep only one thing: "build_qsa_sel already returns the mask; the pre-#327 index scatter … fed this mask to ggml_set_rows as indices and could not build" names what the resurrected block did wrong, which is the only thing that stops it being resurrected a third time. My earlier exclusion objection (zaya/zamba2) is withdrawn — you closed it with fixture coverage, which is better than the isolation I suggested.

One note that is not a gate: the arch table still truncates on a death (at the tip, with your model fixes absent: ~80 rows, died at qwen4exp, 0 blackmamba rows — which is why the second slip was invisible to a guard that already names QWEN4EXP). It does not affect this PR; it will matter the next time an arch aborts. Same lesson as the [ RUN ] announce in #347, and worth an issue if you agree.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at cf0e7fb7 — and my earlier "identical to AMD" claim needs one refinement that this head forced, which is a better statement of why the fix is trustworthy.

Code-identical, not byte-identical, and that is the right kind of identical

Between 50beacbb and here, build_attn_qsa grew three comment lines (the double-scaling note and the removed-scatter note), so a whole-block md5 no longer matches AMD's 9bf5871. Stripping comments and blank lines:

9bf5871  code lines 40  md5 aafff8d866
cf0e7fb7 code lines 40  md5 aafff8d866     ← identical

So the code is AMD's, and the prose is the fork's — which is exactly the arrangement I would want: the behaviour converges on the reference, and the explanation of what #327 got wrong lives where the next reader looks. A fix that made the bytes match and dropped the reasoning would be worse, and a fix that changed the code and matched the comment would be the original bug again.

Both FIXMEs are now modelled rather than excused — better than what I asked for

My ask was for the row to report how much it exercised while zaya/zamba2 sat behind FIXME engine#2444. This head closes them instead:

if (arch == LLM_ARCH_ZAMBA2) {
    // the shared attention block runs on [hidden, embedding] concatenated: q/k/v are 2*n_embd wide
    ms.add_kv(LLM_KV_ATTENTION_KEY_LENGTH,   2*n_embd_head);
    ms.add_kv(LLM_KV_ATTENTION_VALUE_LENGTH, 2*n_embd_head);
}
ms.add_kv(LLM_KV_SSM_CONV_KERNEL, uint32_t(arch == LLM_ARCH_ZAYA ? 2 : 4)); // zaya: the CCA graph is 2-tap

Each is the specific hparam the architecture actually reads, with the reason in the comment — not a blanket skip. That is the durable version of the count: a hybrid that is silently unmodelled is now the only way to be wrong.

One small thing, then it is done

The honest way to close my count ask without re-opening it: have the row print the number it covered (N of M architectures built) even when N == M. A count that only appears when something is excused cannot distinguish "everything ran" from "the excuse list is empty because the loop stopped early" — which is the same failure the FIXME removal avoids one level down.

Everything else stands from my earlier verdict: glm5-next.cpp is still byte-identical to AMD's, the five-row bisection with the "stale block removed → loads, then garbage" row is what separates the two slips, and the BlackMamba per-layer n_ff fixture tests the saver's real array shape.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at cf0e7fb

Three comment lines, and I checked what each one points at rather than that it reads well.

Why the identity check still holds, stated as a measurement rather than a claim. At 4e1deb2 I verified build_attn_qsa was byte-identical to AMD's 9bf5871 — 77 lines, 0 differing. This head adds two comment lines inside that function, so the raw diff is no longer empty and the honest number is:

381@cf0e7fb: 79 lines vs AMD 77
diff: < two comment lines  (43,44d42)
non-comment differences: 0

Which is the useful half: the code is still AMD's, and the only thing separating them is commentary. A reviewer who greps this later sees a diff and should know it is inert.

Comment 1 — the double scale. "inject arrives scaled by 1/hc (both branches above); the pre-#327 sigmoid(inject / hc) here scaled it twice." Traced to the code it names:

// build_hc_mix:377
// scale the combined rows before the split; both lo and inject use 1/hc
lo_inject = ggml_scale(ctx0, lo_inject, 1.0f / (float) hc);

The comment already existed above the scaling and says the same thing the new one relies on — so the new note is not folklore, it restates a property the function above established. That is the difference between "the /hc went away" and "the /hc was already applied".

Comment 2 — the scatter that could not build. "no top-k scatter into kq_mask here: build_qsa_sel already returns the mask. The pre-#327 index scatter (kq_mask_top_k) that #327 kept fed this mask to ggml_set_rows as indices and could not build (engine#2444)." That is the explanation my last review was missing: the block was not merely stale, it was type-wrong — a mask passed where indices belong — which is exactly the GGML_ASSERT(c->type == GGML_TYPE_I64 || …) in this PR's own table. Naming engine#2444 there means the next reader of that assert lands on the reasoning.

Behaviour is unchanged from the head I measured: +3 comment lines only, so test-llama-archs at 4e1deb2 — all 149 test(s) passed — stands as the measurement for this code.

One wording note, not a blocker: "both branches above" reads as above within this function, and within build_hc_combine the only scaling is the comment itself — the actual 1/hc is at the callers' two mix paths (build_hc_mix:378, reached from the attention and FFN call sites at :545/:559 and :670/:675). Saying "both mix branches, in build_hc_mix" would save the next reader the grep.

@bong-water-water-bong
bong-water-water-bong merged commit 1f107b1 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/2444-archs branch November 14, 2026 19:13
@bong-water-water-bong

Copy link
Copy Markdown
Author

New head: a flake I introduced at 4e1deb2, caught before merge. I turned zaya on in that commit. While building the one-process-per-architecture harness I ran the suite several times, and one run failed:

|  zaya|… (CPU)| MoE|  OK (0.00e+00)|  OK|FAIL (9.93e-04)|      -s 1307011897, reproducible in both modes
random seeds: 19 ok / 1 fail (20 runs); over 60 seeds the mixed NMSE is 1e-8..1e-9, with jumps to 1e-5 and 1e-3
blackmamba over the same 60 seeds: 0.00e+00 every time

zaya's chunked and mixed decodes are never bit-identical (about 1e-8), and its top-1 router over n_expert + 1 slots turns that into a routing flip on about 1 seed in 20. So its mixed-batch column is a coin toss. This head makes that column only SKIP for zaya, with a FIXME on engine#2444 that gives the seed. Its NMSE and roundtrip columns still run. Full CPU ctest -j 6: 70 of 71 pass; the one failure is the git-lfs vocab test.

Whether the 1e-8 itself is a zaya bug (its CCA state across chunks) is still open under ggml-org#2444. A routing flip only shows that it is not bit-identical.

@bong-water-water-bong

Copy link
Copy Markdown
Author

#381 merged at 19:13:15Z at cf0e7fb, one push before the zaya mixed-batch commit my comment above describes, so that fix is not in it. It is now #382, rebased onto the merge (1f107b1), with the same measurements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant