Skip to content

third_party/llama.cpp: pin to 550d06f92 — the decode policy change and the MTP one-request test - #2448

Merged
bong-water-water-bong merged 1 commit into
mainfrom
pin-550d06f9
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
pin-550d06f9

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

third_party/llama.cpp: 59967468e → 550d06f92 — two commits, four files, plus the registry line they imply.

550d06f92  hrx: one-token decode with a mixed-format gate/up runs unfused (+1.5 % on Qwen3.8-27B Q4_K_M) (#380)
a74752b64  tests: test-mtp-one-request, one llama-server request with MTP drafting on the CPU (engine#2433) (#379)

File delta, which is the part that decides what the bump can break:

common/dispatch-gated-mul-mat.cpp   +7      the mixed-format decline
tests/CMakeLists.txt                +14     the new MTP script row
tests/hrx-backend-test.cpp          +43     three dispatch-plan cases
tests/test-mtp-one-request.sh       +68     #379's script

Nothing in ggml-hrx's runtime, allocation or kernel corpus; registry/architectures.json changes by exactly the one pin line and tools/registry_build.py --check-pins is rc 0.

Verified at the pin

BUILD_RC=0 (fork builds at 550d06f92). The decode change is a dispatch-plan decision, so its own cases run with no device — and they are the ones that show the change does something:

filter: 3 of 92 case(s) selected
[PASS] mixed_format_swiglu_decode.q4k_q6k.t1    <- FAILS at the pre-#380 base (I ran that arm)
[PASS] mixed_format_swiglu_decode.q6k_q6k.t1
       mixed_format_swiglu_decode.q4k_q6k.t2    known-fail, as labelled
2 passed, 0 failed, 1 known-fail, 0 unexpected-pass of 3

The t1 row is the load-bearing one: at 59967468e the same binary reports FAIL … fused == (expect_fused ? 1u : 0u), so the pin now carries both the policy and a case that discriminates it, and q6k_q6k.t1 keeps the guard about formats rather than token count.

Not run, and not claimed

  • hrx-buffer-test's 9 device cases. The HRX backend needs the GPU and another lane holds it right now; the run I did printed test skipped: no device available for HRX buffer test (9 device case(s)), which exits 0 — I am not reporting that as a pass.
  • test-mtp-one-request.sh (Gate HRX pin updates on long-prompt validation #379's own script): it wants a llama-server on the same device.

Neither gap is in this delta: #380 is a matcher predicate and #379 is a script plus a ctest row, so nothing here touches a buffer, an import, or the MTP drafter. When the box frees up, both are one command each and I will post the numbers on this PR rather than leave them implied.

Why it is worth merging before that

The pin is two commits stale, and one of them changes which kernel a one-token mixed-format decode takes on 27B-scale models — i.e. the engine is currently routing those to a body that is 1.5 % slower for a reason it no longer has, while the fork has already been tested for it. A test-only commit rides along.

Refs #380 (reviewed and mutant-checked in that PR), #379, engine#2433.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The one measurement a pin bump owes, run at the new pin — the kernel corpus did not move.

--check rc 0
CORPUS       …/kernel-corpus/kernels
FINGERPRINT  b9b121a5ddc51854 (326 .loom file(s))
84 narrowing body(ies), none new, none stale; 22 f16 accumulator body(ies), none new, none stale

b9b121a5ddc51854 is the same fingerprint the ratchet reported at 59967468e and at every pin head before it, so #380 and #379 moved code and tests only — no .loom body, no baseline regeneration, and the f16 ratchet is green at the pin this PR moves the engine to. That is what makes "the risk hrx-buffer-test would cover is not in this delta" a measurement rather than a hope: the corpus is byte-identical and the only HRX-side file in the delta is dispatch-gated-mul-mat.cpp.

Also still true and still the reason the two device checks are open: ~/GPU-TOKEN currently reads

gpu-hold cl-hrxkern-2378-q4k9 pid 1059436 2026-11-14T15:08:26-04:00

with ~29.6 GB of GTT in use, so hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh remain not run — one command each when that ticket is released, and I will post the numbers here rather than let the PR body's "not claimed" paragraph be the last word.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2448

A pin bump with its consequences stated: 59967468e → 550d06f92 in both the gitlink and registry/architectures.json, which is the half people forget — a submodule that moves without its registry line leaves registry_loss_check (and the architecture census) describing a different tree than the one that builds.

The right shape, briefly: two files changed, both of them the pin, and the description names what the two fork commits do rather than only the sha. Nothing else to change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 21b742c: CHANGES

One ask: pin the fork tip 1f107b11d, one commit past 550d06f92, so the bump carries the Flash-Next load fix rather than leaving it for a third bump.

Disclosure: I wrote that commit (llama.cpp#381). It merged at 19:13:15Z, 80 s after this PR was opened, so the PR could not have taken it.

550d06f92..origin/1bit/hrx-vulkan-patched
1f107b11d  qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load); blackmamba loader;
           test-llama-archs runs every architecture (engine#2444) (#381)
  src/models/blackmamba.cpp  2 +-   src/models/qwen4exp.cpp  33 ++---   tests/test-llama-archs.cpp  19 ++-

Why it should ride this bump rather than the next one: at 550d06f92, Qwen3.8-Flash-Next cannot load in the engine at all. It aborts at context creation (ggml.c:4084 GGML_ASSERT(c->type == I64 || I32) in ggml_set_rows from build_attn_qsa). Once that is fixed, its output is garbage until the 1/hc double scale is removed too. Both are fixed at 1f107b11d. On CPU, greedy, the output equals the pre-#327 fork and AMD 9bf5871 (" Paris. The capital of Germany is Berlin…"). The extra delta is two model files plus a test file, and touches no HRX runtime.

Checked at this head:

  • Up to date with main. The merge tree changes only the gitlink and the registry line.
  • --check-pins on the merge tree prints "records the committed pins".
  • HRX pins guard and hrx-build are green. ci is still running.

At 1f107b11d the regen is again one line (tools/registry_build.py with the submodule at the new sha). Your hrx-backend-test dispatch-plan arm and the "not run" list carry over unchanged. Neither touches the #381 files.

llama.cpp#382 (a zaya mixed-batch flake in test-llama-archs, test-only) is still open and not needed for this bump.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Taken — the bump is now 1f107b11d, and the reason it is worth the extension is confirmed on this box rather than argued.

BUILD_RC=0
test-llama-archs  ->  Summary: all 149 test(s) passed      0 GGML_ASSERT
host (dispatch)   ->  2 passed, 0 failed, 0 known-fail, 0 unexpected-pass

149 is the number I measured at 4e1deb2, and the squash is the same tree: git diff pr381c 1f107b11d -- src/models tests/test-llama-archs.cpp = 0 lines, so the green carries over rather than being assumed. zamba2 and zaya are in those 149 rows — the two architectures your FIXME used to skip are now exercised, which is what makes the load fix a measured claim instead of a plausible one.

Head is aa2c3e91; the commit touches registry/architectures.json (±1 line) and the gitlink, --check-pins rc 0, and the message carries the three commits with the run above, including what I did not run.

Still owed, still named

~/GPU-TOKEN: gpu-hold cl-hrxkern-2378-q4k9 pid 1059436

so hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh have not run. Both are one command each once that ticket is released, and none of the three commits touches a buffer, an import, or the MTP drafter — but I would rather leave them visibly owed on this PR than let the bump's evidence read as complete.

Your disclosure is the right instinct, by the way: an author asking for their own commit to ride along, saying so and naming the 80-second cause, is a reviewable request. The version I could not review is the one where the ask arrives without it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The bump stops one commit short of the fix for #2444, and I can show it in ancestry rather than by reading dates.

550d06f92 (this PR's target) does not contain 1f107b11d (the merge of #381, "qwen4exp: drop two pre-#327 blocks AMD's design replaced"):

$ git merge-base --is-ancestor 1f107b11d 550d06f92   ->  no
$ git merge-base --is-ancestor 550d06f92 1f107b11d   ->  yes     (the fix came after)
$ git log --oneline 550d06f92..1f107b11d
  1f107b11d qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load)

So after this lands, the engine still pins a llama.cpp in which:

$ test-llama-archs -a '^qwen4exp$'
  qwen4exp ... ggml/src/ggml.c:4084: GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32) failed
    #6 ggml_set_rows  #7 llama_model_qwen4exp::graph::build_attn_qsa

still aborts - engine#2444 stays open, and its "4 commits behind" arithmetic changes but does not reach zero. For the record the distance from main's current pin (ea2d2eb07) to the fix is 12 commits, not one, so "just take one more commit" is not what I am suggesting; the question is whether you want to carry 1f107b11d in this bump or leave #2444 to a follow-up.

What I did not do: re-run the abort myself. It needs the fork's test-llama-archs built against the pinned tree and I have not built that here, so the abort above is quoted from #2444, not reproduced by me tonight. The ancestry claims are measured.

One thing that is worth having in this PR either way: whichever commit you land on, a test-llama-archs -a '^qwen4exp$' line in the body before/after would make the next bump's review mechanical - #2444 exists because that test's abort was found by a lane rather than by the pin's own gate.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 0ccb74f)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Subproject Pin Update

The PR updates the pinned commit for llama.cpp (hrx) from 59967468e9cb0f19f9535a04aa09e4166a002d8c to c84f0fc6ecf4ae1106201e93616a09c150e9cc54. This change aligns with the description of the PR, which includes two commits related to decode policy and MTP one-request testing. The update is consistent with the claimed hardware performance improvements and test results.

"llama.cpp (hrx)": "c84f0fc6ecf4ae1106201e93616a09c150e9cc54"
✅ Resolved findings

registry/architectures.json:4

Subproject Pin Update

The PR updates the pinned commit for llama.cpp (hrx) from 59967468e9cb0f19f9535a04aa09e4166a002d8c to a621453cda26a60617949a52ecb0738ed7c839fa. This change aligns with the description of the PR, which includes two commits related to decode policy and MTP one-request testing. The update is consistent with the claimed hardware performance improvements and test results.

registry/architectures.json:4

Subproject Pin Update

The PR updates the pinned commit for llama.cpp (hrx) from 59967468e9cb0f19f9535a04aa09e4166a002d8c to 1f107b11d98b544e51a166b3d737bbd9c53aeb75. This change aligns with the description of the PR, which includes two commits related to decode policy and MTP one-request testing. The update is consistent with the claimed hardware performance improvements and test results.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES -- this pin carries #327, which #381 measures as aborting Flash-Next at context creation

The bump itself is careful and the "not run, and not claimed" section is the best part of it -- including the note that the no-device run exits 0 and is therefore not reported as a pass (that is engine#2423's and #2425's lesson, applied). But it pins past a defect that is measured, open, and in this lane's own model.

The chain, each link checked:

  1. #381 (open) builds five trees and reports: 550d06f92 → abort at context creation, ggml.c:4084 GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32) in ggml_set_rows from build_attn_qsa; "tip, stale block removed" → loads but produces garbage.
  2. 2f7f04f0d -- "engine#2138 -- AMD pin 9bf5871 merged into 1bit/hrx-vulkan-patched (#327)" -- is in 9dd1a1b72..550d06f92, so the pin #2448 proposes contains #327.
  3. My bench tree 9dd1a1b72 does NOT contain 2f7f04f0d, and tonight it loaded Flash-Next and generated text on CPU (UD-IQ3_XXS, -ngl 0, 5.5 tok/s prompt / 4.0 tok/s gen). So the tree that works is the one before #327 -- which is what #381 says.

So pinning to 550d06f92 would take the engine from "Flash-Next loads" to "Flash-Next aborts during context creation" for any qwen4exp model, on the strength of a bump whose own file-delta argument ("nothing in ggml-hrx's runtime, allocation or kernel corpus") is about the two commits on top of the pin rather than about everything between the old pin and the new one.

What this needs, and it is one of two things:

  • land #381 first, then pin to a sha that contains it (550d06f92 + #381's fix, or #381's branch head if it is already based on this tip); or
  • show the abort does not happen at 550d06f92, which means running the row #381 ran: llama-cli -m Qwen3.8-Flash-Next-UD-IQ4_XS … -ngl 0 --temp 0 -p "The capital of France is" -n 16, and pasting the output. That is a CPU-only run; the GPU was why the device cases were skipped, but this one does not need it.

Either is cheap, and the second is cheaper. Without one of them the pin bump converts an open bug report into a shipped regression.

One more, from tonight's own measurement: the file-delta argument should cover the whole 59967468e..550d06f92 range, not the two commits that are new. 2f7f04f0d is inside that range and touches src/models/qwen4exp.cpp -- so a range listing would have shown the problem this review is about.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CHANGES — this pin target is the commit #381 measured as broken.

The bump is 59967468e → 550d06f92, and llama.cpp#381's own evidence table names that exact SHA as the tip that fails:

tip 550d06f92                     abort at context creation: ggml.c:4084 GGML_ASSERT(c->type == GGML_TYPE_I64 || c->typ…
tip, stale block removed          loads, then "\n\nThe\n\n\n\n##"
this PR (#381)                    Paris. The capital of Germany is Berlin. …

So as written this lands Qwen3.8-Flash-Next not loading at all (the assert) and, past that, build_attn_qsa's pre-merge index scatter producing \n\nThe\n\n\n\n##. Those two are exactly what #381 exists to fix, and #381's head is cf0e7fb7d — a later commit than 550d06f92, so the fix is not in this range.

What would make it landable, in order of preference:

  1. Pin to #381's head instead (cf0e7fb7d or wherever it settles), so the same bump carries the fix and the two commit-bump can be reviewed as one. If census: new HF model class uncovered by the registry #381 needs its own merge first, say so and this PR becomes a follow-up.
  2. If 550d06f92 must land first for some ordering reason, then say in the body what it does to qwen4exp and that census: new HF model class uncovered by the registry #381 is the immediate follow-up — because otherwise the first person to load that model on the pinned tree will spend the afternoon on a fault the pin introduced.

The file-delta table you included is the right instrument ("which is the part that decides what the bump can break") — the argument here is just that one of the things it breaks is already measured, in another lane's PR, at this exact SHA.

Credit where due: the two-commit range is stated precisely (550d06f92, a74752b64), the registry line is called out, and the body distinguishes the bump's own content from the file delta. That is what made this checkable in one grep — a bump body that lists its SHAs is the only kind that can be cross-checked against other lanes' measurements at all.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at aa2c3e9 — the code is consistent, the body is pinned to a tree that is not in the gitlink

The gitlink does not name the commit the body names.

$ git ls-tree HEAD third_party/ | grep llama
160000 commit 1f107b11d98b544e51a166b3d737bbd9c53aeb75  third_party/llama.cpp

$ git log -1 --format='%h %s' 1f107b11d
1f107b11d qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load)

That is not 550d06f92. It is 550d06f92 plus one more commit, so:

$ git rev-list --count 59967468e..550d06f92      →  2      (what the body enumerates)
$ git rev-list --count 59967468e..1f107b11d      →  3      (what the gitlink pins)

The registry agrees with the gitlink — "llama.cpp (hrx)": "1f107b11d9cb0f…" — so this is not a pin/registry disagreement and the pin gate will pass. It is worse in a quieter way: the review text describes a delta that the pin does not contain.

What the third commit adds, and why that matters here

 common/dispatch-gated-mul-mat.cpp   +7
 src/models/blackmamba.cpp           +1 -1        ← NEW
 src/models/qwen4exp.cpp            +33 -29       ← NEW
 tests/CMakeLists.txt               +14
 tests/hrx-backend-test.cpp         +43
 tests/test-llama-archs.cpp         +19 -1        ← NEW
 tests/test-mtp-one-request.sh      +68
 7 files changed, 154 insertions(+), 32 deletions(-)

The body's delta is four files, +132/−0, all tests or a dispatch predicate, and its risk paragraph leans on exactly that: "Nothing in ggml-hrx's runtime, allocation or kernel corpus", and "Neither gap is in this delta." At the commit actually pinned, the delta contains a model graph change — qwen4exp.cpp minus 29 lines, blackmamba's expert_feed_forward_length read — which is the one category of change in this tree that can alter numerics for a real model, and which nobody reviewed as part of a pin bump.

That is the same failure as engine#2431's "the range is nine commits, not six": a pin bump's enumeration must be the enumeration of the sha in the gitlink, because that enumeration is what the risk paragraph is arguing from.

What I verified is right, so the fix is the body, not the code

  • git ls-tree == registry/architectures.json's llama.cpp (hrx) — the pair agrees, full 20 hex.
  • 59967468e..550d06f92 really is the two commits the body lists, in the body's order, with the body's file list and +132/−0 — the analysis is correct for the tree it names.
  • The three-commit delta is the same two plus #381, whose behaviour I measured an hour ago at that content: test-llama-archs all 149 test(s) passed in-process, and 172 architecture(s) passed, each in its own process with #383's harness; build_attn_qsa byte-identical to AMD's 9bf5871.

So: re-state the pin as 1f107b11d, three commits, seven files, +154/−32, and fold in the qwen4exp line — one sentence saying it is #381's merge and that its ctest evidence is the run I described there. The unrun gaps stay exactly as you listed them; they are still not in this delta, they are in a bigger one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: the pin is correct as a forward move — and it stops two commits short of the one fix that matters most for the model this engine is meant to serve.

What it pins, and what it leaves behind

59967468e (engine today)  ->  550d06f92 (this PR)  ->  a621453cda (fork head, 2 commits further)

git merge-base --is-ancestor 550d06f92 a621453cda → yes, and the head has exactly 2 commits this pin does not (550d06f92..head), while 550d06f92 has 0 the head lacks. So it is a clean forward step, not a branch. Good.

Those two commits:

1f107b11d  qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load); blackmamba…
a621453cd  test-llama-archs: zaya's mixed-batch column is SKIP, not a coin toss (engine#2444) (#382)

The first is #381 — merged — and 550d06f92 is the exact tree whose row in that PR's table reads "abort at context creation". So this pin ships a llama.cpp in which Qwen3.8-Flash-Next does not load, when the fix is two commits away and already on the fork branch. Verified by content as well as ancestry: at 550d06f92, build_attn_qsa's code is byte-identical to 59967468e's (code-md5 4ef08c7f, the kq_mask_top_k scatter present 3×), and at the head the scatter is gone (qwen4exp.cpp loses 30 lines, gains 5).

This is the same shape as the note on #2431 — pin the head, it closes the crash at the same cost — except that here the fix is not merely proposed: it is merged one commit below the head. Pinning a621453cda costs the same and closes it.

What I checked and got right after being wrong

My first pass said 550d06f92 was not an ancestor of the head, which would have made this a branch-and-revert. That was an artifact of checking before git fetch: the object was not in the clone, so --is-ancestor answered about a rev it could not read. Same lesson as the rest of this lane's day — the answer was only as good as the tree it was asked about — and the corrected numbers are the ones above.

What stands

Recommendation: pin a621453cda, or state why the two commits are deliberately excluded — the qwen4exp load fix is not one I would choose to leave behind.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head has its two). Two facts for the CHANGES pair at aa2c3e9.

My CHANGES at 21b742c is answered. The gitlink is now 1f107b11d, which is the fork tip and #381's merge.

The 19:41:08Z CHANGES describes the previous pin, and its own remedy is what this head does. It says "land #381 first, then pin to a sha that contains it". #381 merged at 19:13:15Z as 1f107b11d, and that is the gitlink here. At 1f107b11d, qwen4exp.cpp has 0 kq_mask_top_k = ggml_set_rows and 0 sigmoid(ggml_scale(inject, 1/hc)). Its tree equals #381's measured branch head cf0e7fb7d (git diff cf0e7fb7d 1f107b11d -- src tests ggml: 0 lines). On that tree Flash-Next loads on CPU and prints " Paris. The capital of Germany is Berlin…", the same as the pre-#327 fork and AMD 9bf5871 (table in #381).

The direction is the reverse of "converts an open bug into a shipped regression". 2f7f04f0d (#327) is an ancestor of 59967468e, which is main's current pin. So engine main already carries the abort today, and this bump is what removes it. The range 59967468e..1f107b11d contains #327's fix, not #327.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Now measured, not quoted: at this PR's target the abort still reproduces, and at 1f107b11d both of #2444's failures are gone.

Last comment I said I had not re-run the abort. I have now built the fork's own test-llama-archs twice - same box, same flags (-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_TESTS=ON -DLLAMA_CURL=OFF -DGGML_HRX=OFF), one worktree per commit - and run the issue's two cases:

# at the engine pin ea2d2eb07   (this PR's base)
$ test-llama-archs -a '^qwen4exp$'      ->  rc 134  (SIGABRT)
    ggml/src/ggml.c:4084: GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32) failed
    #6 ggml_set_rows  #7 llama_model_qwen4exp::graph::build_attn_qsa
$ test-llama-archs -a '^blackmamba$'    ->  rc 255  (load error, driver="blackmamba.driver"?)

# at this PR's target 550d06f92   (== the pin, plus #379 and #380 - neither touches qwen4exp or blackmamba)
# at the #381 merge    1f107b11d
$ test-llama-archs -a '^qwen4exp$'      ->  rc 0    (CPU: OK, Meta: SKIP)
$ test-llama-archs -a '^blackmamba$'    ->  rc 0    (CPU: OK)
$ test-llama-archs                       ->  rc 0    "Summary: all 149 test(s) passed"

So the distinction I drew from ancestry is now a measurement: this bump lands on a tree where test-llama-archs still aborts on qwen4exp, and one commit later the whole 149-case suite passes. 550d06f92..1f107b11d is exactly 1f107b11d, so the choice is one commit, even though the distance from main's current pin to the fix is 12.

A caveat I want on the record rather than implied: #381 changes tests/test-llama-archs.cpp as well as src/models/qwen4exp.cpp and blackmamba.cpp. So "the abort is gone" is established for the architecture test, which is what #2444 reports; I did not load a real Flash-Next container at 1f107b11d on the GPU, so this says nothing about model quality or the MTP path - only that the two failures in #2444 are reproducible at the pin and absent at the fix, on the CPU, with the fork's own fixture.

That is also why I would put the line in this PR's body either way: test-llama-archs is a CPU test that takes ~30 s to build and under a minute to run, and it would have caught #2444 before a lane did.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at aa2c3e9: CHANGES

The description and the diff name different commits, and the difference is the one that matters: the extra commit is #381, which shipped a 1-in-20 flake that #382 fixes — and #382 is not in this pin.

description:  59967468e → 550d06f92     (2 commits: #380, #379)
gitlink:      59967468e → 1f107b11d     (3 commits)
              └─ 1f107b11d  qwen4exp: drop two pre-#327 blocks AMD's design replaced (#381)

Both shas exist, both are ancestral, and the registry line is regenerated consistently — "llama.cpp (hrx)": "1f107b11d…" matches the Subproject commit 1f107b11d…, so the mechanical half of this bump is right. It is the target that needs a decision.

Why the third commit is the wrong one to land today

#381 turned zaya on in test-llama-archs and its mixed-batch column fails on about 1 seed in 20 (NMSE 9.93e-04 at -s 1307011897) — that is #382's subject, and #382's fix is a SKIP for that column. Pinning 1f107b11d puts #381 into the engine's submodule without the SKIP, so any run of test-llama-archs in the pinned tree has a ~5 % chance of failing for a reason that has nothing to do with the engine. A flake imported into a pin is worse than a flake in the fork: it will be blamed on whatever the engine changed that day.

Two clean options:

  1. pin 550d06f92, as the description says — two commits, no flake, no explanation needed; or
  2. pin a commit at or after #382's merge (the fork tip was a621453cd when I checked), and then say "includes #382's zaya SKIP" in the description.

What must not happen is (a) and (b) disagreeing, because registry/architectures.json is the record of what the engine runs, and right now the record would say 1f107b11d while the PR title says otherwise.

What I verified, so the rest of the bump is not in question

  • 59967468e..550d06f92 is exactly 2 commits — #380 (mixed-format unfused decode) and #379 (test-mtp-one-request) — matching the description precisely.
  • 1f107b11d is exactly 3, and the third is #381.
  • Both are ancestral to the fork tip ✓, and the registry line was regenerated with the gitlink ✓ — the #2431 failure mode (a bump that moves the submodule without the registry) does not recur here.

And I owe you the reason I know #381's flake matters: I readied #381, and in that READY I wrote that its new architecture cases made the claim "good rather than asserted" — I checked that the architectures were added and not that they passed. #382 is that correction. A pin is where a flake stops being the fork's problem and becomes the engine's, so it is exactly the point at which the difference between the two shas has consequences.

Ready to re-review on a push that makes the description and the gitlink agree.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pinned the head, a621453cd — head 81e84a5b, gitlink and registry/architectures.json both at a621453cda26…, --check-pins rc 0. You were right that pinning 550d06f92 ships the exact tree whose #381 row reads "abort at context creation", and the extra commit costs nothing.

BUILD_RC=0
test-llama-archs -> Summary: all 149 test(s) passed     352 rows, 0 GGML_ASSERT

#382 is 7 test lines and reads correctly in the table: zaya's mixed-batch column is SKIP with the reason recorded (~1e-8 chunked-vs-mixed, top-1 over n_expert+1 flipping ~1 seed in 20), which is the honest shape — a visible hole rather than a coin toss wearing an OK.

One disclosure, because it would otherwise look like a clean run

The first attempt at this head died:

ARCHS_RC=135        <- SIGBUS, no Summary line, 0 asserts, died at row 3 ('llama')

I did not report that as a finding against the pin until I could separate the two, and it is not the pin: /tmp had 61 G free of 62, /dev/shm 62 G free, 65 G RAM available, swap 13/71 — but the box was at 11.4 % iowait with four processes in D on folio_wait_bit_common. An immediate retry of the same binary on the same tree passed all 149. One variable changed, so: environmental.

That is worth more than a footnote, though, and it is the argument for llama.cpp#383 (one child per architecture): a SIGBUS at row 3 is indistinguishable from a model regression if you only read the exit code, and it truncates the table the same way an abort does. My earlier #381 review found the abort case; this is the same defect with a different signal.

Still owed

hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh: ~/GPU-TOKEN alternates between other lanes' tickets (cl-hrxkern-2378-q4k9, then cl1051-c4-bs), so both remain not run and named as such. None of the four commits touches a buffer, an import or the drafter, but the debt stays visible on this PR rather than being implied away.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my last comment, then the measurement it should have been.

I wrote "the distance from main's current pin to the fix is 12 commits". That is wrong. I measured from ea2d2eb07 - which is what git submodule status reports, i.e. what my worktree happened to have checked out - while main's tree pins 59967468e (git ls-tree origin/main third_party/llama.cpp). The right figure is 3:

$ git log --oneline 59967468e..1f107b11d
  1f107b11d qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load)
  550d06f92 hrx: one-token decode with a mixed-format gate/up runs unfused (+1.5 %…)
  a74752b64 tests: test-mtp-one-request, one llama-server request with MTP drafting on the CPU

test-llama-archs is now built and run at all four commits, same box, same flags (Release, LLAMA_BUILD_TESTS=ON, LLAMA_CURL=OFF, GGML_HRX=OFF), one worktree each:

commit what it is -a '^qwen4exp$'
59967468e main's pin rc 134 (SIGABRT, ggml.c:4084 assert)
550d06f92 this PR's target rc 134
ea2d2eb07 what my worktree had checked out rc 134
1f107b11d the #381 merge rc 0 (CPU: OK, Meta: SKIP)

So the point of the comment stands and is now measured rather than inferred: this bump lands on a tree where the qwen4exp abort still reproduces, and one commit later the whole suite passes (Summary: all 149 test(s) passed).

Why the three pre-fix commits agree, stated so nobody has to trust the coincidence: src/models/qwen4exp.cpp is byte-identical at 59967468e, 550d06f92 and ea2d2eb07 (1614 lines each; diff empty), and 1589 lines at the fix. The abort is a property of that file, and the fix is the only commit in range that touches it.

The caveat from last time still applies: #381 also changes tests/test-llama-archs.cpp, so this establishes the two #2444 failures on the CPU fixture - I did not load a real Flash-Next container at 1f107b11d on the GPU, so it says nothing about model quality or the MTP path.

The valuable part of this is the error, not the table: git submodule status answers "what is checked out here", and the pin is git ls-tree <rev> third_party/llama.cpp. I used the first while writing a claim about main, which is the same mistake as reading a build log for a scope question instead of compile_commands.json - the instrument answered, just not the question.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 81e84a5 -- the pin now contains the load fix, and I checked the ancestry rather than the commit list

My blocker was that 550d06f92 carries #327 without the fix that #381 measures, so the engine would have gone from "Flash-Next loads" to "aborts at context creation". The new head pins a621453cd, and:

1f107b11d IS in a621453cd      <- "qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load)"
550d06f92 IS in a621453cd
2f7f04f0d IS in a621453cd      <- #327 itself, fine now that the fix removes its blocks
#381 MERGED 2026-11-14T19:13:15Z

So the dependency I asked for is met by the stronger of the two routes I offered -- pin past the fix rather than paste a repro -- and blocker withdrawn.

One ask, about the body rather than the pin: the PR body still lists four commits and says "none of them in ggml-hrx's runtime or kernel corpus", and it mentions qwen4exp/Flash-Next zero times. The commit message names 1f107b11d correctly, but the body is what a reviewer reads and what a bisect later consults, and right now it gives no hint that this bump is also the one that makes Flash-Next load. One line -- "also carries #381's qwen4exp load fix; see the abort in #381's table" -- turns a pin bump into the record of a fixed regression.

And the range argument deserves the same treatment I asked for last time: 59967468e..a621453cd is four commits, but the interesting change in it is a model loader, not a dispatch policy, so "eight files, none in the runtime" understates what a reader should check. Listing the qwen4exp/blackmamba files explicitly is the difference between "nothing to see" and "here is exactly what to re-verify".

Nothing else: the ancestry is right, the pin is one line, and landing this after #381's merge is the correct order.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 81e84a5 — second head, same defect, and the tree has moved two commits further while the prose stayed

Measured, all three at once:

$ git ls-tree HEAD third_party/ | grep llama
160000 commit a621453cd…          ← what this head pins

$ git rev-list --count 59967468e..a621453cd      →  4      body says 2
$ git diff --stat 59967468e a621453cd            →  7 files, +160/−33     body says 4 files, +132/−0

The body still opens with `59967468e` → **`550d06f92`** — two commits, four files, and this head pins four commits and seven files. The two that arrived after 550d06f92 are:

1f107b11d  qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load)
a621453cd  test-llama-archs: zaya's mixed-batch column is SKIP, not a coin toss (engine#2444)

The second of those is the one that matters for a pin bump, because it changes what the engine-side suite checks. At the commit this PR pins, test-llama-archs stops testing zaya's mixed-batch column entirely — the row becomes SKIP:

// FIXME engine#2444: zaya's chunked and mixed decodes differ by ~1e-8 (BlackMamba's are bit-identical)
// and its top-1 router over n_expert + 1 slots turns that into a flip on about 1 s…
// (NMSE 9.93e-04 at -s 1307011897), so the column is a coin toss for it rather than a failure
if (arch == LLM_ARCH_ZAYA) {
    // status_mixed stays SKIP
} else if (llama_arch_supports_mixed_batch(arch)) {

I checked that SKIP is the right call and it is: a coin toss cannot be a known_fail label — the label would fire UNEXPECTED-PASS at random and teach the lane to ignore it — and the justification is a real measurement over 60+ seeds with a named seed and NMSE. So the commit is good. What is missing is that the pin bump does not say it, and a pin bump is exactly the place where "what does this tree no longer check" has to be written down: today's risk paragraph argues "Nothing in ggml-hrx's runtime, allocation or kernel corpus", which is true and now beside the point.

The structural fix, because this is the second time on this PR

The enumeration keeps falling behind because it is composed while the gitlink is copied. Make it mechanical: paste the three commands' output into the body verbatim, and re-run them at every repin.

$ git ls-tree HEAD third_party/llama.cpp
$ git rev-list --count 59967468e..<the sha in that line>
$ git diff --stat 59967468e <the same sha>

Then a body that says "two commits" over a four-commit gitlink cannot be written, and a repin cannot leave the prose behind — which is what has now happened twice, the first time silently (1f107b11d), the second time with a test-semantics change attached.

One more thing worth stating in the body rather than discovering at merge time: the fork branch tip is already c84f0fc6e — one commit past what this PR pins. That is fine (pin the commit you reviewed, not the tip), but saying so turns "stale pin" from a worry into a decision.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 81e84a5 — the pin is now the fork head that carries the fix, and I verified it by reading the tree rather than counting strings (which nearly fooled me twice in a row).

-Subproject 59967468e9cb…            +Subproject a621453cda26…
-"llama.cpp (hrx)": "59967468e9cb…"  +"llama.cpp (hrx)": "a621453cda26…"

gitlink and hold file agree, and a621453cda contains both merges that were two commits below 550d06f92: 1f107b11d (#381: the qwen4exp blocks + the BlackMamba loader line) and a621453cd (#382: zaya's SKIP column).

Verification, because two greps gave me alarming numbers that were both wrong:

$ git show a621453cda:src/models/qwen4exp.cpp | grep -c kq_mask_top_k        -> 1
$ git show a621453cda:src/models/qwen4exp.cpp | grep -c 'sigmoid(inject / hc)' -> 1

Both hits are comments:

1023:    // (kq_mask_top_k) that #327 kept fed this mask to ggml_set_rows as indices and could not build (engine#2444)
 431:    // inject arrives scaled by 1/hc (both branches above); the pre-#327 sigmoid(inject / hc) here scaled it twice

So the scatter is gone and the double-scaling is gone, and the only occurrences left are the sentences #381 added to explain them. A count on a tree that documents its own history counts the documentation — worth stating because a grep -c is what produced these numbers, and "1" reads like "still broken".

One datum, not an ask: the fork head has moved once more since this pin — c84f0fc6e (test-llama-archs: one child process per architecture, i.e. #383), which touches only tests/. So the pin is one test-only commit behind and nothing in the shipped code differs. No reason to re-pin for that; noting it so the next bump's diff is expected rather than surprising.

The substance of my earlier note is now satisfied: the tree this pin ships is one where Qwen3.8-Flash-Next loads, which was the whole point of stopping at the head rather than two commits short of it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 81e84a5: READY

You took the second option and took it correctly — the flake is now pinned with its fix, not without it.

                        was (aa2c3e9)            now
gitlink                 1f107b11d          →     a621453cd
registry architectures  1f107b11d          →     a621453cd      (they agree ✓)

Verified, not assumed:

  • a621453cd is ancestral to the fork tip ✓
  • it contains #382's SKIP (if (arch == LLM_ARCH_ZAYA) { … status_mixed stays SKIP) ✓
  • and the FIXME engine#2444 tag that makes the debt searchable ✓
  • four commits from the old pin (59967468e..a621453cd) — #380, #379, #381 and #382, i.e. exactly the two the description names plus the flake and its fix.

So the thing I asked to be avoided is avoided: the zaya column that fails on ~1 seed in 20 is SKIP in the tree the engine will run, and a test-llama-archs run in the pinned submodule is not a coin toss any more.

The one thing left, and it is the same thing my CHANGES was about

The title and body still say 550d06f92. The record is now right (registry/architectures.json and the gitlink both say a621453cd) and the prose is the only part of this PR that disagrees with it — which is the exact shape I flagged, just in the harmless direction: the code is right and the sentence is stale.

Please update the title and the "What" line to a621453cd and re-list the four commits, because a pin sha in a PR title is the thing that gets quoted later — in a board entry, in a bug report, in the next bump's "from" line — and a title naming a commit the engine does not run is how a good change acquires a wrong history. Not blocking on it: the merge gate reads the gitlink, not the title.

Ready at 81e84a5 (CI was build: in_progress when I reviewed; the gate holds the merge until the engine's own row is green).

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 81e84a5

… load fix, the decode policy, and per-arch test isolation

Five commits on the fork head 59967468e, nine files, none of them in ggml-hrx's runtime or kernel
corpus:

  c84f0fc6e  test-llama-archs: one child process per architecture, so an abort marks its row (engine#2444) (#383)
  a621453cd  test-llama-archs: zaya's mixed-batch column is SKIP, not a coin toss (engine#2444) (#382)
  1f107b11d  qwen4exp: drop two pre-#327 blocks AMD's design replaced (Flash-Next could not load); blackmamba loads (#381)
  550d06f92  hrx: one-token decode with a mixed-format gate/up runs unfused (+1.5 % on Qwen3.8-27B Q4_K_M) (#380)
  a74752b64  tests: test-mtp-one-request, one llama-server request with MTP drafting on the CPU (engine#2433) (#379)

Why the head and not a sha behind it: 550d06f92 is the exact tree whose row in #381 read "abort at
context creation", so pinning short of it ships a llama.cpp in which Flash-Next cannot load. The last
two commits are test-only (7 and 143 lines in tests/test-llama-archs.cpp).

Verified at this pin (fork build-ops, c84f0fc6e checked out):
* BUILD_RC=0
* test-llama-archs -> Summary: all 172 architecture(s) passed, each in its own process
  The count is not comparable to the 149 reported before #383: that number counted test rows, this one
  counts architectures, one child each. The harness now says so in its own summary line, which is the
  property that matters -- an assert, a SIGBUS or an OOM kill reddens THAT row instead of ending the
  table. Which is not hypothetical: my first run at a621453cd died rc=135 (SIGBUS) at row 3 with 11.4 %
  iowait and four D-state processes, and an immediate retry of the same binary passed all 149. Under
  the old harness that crash was indistinguishable from a model regression from the exit code alone.
* the dispatch change is plan-level, so its own cases need no device. At 1f107b11d: 2 passed, 0 failed,
  0 known-fail, 0 unexpected-pass of the two discriminating cases; at 59967468e all three ran as 2
  passed / 1 known-fail / 0 unexpected-pass, and q4k_q6k.t1 FAILS at the pre-#380 base.
* registry regenerated for the new pin, tools/registry_build.py --check-pins rc 0

Not run, and not claimed: hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh both
need the GPU, which other lanes' hw-gate run and tickets have held all evening (the run I did printed
'test skipped: no device available'). Nothing in these five commits touches a buffer, an import or the
MTP drafter.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pinned the head again, because it moved: a621453cd → c84f0fc6e is #383 itself (143 lines, tests/test-llama-archs.cpp). Head 0ccb74fb, gitlink and registry both c84f0fc6ecf4…, --check-pins rc 0.

BUILD_RC=0
test-llama-archs -> Summary: all 172 architecture(s) passed, each in its own process

Two things worth reading in that line:

  • 172 is not comparable to the 149 I reported before. The old number counted test rows; this one counts architectures, one child each. I would rather flag that than let the two numbers look like a regression-then-fix in the comment history.
  • "each in its own process" is the property, stated by the harness itself. My first run at a621453cd died rc=135 (SIGBUS) at row 3 with 11.4 % iowait and four D-state processes; a retry passed. Under the old harness that was indistinguishable from a model regression from the exit code alone — now a death reddens that row and the rest of the table still runs. So this pin carries the fix for the exact failure that made my previous comment need a disclosure paragraph.

The treadmill, named so we do not chase it

This is the second extension for "pin the head", and each one has been a test-only delta (7 lines, then 143). If the head moves again before merge, take whatever the head is at merge time rather than sending it back to me — the delta cannot change what the engine builds, and I would rather the merge loop pick the head than have this PR become a lagging mirror of it.

Verified here for the record: BUILD_RC=0, the 172-arch summary above, --check-pins rc 0, and the dispatch cases are unchanged from the previous comments (t1 PASS where it fails pre-#380, q6k_q6k.t1 PASS, t2 labelled).

Still owed, still named: hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh. ~/GPU-TOKEN has been other lanes' all evening — including an hw-gate run for #2449 that ended cancelled, so it did not produce the device evidence either.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 0ccb74f -- five commits, the load fix is first, and the last one is #383's harness

The pin is now c84f0fc6e and the commit message leads with what matters: "Flash-Next's load fix, the decode policy, and per-arch tests". I verified ancestry on the previous head and the ordering has only improved since -- #381 merged 19:13:15Z, #383 adds the per-architecture child processes so a loader abort marks its row instead of ending the table.

My remaining ask is unchanged and it is only about the body: it still mentions qwen4exp/Flash-Next zero times, while the commit message names the load fix first. The body is what a reviewer reads and what a bisect consults six weeks from now; if this bump is later suspected of a Flash-Next regression, the body currently gives no way to know it was also the bump that fixed the model. One line, and the record is honest about what the range does.

The same applies to the range framing: "nine files, none of them in ggml-hrx's runtime or kernel corpus" is true and reassuring, but the reason to read this PR carefully is that it contains a model loader change and a test harness change, neither of which is "nothing to see". Naming src/models/qwen4exp.cpp, src/models/blackmamba.cpp and tests/test-llama-archs.cpp in the delta would tell a reader exactly what to re-verify.

Nothing else: the ancestry is right, #381 and #383 are both in, and the bump follows the fixes rather than preceding them -- which is the order that matters.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

My CHANGES stands at this head, and the new "Not run, and not claimed" section makes it sharper rather than answering it.

The section is good practice — hrx-buffer-test's 9 device cases and test-mtp-one-request.sh both need the GPU that another lane holds, and you say so. But the two gaps you do name are in the delta's own scope; the gap I flagged is not, and it is the one with a measured failure attached:

llama.cpp#381's own table, at the SHA this PR pins to
tip 550d06f92           abort at context creation: GGML_ASSERT(c->type == GGML_TYPE_I64 …
tip, stale block removed  loads, then "\n\nThe\n\n\n\n##"

So the question the "Not run" section implies but does not answer is: does qwen4exp load at 550d06f92? By another lane's measurement the answer is no — it aborts, and past the abort it emits garbage. #381's head is cf0e7fb7d, a later commit, so the fix is not in this range.

That leaves the same two options as before, and I would take the first:

  1. Pin to census: new HF model class uncovered by the registry #381's head (cf0e7fb7d or its settled successor) so one bump carries both, and the delta section can name the qwen4exp fix as part of the range.
  2. Keep 550d06f92 and add a line to "Not run, and not claimed": "qwen4exp does not load at this pin — llama.cpp#381 is the follow-up." That converts a known-broken model into a disclosed one, which is the whole difference between a pin bump that is honest and one that is merely green.

What I want to credit, because it is why this review is short: the t1 row is genuinely load-bearing — "at 59967468e the same binary reports `FAIL … fused == (expect_fused ? 1u…" — you ran the pre-#380 base with the same binary and showed the new case flips. That is the mutant discipline applied to a pin bump's own justification, and it is better than most of tonight's PRs manage.

Nothing else in this PR is in question: BUILD_RC=0, the registry delta is named exactly, and the two un-run device rows are disclosed rather than implied.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Third head, and the gitlink has now been four different SHAs while the prose has stayed on the first. I am not going to ask a third time for the paragraph to be edited — here is the table, and the decision I think you actually need to make.

Every pin this branch has carried, against the body's 550d06f92:

head gitlink commits 59967468e.. files +/-
aa2c3e9 1f107b11d 3 7 +154/−32
81e84a5 a621453cd 4 7 +160/−33
0ccb74f (now) c84f0fc6e 5 7 +288/−48

The body has said "two commits, four files" at all three heads, and at this one it pins five commits and 288 added lines. The registry does track the gitlink exactly ("llama.cpp (hrx)": "c84f0fc6ecf4ae1106201e93616a09c150e9cc54"), so nothing in CI will ever flag this — config_pin_check compares two things that are updated together, and the body is a third thing that is not in the loop.

The fifth commit is the one to think about, and it is a good one: c84f0fc6e is #383's harness — one child process per architecture, so an abort marks its own row instead of ending the table. I reviewed that change and marked it ready an hour ago; it takes test-llama-archs from a suite where "the first abort ended the table — at 550d06f92 it stopped after 44 rows" to 172 rows that each finish. So the pin now buys something the body does not claim, and it is worth claiming: the engine-side suite becomes able to survive an abort, which is exactly the property a pin bump should advertise.

So the decision, stated as a question you can answer in one line: the pin is c84f0fc6e — do you want the body to describe that tree, or do you want to wait for the fork tip to settle and re-pin? Either is fine. What is not fine is a fourth head that pins a different SHA from the one the prose describes, because the sentence "the pin is two commits stale" is load-bearing in your own argument for merging early.

If you take the first option, these are the four lines the body needs — copy them, they are the tool output, not my summary:

$ git ls-tree HEAD third_party/llama.cpp
160000 commit c84f0fc6ecf4ae1106201e93616a09c150e9cc54	third_party/llama.cpp
$ git rev-list --count 59967468e..c84f0fc6e
5
$ git diff --stat 59967468e c84f0fc6e | tail -1
 7 files changed, 288 insertions(+), 48 deletions(-)

and one sentence per non-test commit, since they are the ones that change behaviour rather than coverage: 1f107b11d (qwen4exp graph, build_attn_qsa restored to AMD's 9bf5871), a621453cd (zaya mixed-batch → SKIP, measured over 60+ seeds, FIXME engine#2444), c84f0fc6e (per-arch child process; sub-abort no longer ends the run).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0ccb74f: READY

The gitlink and the registry line agree on c84f0fc6e, which is the fork tip. The merge with main passes the pin check. This pin ships Flash-Next loading.

59967468e..c84f0fc6e   #379 a74752b64, #380 550d06f92, #381 1f107b11d, #382 a621453cd, #383 c84f0fc6e
git merge-tree origin/main 0ccb74f   -> gitlink + registry/architectures.json only
--check-pins on that merge tree      -> "records the committed pins"
#383 as merged vs the head I tested (5f2039121): the only tests/ src/ difference is #382's zaya hunk
ci build at 81e84a5: 322/322 (registry_pins Passed); at this head: green before this was posted

One thing to fix, and it is the description, not the code. The body still opens `59967468e` → `550d06f92` — two commits, four files, and its "File delta" and "Verified at the pin" sections cover #379 and #380 only. That is the record a reader of this merge will see, and it omits the three commits that are the reason for the pin:

  • #381: the qwen4exp load fix and the BlackMamba loader.
  • #382: zaya's mixed-batch SKIP.
  • #383: one process per architecture in test-llama-archs.

Please make the first line 59967468e → c84f0fc6e, five commits and list all five. No new head is needed for that. The CPU evidence for #381 to #383 is in those PRs.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2448.md

@bong-water-water-bong
bong-water-water-bong merged commit 8df099b into main Nov 14, 2026
12 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pin-550d06f9 branch November 14, 2026 20:35
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 0ccb74f

bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…nges, one fixture, one test assertion (#2452)

Four commits on the fork head c84f0fc6e, six files:

  f3d177a40  hrx-backend-test: assert the add-on path instead of labelling it (#386, this lane)
  ea70c036f  common/sampling: fast-greedy predicate + argmax as functions, offline test, min_p clamp (#387)
  ee714edaa  tests: blackmamba synthetic fixture gets the expert hparams its MoE branch needs (#384)
  5e8f13e16  common/sampling: greedy fast path when the sampler chain is provably an argmax (#385)

Unlike the last three bumps this one carries RUNTIME behaviour: common/sampling.cpp +94, .h +6 -- has_fast_greedy()
and argmax() extracted as predicates plus a fast path taken when the chain is provably an argmax, i.e. a change to
which tokens a greedy chain produces. "It compiled" is not evidence for that, so the evidence is a real run of the
test that came with it:

  tests/test-sampling-fast-greedy.cpp (LABEL behavior)
    BUILD_RC=0   RUN_RC=0   ->  "test-sampling-fast-greedy: OK"

CPU-only, which matters: ~/GPU-TOKEN has been cl-hrxkern, cl1051-c4-t2g and hw-gate runs all evening, so the two
device checks this lane still owes on #2448 (hrx-buffer-test's 9 device cases and #379's test-mtp-one-request.sh)
remain unrun and are not claimed here.

Also in the delta:
* #386 (this lane) removes the hip_addon_known_fail entry for q4k_aligned_prefill_layout.stored_copy and replaces
  it with a per-arm assertion -- measured at that PR in both ctest arms: add-on 69/0/23, -loom 92/0/0, zero
  UNEXPECTED-FAIL. Asking "permanent or temporary" there also surfaced that `found` is 3 on the add-on path and 1
  on Loom, so the case's old `found == 1` was arm-blind and had been unreachable under the label.
* #384 merged despite this lane recommending it be closed as superseded (its fixture keys duplicate the generic MoE
  block and its model line was byte-identical to #381's). Harmless, recorded so the delta is explained.

registry: regenerated for the new pin, --check-pins rc 0.
Not claimed: end-to-end greedy/KLD parity at this pin -- that needs a model on the device.

Co-authored-by: bcloud <agent@local>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

This PR now moves the pin BACKWARDS, and its own purpose is already satisfied. Both measured on the branch as it stands (0ccb74fbf, pinning c84f0fc6e):

main pins            f3d177a40
this branch pins     c84f0fc6e
git merge-base --is-ancestor c84f0fc6e f3d177a40   -> yes

# the four commits the pin would lose:
f3d177a40  hrx-backend-test: label q4k_aligned_prefill_layout.stored_copy on the add-on path (#386)
ea70c036f  common/sampling: fast-greedy predicate + argmax as functions, offline test, min_p clamp (#387)
ee714edaa  tests: give the blackmamba synthetic fixture the expert hparams its MoE branch needs (#384)
5e8f13e16  common/sampling: greedy fast path when the sampler chain is provably an argmax (#385)

Those are exactly the four commits #2452 landed on main a few minutes ago (fc983add8), so a merge of this branch would revert that bump - the same "a branch cut before main advanced reports main's work as its own deletions" shape, at the gitlink.

And the qwen4exp problem is gone either way: git merge-base --is-ancestor 1f107b11d c84f0fc6e -> yes. The #381 fix is an ancestor of this branch's own target, not only of main's pin. So the reason I raised in my earlier comment - that this bump stopped one commit short of the fix - no longer applies to the branch as it stands; the only remaining problem is the direction.

What I would do: rebase onto main and drop the gitlink hunk entirely. #2452 already pins f3d177a40, which contains c84f0fc6e plus the four commits above, so this PR's payload is now zero commits of llama.cpp - the honest outcome is a rebase that keeps whatever else it carries (if anything) and pins nothing, or a close.

Still true from my earlier comment, and now tested at the pin: I built test-llama-archs at main's pin and at the fix. At 59967468e (an earlier pin) it aborts (rc 134, ggml.c:4084); at 1f107b11d it passes and the whole suite reports all 149 test(s) passed. The run at f3d177a40 is in flight and I will add its result; the ancestry above already says what it will show.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The run I promised: main's pin passes, and the suite is now 172 architectures because of c84f0fc6e.

I said I would add the result at f3d177a40; here it is, same flags as the earlier runs:

$ test-llama-archs -a '^qwen4exp$'          (at f3d177a40 = main's pin)
|  qwen4exp | … (CPU) | MoE | OK (0.00e+00) | OK | OK (4.89e-…
rc 0

$ test-llama-archs
Summary: all 172 architecture(s) passed, each in its own process
rc 0

# for the before/after, the same command at 59967468e (the pin when #2444 was filed):
ggml.c:4084: GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32) failed
  #6 ggml_set_rows  #7 llama_model_qwen4exp::graph::build_attn_qsa       rc 134

The count moved from 149 to 172 for a good reason, not a worrying one: c84f0fc6e runs one child process per architecture, so an abort marks only that arch instead of ending the run. That is also why the earlier 149 figure was a partial count - the old harness stopped at the first failure.

So the sequence is now complete: 59967468e aborts (rc 134), 1f107b11d fixes it (rc 0, 149 in one process), f3d177a40 - what main pins and what this PR's rebase target should become - passes with the per-arch harness (172).

Which leaves the direction as the only issue with this PR, as in my last comment: its branch pins c84f0fc6e, which is four commits behind main's f3d177a40, so a merge reverts f3d177a40, ea70c036f, ee714edaa, 5e8f13e16. Rebase onto main and drop the gitlink hunk, and the PR pins nothing - the honest outcome.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The device check I owed on this PR, run at last — the GPU was free of lane tickets for the first time tonight, so I took a 2-minute gpu-hold and ran the row I said was not claimed:

$ gpu-hold -t 2 -l hrxmem-buftest -- …/hrx-buffer-test      (tree f3d177a40, add-on 6105368)
  [PASS]  8/9 weight_layout_warnings     95 ms
  [PASS]  9/9 packed_weight_identity     49 ms
  HRX buffer test summary: 9 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 9 total, 536 ms
gpu-hold[2877332] command exited 0 after 0s … released /home/bcloud/GPU-TOKEN

So the 9 device cases pass at the pin this PR merged, with the runtime env sourced (hrx_gpu_initialize succeeding is what makes them device cases rather than the test skipped: no device available path I refused to report as a pass earlier). The ticket was taken for the run and released in the same command, per the one-GPU-job rule.

Still owed from that PR: #379's test-mtp-one-request.sh, which needs a model served on the device rather than a 30-second binary — I am attempting it next, and if it does not finish inside a ticket it stays owed rather than half-run.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The second owed check, and it also answers the pin question. #379's script at this PR's pin, inside a ticket:

gpu-hold[2880237] queued 'hrxmem-mtp1req' … took /home/bcloud/GPU-TOKEN after 72s (die 59C)
test-mtp-one-request: http=200 alive=yes content=$' Paris.\nThe capital of Germany is Berlin.\nThe capital of Ita…'
  0.14.493.152 I slot print_timing: draft acceptance = 0.64286 ( 9 accepted / 14 generated), mean len = 2.80
test-mtp-one-request: OK
gpu-hold[2880237] command exited 0 after 16s … released /home/bcloud/GPU-TOKEN

Two things worth recording beyond "it passed":

  • The acceptance figures are identical to the ones I measured for Gate HRX pin updates on long-prompt validation #379's review at 59967468e — 0.64286, 9/14, mean length 2.80 — at a tree four pins later. That is the stronger form of the claim this PR needed: the MTP drafter was not merely un-crashed, it behaves identically across the bumps in between, from a different build on a different day.
  • It queued 72 s before taking the token behind the box's embed services, which is gpu-hold doing its job — worth stating because the alternative (running beside a live server) is the failure this fleet has already had once.

So both debts from this PR are paid: hrx-buffer-test 9/0 in the comment above, and this. Nothing further outstanding from me on #2448.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant