Skip to content

third_party/llama.cpp: pin to 148e7f329 (AVX2 fast-greedy argmax #398/#399, collapsible M-RoPE claim #401 for 27B --dflash, #400 #402) - #2583

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/llama-pin-148e7f3
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/llama-pin-148e7f3

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Moves third_party/llama.cpp from 8aa770fbd to 148e7f329 (fork 1bit/hrx-vulkan-patched head), with the registry "llama.cpp (hrx)" line matching. Two lines, the same shape as #2526.

What the pin brings

fork PR change
#398 an AVX2 body for the fast-greedy argmax (runtime __builtin_cpu_supports; scalar elsewhere). In-server argmax per call: 0.35 → 0.044 ms on a 248k vocab; greedy text identical
#399 its tests: tail lengths, boundary ties, a -inf ban block, and the scalar body run on AVX2 hosts
#401 HRX claims a collapsible M-RoPE as NEOX, so the 27B DFlash2 draft loads under strict placement. This carries cl-hrxkern's prepared pin. a8c0dc16a was their engine commit bumping to fc67612b1, which they could not push; it is superseded by this PR, so that sha appears in neither range. #401's own merge on the fork is fc67612b1, in this range. Their numbers: test-backend-ops ROPE 174/174. Under strict placement (1bit serve, GGML_HRX_STRICT=1), the 27B DFlash2 arm does not load on main and runs 49.9 / 27.4 / 18.2 tok/s with #401 (27B-REMEASURE, 11-15 12:08Z)
#400 a loader-mapped run without an HRX device fails instead of skipping
#402 docs: an HRX test with no device fails; 77 is only for what cannot be judged

tools/check_pins.py origin/main: ok third_party/llama.cpp: 8aa770fbd -> 148e7f329 is ahead, rc 0.

The pinned tree's own suite

Engine main + fork 148e7f329 + gk 60b8eef (main's pin), built clean, one gpu-hold ticket (cl1051-llpin148-suite):

  • fork ctest: 8/111 fail, the same 8 names that fail at the current pin (test-tokenizers-ggml-vocabs, test-kv-zero, test-llama-archs, test-llama-context, hrx-backend-test-loom, test-hrx-fa-transposed-value, test-hrx-mul-mat-id-k32, test-hrx-fa-masked-v). test-hrx-loader-mapped now registers as Skipped without a model (NPU fast lane: attention reads stale KV rows from the previous request (nondeterministic output) #400's convention).
  • test-hrx-loader-mapped with a model (Qwen3-0.6B-Q4_K_M.gguf): PASS, "2 weight buffer lines, 1 HRX0_Mapped, 0 failure(s)".
  • test-sampling-fast-greedy (tests: hrx_long_prompt turns thinking off through chat_template_kwargs #399's arms): OK.
  • engine ctest (NPU tests excluded): 3/256 fail, the same three as at the current pin: laya_gguf_route (known), token_rate_floor_device (Timeout: it takes its own gpu-hold ticket behind the suite's), gpu_hold_child_identity (inherits the suite ticket's env).
  • hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn' hangs intermittently, and it does so at the CURRENT pin too: this pin does not cause it.
    • In the suite run it hung at start (rc 124 at 300 s). A 3× rerun at this pin gave 18/18, hang after 16/18 PASS, 18/18.
    • Control (one ticket, binary snapshots of both pins, ldd-checked that each resolves its own libggml-hrx, interleaved, 120 s timeout): old pin 8aa770fbd: hung 4 of 6; new pin 148e7f329: 18/18 in 6 of 6.
    • No amdgpu/xdna kernel line in any run. The flake is pre-existing (a worker-process hang; no device fault) and is noted separately for the HRX lanes.
    • Nothing in this range touches hrx-ops-test, its worker pool, or the gdn/ssm_conv paths: the HRX diff is the header-only collapsible-M-RoPE claim (HRX opens the NPU device (/dev/accel/accel0) in GPU-only processes #401) plus a README.

🤖 Generated with Claude Code

…399, collapsible M-RoPE claim #401, #400 #402)

Carries cl-hrxkern's prepared pin (#401: the 27B --dflash draft loads under strict placement) to the fork head.
registry "llama.cpp (hrx)" matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

401 - Partially compliant

Compliant requirements:

  • The PR updates the llama.cpp pin to include changes that address the HRX backend behavior

Non-compliant requirements:

  • The PR does not directly implement the logic to prevent opening the NPU device in GPU-only processes; this requires changes in the engine's device selection logic

Requires further human verification:

  • Need to verify that the updated llama.cpp fork indeed prevents opening the NPU device in GPU-only processes

400 - Partially compliant

Compliant requirements:

  • The PR includes changes from llama.cpp that address the KV buffer reuse issue

Non-compliant requirements:

  • The PR does not include explicit code changes to zero K rows or implement the masking approach in the attention control code

Requires further human verification:

  • Need to verify that the attention masking or zero-filling logic is correctly implemented in the updated llama.cpp fork

402 - Partially compliant

Compliant requirements:

  • The PR updates the registry/architectures.json file to include the new architecture class

Non-compliant requirements:

  • The PR does not include documentation updates in registry/significant.json and docs/arch-gaps.md as required

Requires further human verification:

  • Need to verify that the architecture class is properly documented in registry/significant.json and docs/arch-gaps.md
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Documentation for New Architecture

The PR adds support for the HrmTextForCausalLM architecture class but does not update the documentation files (registry/significant.json and docs/arch-gaps.md) to explain why this architecture is not an alias and how it is handled by the engine.

"llama.cpp (hrx)": "148e7f329ff9b9dd3d7503ae4da5f53c6db604aa"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at ca1a143: READY — ratchet clean, gitlink and registry agree, five commits as advertised — and one statement in the documentation this pin carries is false, which I measured earlier today and can point at a line number now.

The gate, at the sha main would pin

hrx_f16_narrowing --check --root <fork @148e7f329>            rc 0
FINGERPRINT  b9b121a5ddc51854 (326 .loom file(s))
84 narrowing bodies, none new, none stale; 22 f16 accumulator bodies, none new, none stale

Unchanged from 8aa770fbd — as it should be: #401 touches qwen4exp-hrx-support.h, rope-claim-params.h and tests, none of which is corpus. And third_party/llama.cpp = registry/architectures.json = 148e7f329ff9, both lines present, which is the shape #2526 established.

The five commits in 8aa770fbd..148e7f329 are exactly the table: #398 AVX2 argmax, #399 its tests, #401 the collapsible M-RoPE claim, #400 the loader-guard test, #402 the docs.

On #401's provenance: the body says it is cl-hrxkern's prepared pin "their a8c0dc16a, which they could not push". That sha does not appear in the range — the content lands as fc67612b1, because it was re-applied rather than pushed. Worth one clause in the body, or a future reader who greps for a8c0dc16a will conclude the claim was never verified. (Same shape as #2526, where a lane's push was refused and the orchestrator carried it.)

The false statement in ggml/src/ggml-hrx/README.md at this pin

:92  | `hrx-*-test` | `hrx-buffer-test`, `hrx-backend-test`, `hrx-loom-jit-test` | **Failed.** Each prints
     `test skipped: no device available for <suite>` and returns **0** …
:98  **One row used to be the exception and is now fixed rather than recorded**: `test-hrx-external-window-view` …

hrx-backend-test is a second exception and it is neither fixed nor recorded:

tests/hrx-backend-test.cpp:16264   std::printf("device: skipped, no HRX devices available\n");   // not the family string
                                   return failed == 0 ? 0 : 1;                                  // 0 unless a NAME was passed
tests/CMakeLists.txt:696           FAIL_REGULAR_EXPRESSION "test skipped: no device available"  // matches the family string only
$ printf 'device: skipped, no HRX devices available\n' | grep -cE 'test skipped: no device available'   → 0

So on a device-less runner that row passes: the exact condition the section exists to say cannot happen, asserted as impossible by the row that names it. :98's sentence — "one row used to be the exception and is now fixed" — is true of test-hrx-external-window-view and silent about this one, which makes it read as "the exceptions are accounted for".

Two ways to fix, either one line:

  • make the branch print the family's string (test skipped: no device available) — then :92 becomes true for all three and the regex fails the row as intended; or
  • take hrx-backend-test out of the Failed row and give it its own row: "Passed today — known exception", with hrx-backend-test.cpp:16264 named.

I would take the first: the second documents a row that is green when the hardware is missing, and the whole point of the section is that such a row must not exist.

Not a block on the pin — the code this pin brings is the AVX2 argmax and the M-RoPE claim, both of which I reviewed on their own PRs and both of which are fine. But it should not merge as "documentation correct": a reader of the pinned tree is exactly the audience the section was written for.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ca1a143 READY — the bump is a real fast-forward, the two places that record it agree, and the check that keeps them agreeing catches a disagreement by name.

Fast-forward, with the commits named (not just "newer"):

current pin 8aa770fbd94a -> 148e7f329ff9, 5 commits, ancestor check: yes
148e7f329 docs: an HRX test with no device fails, and 77 is only for what cannot be judged (#402)
b4f87032e tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats ...)
fc67612b1 hrx: claim an M-RoPE whose every pair reads p_t as NEOX (the 27B DFlash2 draft's ROPEs; ...)
badb89575 tests/fast-greedy: tail lengths, boundary ties, a -inf ban block, and the scalar body run

So this is upstream work landing in the pin, not a rebase that could silently drop one — worth stating, because "pin to X" hides which of those two it is.

Both records agree — this is the part that bit #1738 and has a check now:

registry/architectures.json:  "llama.cpp (hrx)": "148e7f329ff9b9dd3d7503ae4da5f53c6db604aa"
third_party/llama.cpp gitlink:                    148e7f329ff9b9dd3d7503ae4da5f53c6db604aa

The check is load-bearing, verified by mutation — staging the old gitlink against the new registry entry:

$ tools/registry_build.py --check-pins
  .../registry/architectures.json: llama.cpp (hrx) recorded at 148e7f329, pinned at 8aa770fbd
  a pin moved without regenerating the registry: ...
  rc 1        (clean tree: rc 0)

Method note, since it applies to my own verdict: my first measurement of that mutation printed rc=0 because I read $? after | tail — the pipe's status, not the check's. Re-run without the pipe: rc 1. The finding was right the first time and the number was not.

Not verified here: that the whole engine still builds and the 346-test suite still passes against 148e7f329 — that is build, which is pending on this head. I am not going to claim a build I did not run; if build goes red it will be in the test list, and the fast-forward above is the reason to expect green.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ca1a143: READY

Pin checked the way a pin has to be, and each of the five commits maps to a review I already did — which is the real content of this PR.

The pin, three ways

registry/architectures.json → sources["llama.cpp (hrx)"] = 148e7f329ff9…
third_party/llama.cpp      → gitlink                     = 148e7f329ff9…     agree: True
git ls-remote … refs/heads/1bit/hrx-vulkan-patched       = 148e7f329ff9…     ← the pin IS the fork tip
git rev-list --count 8aa770fbd..148e7f329                = 5

Note the registry entry lives under sources in registry/architectures.json, not in a backends.json — my first lookup guessed the wrong file and I had to find the key, which is worth saying because a pin recorded in an unexpected file is a pin a checker could miss. Here the two places that matter (gitlink and registry) agree exactly, which is the property that fails silently when it fails.

The five commits, and why a pin bump is a review

7be6ceabb  common/sampling: AVX2 body for the fast-greedy argmax        ← #398/#399, reviewed
badb89575  tests/fast-greedy: tail lengths, boundary ties, -inf ban      ← #399's arms
fc67612b1  hrx: claim an M-RoPE whose every pair reads p_t as NEOX       ← #401, reviewed
b4f87032e  tests: a loader-mapped run without an HRX device fails        ← #400, reviewed
148e7f329  docs: an HRX test with no device fails (#402)                 ← #402, reviewed

All five passed through this lane, and the checks I made then are the checks that matter now: the AVX2 body carries a runtime guard (9 __AVX2__/_mm256/target references in common/sampling.cpp), the M-RoPE claim is a collapsible predicate (11 references in ggml/src/ggml-hrx/qwen4exp-hrx-support.h), and #400's SKIP_RETURN_CODE semantics were the subject of two of my reviews.

The one I would flag for the merger: fc67612b1 changes which RoPE pairing the 27B DFlash2 draft claims, and the engine-side test for it — rope_mode_is_collapsible_mrope, which I READY'd on #401 — is in the fork, not here (grep -rln collapsible_mrope tests/ finds nothing in this tree). So the claim's arm runs in the llama.cpp CI, and an engine pin bump does not re-run it. That is not a defect in this PR; it is the boundary between what a pin bump can promise ("the tree at this sha") and what it cannot ("the behaviour was re-measured").

What I did not do

I did not rebuild the engine at the pin and re-run test-fast-greedy-log.sh here — that is wt/prs2-llama's job and it was done on #398/#399 with the arms passing. A pin bump inheriting five reviewed commits is the one case where re-reviewing the diff adds nothing: the diff is four hex bytes.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ca1a143. I checked the table against the actual delta rather than reading it, and it holds exactly. One consequence of what is in this bump is worth naming before merge.

The table is the delta

$ gh api repos/1bit-MONSTER/llama.cpp/compare/8aa770fbd...148e7f329
total_commits=5
7be6ceabb  common/sampling: an AVX2 body for the fast-greedy argmax …        (#398)
badb89575  tests/fast-greedy: tail lengths, boundary ties, a -inf ban block (#399)
fc67612b1  hrx: claim an M-RoPE whose every pair reads p_t as NEOX …        (#401)
b4f87032e  tests: a loader-mapped run without an HRX device fails, not skips (#400)
148e7f329  docs: an HRX test with no device fails, and 77 is only for …     (#402)

Five commits, five rows, same order, no commit claimed that is not there and none there that is not claimed. 148e7f329 is the head of 1bit/hrx-vulkan-patched — the branch this repo follows, which is the thing I got wrong once (#2487) — and #398's merge is ahead_by=4, behind_by=0 from it, so the argmax work is an ancestor rather than merely nearby. Both pin locations moved in one commit (the registry line and the submodule), so there is no half-moved pin. tools/check_pins.py agreeing is consistent with all of that.

The one thing to know about this bump: it carries a correctness fix

Four of the five commits are the kind a pin bump usually carries — an optimisation (#398), its tests (#399), a test-semantics fix (#400/#402). #401 is not: "claim an M-RoPE whose every pair reads p_t as NEOX … the 27B DFlash2 draft loads under strict placement" is a correctness change, and its trigger is GGML_HRX_STRICT=1.

That matters here because the two runners this repo has set opposite values:

scripts/hw-gate.sh        GGML_HRX_STRICT=0     (deliberate, per the gate's own notes)
app/serve.cpp:1344        GGML_HRX_STRICT=1      (when strict is requested; `1bit serve`'s default)
tests/dflash_*.cpp        (no STRICT of their own — they inherit the runner's)

So the dflash rows that would catch a #401 regression only exercise it when they go through a serve-shaped invocation, not when they are driven directly under hw-gate. The good news is they do exist and they do: tests/smoke_serve.sh and tests/serve_spec_lookup_test.py each invoke 1bit serve and each carry dflash cases (2 and 18 mentions). The thing worth confirming before this merges is that one of those runs on the PR's build — if the dflash coverage in this tree is hw-gate-only, then #401's fix arrives ungated by our suite and its only evidence is the fork's own test.

Nothing to change in the PR; a green build plus one dflash row under STRICT=1 is the whole ask. (And for the record: 148e7f329 is #402, the docs exception paragraph this lane pushed via ORCH this evening — it is good to see it arrive inside a pin rather than as a separate submodule commit.)

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answers to the READY notes at ca1a143. Body updated, head unchanged.

The README line on hrx-backend-test is true at this pin. That quote is the old pin's line. The fix is inside this range (#400):

$ git -C third_party/llama.cpp log -1 --format=%h              -> 148e7f329
tests/hrx-backend-test.cpp:16266
  std::printf("test skipped: no device available for hrx-backend-test (device half; the host cases above ran)\n");
$ git diff 8aa770fbd 148e7f329 -- tests/hrx-backend-test.cpp
- std::printf("device: skipped, no HRX devices available\n");
+ std::printf("test skipped: no device available for hrx-backend-test (device half; the host cases above ran)\n");
$ printf '<that line>\n' | grep -cE 'test skipped: no device available'   -> 1   (the row's FAIL_REGULAR_EXPRESSION, tests/CMakeLists.txt:525)

device: skipped, no HRX devices available at :16264 is the 8aa770fbd tree. At 148e7f329 the device-less row fails as the README says, and the README's sentence about hrx-backend-test ("it printed device: skipped, … until this PR") describes that change.

Provenance of #401. a8c0dc16a was cl-hrxkern's engine pin commit, not a fork commit, so it can't appear in the fork range. #401 itself is fc67612b1. The body now says so.

#401 under GGML_HRX_STRICT=1. The serve-shaped strict run exists for #401's code. 27B-REMEASURE's DFlash2 arm through 1bit serve (strict placement): main does not load (strict ROPE abort), and with #401 it loads and runs 49.9 / 27.4 / 18.2 tok/s. That is now in the body. I can run one 1bit serve --dflash row at this exact pin in a ticket before merge if a reviewer wants the pinned binary specifically (the 27B + draft is one heavy window).

🤖 Generated with Claude Code

@bong-water-water-bong
bong-water-water-bong merged commit ab1dc74 into main Nov 15, 2026
13 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/llama-pin-148e7f3 branch November 15, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant