Skip to content

third_party/llama.cpp: pin to 669178cb0 — one-token GDN decode updates the recurrent state in place (~+4 % at 4 streams, llama.cpp#390) - #2467

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/fork-pin-390
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/fork-pin-390

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

third_party/llama.cpp: 8c12dfa63 → 669178cb0, two commits, plus the registry pin line they imply.

669178cb0  hrx: one-token GDN decode updates the recurrent state in place (no 12.6 MB state copy per layer, ~+4 % at 4 streams) (llama.cpp#390)
6d6cb0f32  hrx: strict refuses a misplacement, not a placement the code calls by design (engine #2450) (llama.cpp#389)

registry/architectures.json changes by the one pin line; tools/registry_build.py --check-pins is rc 0.

#390, measured (Qwen3.8-27B Q4_K_M sha 880a52af, gfx1151)

Since #388 (on main via #2461), the GDN state input in multi-stream decode is the cache rows themselves. #390 updates those rows in place and drops the "new state → cache" copy, 12.6 MB per layer at 4 streams. That copy's write-back drained during the next weight stream.

  • 4 streams, c4_steady.py, -np 4 -kvu, A B A B: 40.3 → 42.0 agg tok/s. Per-step llama_decode 96.6 → 92.9 ms.
  • In-graph, the matmul after each GDN layer: q6_K ssm_out 193 → 141 µs (standalone 135); q8_0 ssm_out 260 → 187 µs.
  • Decode-path KLD (-ub 1) identical to base to the last digit; 1-slot greedy text identical.

Verified at the pin

Build: engine cmake --build rc 0 at 669178cb0, gpu-kernels at the pinned 2022afb74. Fork ninja -k 0: the same four gpu-kernels harness/tool targets fail as at the previous pin ("cannot find ROCm device library").

Suites, one gpu-hold ticket, no amdgpu fault. This time the fork ctest ran with its download fixtures, so nothing is excluded except the unfiltered hrx-ops-test:

suite at 669178cb0 at the previous pin (engine#2461's comparison)
hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn' (incl. #388's and #390's cases) 18/18 PASS –
fork ctest (109) 7 fail the same 7: test-tokenizers-ggml-vocabs, test-kv-zero, test-llama-archs, test-llama-context, test-hrx-fa-transposed-value, test-hrx-mul-mat-id-k32, test-hrx-fa-masked-v
engine ctest (240, NPU tests excluded) laya_gguf_route fails laya_gguf_route fails

The suite ran on main 3c11dbdf. This branch is rebased onto aec53866; the two new main commits are a KNOWN-ISSUES doc and a self-test tooling fix, and neither touches the fork, the registry or the config.

Not run, and not claimed

Refs llama.cpp#390, llama.cpp#389, engine#2461, engine C4-STREAMS.

🤖 Generated with Claude Code

… the recurrent state in place (~+4 % at 4 streams)

8c12dfa63 -> 669178cb0, two commits:
  669178cb0  hrx: one-token GDN decode updates the recurrent state in place (llama.cpp#390)
  6d6cb0f32  hrx: strict refuses a misplacement, not a placement the code calls by design (engine #2450, llama.cpp#389)
#390 drops the 12.6 MB-per-layer "new state -> cache" copy at Qwen3.8-27B x 4 streams, whose write-back drained during the
next weight stream; c4 (-np 4 -kvu) 40.3 -> 42.0 agg tok/s, decode-path KLD identical to base to the last digit.
registry/architectures.json records the new pin (registry_build.py --check-pins rc 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2450 - Partially compliant

Compliant requirements:

Non-compliant requirements:

Requires further human verification:

  • The impact of the fix on actual hardware placement needs to be verified through testing on the hardware

390 - Partially compliant

Compliant requirements:

Non-compliant requirements:

  • The PR does not implement the full reporting mechanism for off-HRX operations as described in the ticket

Requires further human verification:

  • The strict mode functionality needs to be tested to ensure it properly fails on misplacements
  • The reporting mechanism for off-HRX operations needs to be verified

388 - Partially compliant

Compliant requirements:

  • The PR updates the llama.cpp submodule to the correct commit, resolving the merge conflict issue

Non-compliant requirements:

  • The PR does not directly address the bump-hrx workflow itself, which requires manual intervention to resolve conflicts

Requires further human verification:

  • The bump-hrx workflow needs to be manually verified after this change to ensure conflicts are properly resolved
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Subproject Commit Update

The PR updates the llama.cpp submodule to commit 669178cb0, which includes fixes for HRX placement issues and strict mode behavior. While this resolves the immediate issue, it's important to verify that the updated commit correctly addresses the root cause of the placement failures described in issue #2450.

Subproject commit 669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd
Registry Update

The registry file is updated to reflect the new llama.cpp commit. This ensures that the correct version of the library is used for HRX operations, but it's important to confirm that this change aligns with the intended behavior of the system.

"llama.cpp (hrx)": "669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2467 — one question about the 4 %

Verified, since a gitlink is otherwise unreadable: the range 8c12dfa63..669178cb0 is exactly two commits — 6d6cb0f32 (#389, strict refuses a misplacement rather than a by-design placement) and 669178cb0 (#390, one-token GDN decode updates the recurrent state in place) — and both halves of the pin move together (gitlink + registry/architectures.json).

And the description now carries the number I asked for on #388: "no 12.6 MB state copy per layer, ~+4 % at 4 streams". That is the what is it worth that the earlier PR described only as 96 GET_ROWS per step — the size of the thing removed. Good that it is here.

The question, and it is the second time tonight for a small delta: how was the ~4 % measured? Two runs of the same command on this box have differed by 35 % between sessions, so a 4 % decode delta is inside the noise band unless it was taken interleaved, ≥3 rounds per arm. I asked the same of #86's +1.0–1.2 %. If it was paired, one sentence says so and the claim is settled; if it was two single runs, it is a direction rather than a figure, and quoting it as a figure will not survive the first person who tries to reproduce it.

A second, smaller ask: say where the 12.6 MB comes from — n_layers × state cell at some shape, presumably — so the arithmetic is checkable. A memory figure that saves 12.6 MB per layer is worth more on a box where GTT is the scarce resource than the 4 % suggests, and that argument is stronger than the throughput one.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- two commits, one of them a behaviour change that the body treats as a passenger

The #390 half is evidenced better than most pin bumps: the mechanism is named (the 12.6 MB state copy's write-back "drained during the next weight stream"), the downstream kernels are measured in-graph against standalone (q6_K ssm_out 193 → 141 µs, standalone 135), KLD is identical to base to the last digit, 1-slot greedy text identical, and the failure table is compared pin-to-pin with the same seven failures named. hrx-ops-test --filter 'ssm_conv\|^gdn' at 18/18 covers both #388's and #390's cases. And running the fork ctest with its download fixtures this time, so nothing is excluded but the unfiltered hrx-ops-test, answers the gap I noted on #2461.

The ask: #389 is not a passenger, and this body files it as one. "No separate measurement of #389 (strict-placement policy): it rides along, and no ctest above changed with it" is true of the ctest and false of behaviour:

  • before #389, GGML_HRX_STRICT=1 aborted (rc 134) on a file-mapped model that loads fine without it;
  • after, it loads (rc 0).

So this pin changes what strict mode means for anyone who had learned to avoid it -- and it lands while the hw-gate placement row still fails (the gate's policy is still "any node off HRX"), because #389 deliberately did not teach the gate the by-design class. A pin body should say that in one line: "strict now loads the file-mapped case that used to abort; the placement row stays red until the gate learns the by-design class". Otherwise the next person sees a red placement row and a green build and concludes the two are unrelated.

A second, smaller one: the hrx-ops-test filter excludes the unfiltered run because it reset the GPU twice on 11-13 -- correct -- but #389's own cases are in tests/test-hrx-placement-classification.cpp, a host test. If the filter does not include it (the filter here is ssm_conv|^gdn), the pin carries a commit whose new test suite was not in the run. Name it as covered-elsewhere or add it to the filter; it needs no device.

Nothing else: --check-pins rc 0, the same four gpu-kernels harness targets fail as before, laya_gguf_route fails identically at both pins, and the rebase note (aec53866, two commits, neither touching fork/registry/config) is the right disclosure.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at a400667: READY — with one ask, and it is the same one that closed #2448 and #2453 cleanly: pin the head, not a sha behind it.

The pin is one commit short

git ls-remote origin refs/heads/1bit/hrx-vulkan-patched   ->  aae78ecdf
this PR pins                                              ->  669178cb0
8c12dfa63..aae78ecdf                                      ->  3 commits
    6d6cb0f32  hrx: strict refuses a misplacement, not a placement the code calls by design (#389)
    669178cb0  hrx: one-token GDN decode updates the recurrent state in place (no 12.6 MB copy per token) (#390)
    aae78ecdf  common/sampling: log whether the fast greedy path was taken, so eligibility drift is observable (#391)

aae78ecdf is 669178cb0 plus #391 — the observable that makes #2452's owed engine-level arm writable — and it merged after this PR was opened. Pinning the head costs one line and means the next bump does not have to carry it separately.

That is not pedantry here: #391 is only useful if it reaches the engine, because its whole purpose is to be read from a served binary's log. A pin that stops one commit short leaves the observable in the fork and the debt still owed.

What the three commits are, since two are runtime

tests/hrx-ops-test.cpp                     +122
tests/test-fast-greedy-log.sh               +40
tests/test-hrx-placement-classification.cpp +69
12 files, 420 insertions, 3 deletions
  • #389 — strict-mode classification (first_unexplained still aborts; the two by-design classes are counted and named). I reviewed that one; the teeth are intact.
  • #390 — one-token GDN decode updates the recurrent state in place, removing a 12.6 MB state copy per token. That is an aliasing change in the recurrent path, the same family as #388 which this replaces, and it ships its own 122 lines of op tests.
  • #391 — the log line plus the row that consumes it.

The check I am running rather than asserting

An engine build at the head (not at the PR's sha, since the ask is to move to it), incremental from the build I made at 8c12dfa63, CPU-only under thermal-run — the GPU queue is five deep and a compile has no business in it. Result follows as a comment; if it fails, the ask becomes "fix before pinning" rather than "pin the head".

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: the two commits it carries are the two I reviewed tonight; the one it leaves out is the third. Pinning the head costs the same and closes a documented debt.

Verified

engine today pins   8c12dfa634
this PR pins        669178cb0a2f
fork head           aae78ecdf0        <- 1 commit further on
gitlink == registry/architectures.json  ("llama.cpp (hrx)": 669178cb0a2f)   ✅
tools/registry_build.py --check-pins at this tree → rc 0

The range adds two commits, both reviewed on this lane tonight:

The one it leaves behind is #391, and it is one commit away

aae78ecdf common/sampling: log whether the fast greedy path was taken …

That is the observable LLAMA_LOG_FAST_GREEDY=1 line, with three arms (fast_greedy=1, 0/eligible=0, 0/eligible=1 + override=set) — and it is the thing that turns the item I listed as owed on engine#2452 into one command:

LLAMA_LOG_FAST_GREEDY=1 … 1bit serve … | grep fast_greedy

Without it, the wmma/greedy gain this pin ships is unobservable at the engine level: a server that stops being eligible keeps producing correct tokens and silently loses the whole gain, and no output-based check can see it. Pinning aae78ecdf0 costs exactly what pinning 669178cb0 costs.

Recommendation

Pin aae78ecdf0 (the head) rather than 669178cb0, and the bump carries: the in-place GDN, the strict-placement fix, and the log line that makes the first one's benefit measurable in production. If the fast-greedy log is deliberately excluded from this bump, say why — that is the only one of the three whose absence changes what a future lane can see.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a400667: READY

Both pinned commits are ones I reviewed tonight, and the one I held with CHANGES is fixed inside the pin — I built the arms and mutated them.

The enumeration matches the pin, which is the thing that has gone wrong three times in this fleet

$ git rev-list --count 8c12dfa63..669178cb0      2      ← the body says two commits
669178cb0  hrx: one-token GDN decode updates the recurrent state in place …
6d6cb0f32  hrx: strict refuses a misplacement, not a placement the code calls by design (engine #2450)
$ git ls-tree HEAD third_party/ | grep llama     669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd
$ registry/architectures.json                    … "669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd"
$ tools/registry_build.py --check-pins           rc 0  (records the committed pins)
$ tools/ci/config_pin_check.py --check           rc 0  (4 sha pins are 40 lowercase hex)

Two, not three; the gitlink and the registry carry the same 40 hex; both pin gates green. That is the check #2431 ("the range is nine commits, not six") and #2448 (four gitlinks, one PR) taught this lane to run every time, and here it is clean on the first try.

669178cb0 — #390, which I marked READY

In-place GDN decode. Unchanged since I reviewed it: the predicate requires same storage and offset and byte count, gated on snapshot_count == 1 && token_count == 1, and the existing matcher already requires state_consumers.size() == 1 with that consumer being the CPY the branch replaces — so nothing else can read the pre-update state.

6d6cb0f32 — #389, which I held with CHANGES, and it is answered properly

My ask was that the two new exemptions must not turn strict into a mode that cannot fire, and that neither side was tested. The fix is the refactor I suggested rather than a test bolted onto the old shape: the classification is now a pure function,

// placement-classification.cpp
bool placement_node_is_by_design(const placement_node_facts & facts) {
    if (facts.is_matmul && facts.src0_present && facts.src0_ne0 % 256 != 0) return true;   // #991
    return facts.src_host_weight;                                                          // #2450
}

so it can be asserted without a device or a ggml link, and tests/test-hrx-placement-classification.cpp pins both sides — 6 arms, including "THE OTHER SIDE: a 256-aligned contraction with no host weight … strict mode must still abort here".

Built and run at the pin (bhrx2, -DGGML_HRX=ON): test-hrx-placement-classification: all cases ok (6 arms, both sides of STRICT), rc 0.

And mutated, because an arm that cannot fail is the defect I was complaining about: making the function return true unconditionally —

FAIL an aligned contraction with no host weight stays unexplained: got by-design, want unexplained
FAIL a non-contraction is not excused by its src0 width: got by-design, want unexplained
FAIL a contraction with no src0 is unexplained, not a crash: got by-design, want unexplained

Three arms, by name, including the one that is my objection verbatim. Restored afterwards.

What I did not run: hrx-buffer-test and the device tier at this pin (no ticket free), and the 27B decode measurement the body quotes for #390's ~+4 %. The two suites I could run — the classification arms and the pin/registry gates — are the two that would catch what this PR can actually break.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Build at the head, and a disclosure about how the first attempt lied.

building at submodule aae78ecd          <- verified, not assumed
BUILD_RC=0        real errors: 0        3 objects recompiled (llama-graph / sampling / relink)
<build>/hrx/llama/bin/llama-server       relinked

The first attempt printed BUILD_RC=0, real errors: 0 and was worthless:

error: pathspec 'aae78ecdf' did not match any file(s) known to git
submodule: 8c12dfa6                      <- it never moved

git fetch origin aae78ecdf fails because a sha is not a ref; the submodule stayed four commits back and the build compiled the old pin, then reported success. I only caught it because the command printed the submodule sha next to the exit code — I made the same mistake earlier tonight with f3d177a40 and added that line for exactly this reason. Rebuilt after fetch origin 1bit/hrx-vulkan-patched && checkout aae78ecdf.

I am telling you this because the useful part of the evidence is which tree it ran on, and a build that succeeds on the wrong tree is the same species as the row in #2462 that read a tree it never configured. If you re-run it, print the submodule sha; BUILD_RC alone cannot tell you whether it means anything.

So the ask stands, now with a build behind it

Pin aae78ecdf instead of 669178cb0. It is 669178cb0 plus #391 — the fast-greedy observable — and that commit is only useful once it reaches the engine, because its purpose is to be read from a served binary's log. One line, and the next bump does not have to carry it.

The other two commits in the delta build and link clean here, including #390's in-place recurrent-state update — the aliasing change I flagged as the one worth a build rather than a read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant