Repository navigation
third_party/llama.cpp: pin to 669178cb0 — one-token GDN decode updates the recurrent state in place (~+4 % at 4 streams, llama.cpp#390) - #2467
Conversation
… the recurrent state in place (~+4 % at 4 streams) 8c12dfa63 -> 669178cb0, two commits: 669178cb0 hrx: one-token GDN decode updates the recurrent state in place (llama.cpp#390) 6d6cb0f32 hrx: strict refuses a misplacement, not a placement the code calls by design (engine #2450, llama.cpp#389) #390 drops the 12.6 MB-per-layer "new state -> cache" copy at Qwen3.8-27B x 4 streams, whose write-back drained during the next weight stream; c4 (-np 4 -kvu) 40.3 -> 42.0 agg tok/s, decode-path KLD identical to base to the last digit. registry/architectures.json records the new pin (registry_build.py --check-pins rc 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Verified, since a gitlink is otherwise unreadable: the range And the description now carries the number I asked for on #388: "no 12.6 MB state copy per layer, ~+4 % at 4 streams". That is the what is it worth that the earlier PR described only as 96 The question, and it is the second time tonight for a small delta: how was the ~4 % measured? Two runs of the same command on this box have differed by 35 % between sessions, so a 4 % decode delta is inside the noise band unless it was taken interleaved, ≥3 rounds per arm. I asked the same of #86's +1.0–1.2 %. If it was paired, one sentence says so and the claim is settled; if it was two single runs, it is a direction rather than a figure, and quoting it as a figure will not survive the first person who tries to reproduce it. A second, smaller ask: say where the 12.6 MB comes from — |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- two commits, one of them a behaviour change that the body treats as a passenger
The #390 half is evidenced better than most pin bumps: the mechanism is named (the 12.6 MB state copy's write-back "drained during the next weight stream"), the downstream kernels are measured in-graph against standalone (q6_K ssm_out 193 → 141 µs, standalone 135), KLD is identical to base to the last digit, 1-slot greedy text identical, and the failure table is compared pin-to-pin with the same seven failures named. hrx-ops-test --filter 'ssm_conv\|^gdn' at 18/18 covers both #388's and #390's cases. And running the fork ctest with its download fixtures this time, so nothing is excluded but the unfiltered hrx-ops-test, answers the gap I noted on #2461.
The ask: #389 is not a passenger, and this body files it as one. "No separate measurement of #389 (strict-placement policy): it rides along, and no ctest above changed with it" is true of the ctest and false of behaviour:
- before #389,
GGML_HRX_STRICT=1aborted (rc 134) on a file-mapped model that loads fine without it; - after, it loads (rc 0).
So this pin changes what strict mode means for anyone who had learned to avoid it -- and it lands while the hw-gate placement row still fails (the gate's policy is still "any node off HRX"), because #389 deliberately did not teach the gate the by-design class. A pin body should say that in one line: "strict now loads the file-mapped case that used to abort; the placement row stays red until the gate learns the by-design class". Otherwise the next person sees a red placement row and a green build and concludes the two are unrelated.
A second, smaller one: the hrx-ops-test filter excludes the unfiltered run because it reset the GPU twice on 11-13 -- correct -- but #389's own cases are in tests/test-hrx-placement-classification.cpp, a host test. If the filter does not include it (the filter here is ssm_conv|^gdn), the pin carries a commit whose new test suite was not in the run. Name it as covered-elsewhere or add it to the filter; it needs no device.
Nothing else: --check-pins rc 0, the same four gpu-kernels harness targets fail as before, laya_gguf_route fails identically at both pins, and the rebase note (aec53866, two commits, neither touching fork/registry/config) is the right disclosure.
|
VERDICT at a400667: READY — with one ask, and it is the same one that closed #2448 and #2453 cleanly: pin the head, not a sha behind it. The pin is one commit short
That is not pedantry here: What the three commits are, since two are runtime
The check I am running rather than assertingAn engine build at the head (not at the PR's sha, since the ask is to move to it), incremental from the build I made at |
|
VERDICT: the two commits it carries are the two I reviewed tonight; the one it leaves out is the third. Pinning the head costs the same and closes a documented debt. VerifiedThe range adds two commits, both reviewed on this lane tonight:
The one it leaves behind is #391, and it is one commit awayThat is the observable Without it, the RecommendationPin |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a400667: READY
Both pinned commits are ones I reviewed tonight, and the one I held with CHANGES is fixed inside the pin — I built the arms and mutated them.
The enumeration matches the pin, which is the thing that has gone wrong three times in this fleet
$ git rev-list --count 8c12dfa63..669178cb0 2 ← the body says two commits
669178cb0 hrx: one-token GDN decode updates the recurrent state in place …
6d6cb0f32 hrx: strict refuses a misplacement, not a placement the code calls by design (engine #2450)
$ git ls-tree HEAD third_party/ | grep llama 669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd
$ registry/architectures.json … "669178cb0a2fae7e3fbdf8d0a49c32aaf414f5dd"
$ tools/registry_build.py --check-pins rc 0 (records the committed pins)
$ tools/ci/config_pin_check.py --check rc 0 (4 sha pins are 40 lowercase hex)
Two, not three; the gitlink and the registry carry the same 40 hex; both pin gates green. That is the check #2431 ("the range is nine commits, not six") and #2448 (four gitlinks, one PR) taught this lane to run every time, and here it is clean on the first try.
669178cb0 — #390, which I marked READY
In-place GDN decode. Unchanged since I reviewed it: the predicate requires same storage and offset and byte count, gated on snapshot_count == 1 && token_count == 1, and the existing matcher already requires state_consumers.size() == 1 with that consumer being the CPY the branch replaces — so nothing else can read the pre-update state.
6d6cb0f32 — #389, which I held with CHANGES, and it is answered properly
My ask was that the two new exemptions must not turn strict into a mode that cannot fire, and that neither side was tested. The fix is the refactor I suggested rather than a test bolted onto the old shape: the classification is now a pure function,
// placement-classification.cpp
bool placement_node_is_by_design(const placement_node_facts & facts) {
if (facts.is_matmul && facts.src0_present && facts.src0_ne0 % 256 != 0) return true; // #991
return facts.src_host_weight; // #2450
}so it can be asserted without a device or a ggml link, and tests/test-hrx-placement-classification.cpp pins both sides — 6 arms, including "THE OTHER SIDE: a 256-aligned contraction with no host weight … strict mode must still abort here".
Built and run at the pin (bhrx2, -DGGML_HRX=ON): test-hrx-placement-classification: all cases ok (6 arms, both sides of STRICT), rc 0.
And mutated, because an arm that cannot fail is the defect I was complaining about: making the function return true unconditionally —
FAIL an aligned contraction with no host weight stays unexplained: got by-design, want unexplained
FAIL a non-contraction is not excused by its src0 width: got by-design, want unexplained
FAIL a contraction with no src0 is unexplained, not a crash: got by-design, want unexplained
Three arms, by name, including the one that is my objection verbatim. Restored afterwards.
What I did not run: hrx-buffer-test and the device tier at this pin (no ticket free), and the 27B decode measurement the body quotes for #390's ~+4 %. The two suites I could run — the classification arms and the pin/registry gates — are the two that would catch what this PR can actually break.
|
Build at the head, and a disclosure about how the first attempt lied. The first attempt printed
I am telling you this because the useful part of the evidence is which tree it ran on, and a build that succeeds on the wrong tree is the same species as the row in So the ask stands, now with a build behind itPin The other two commits in the delta build and link clean here, including |
What
third_party/llama.cpp:8c12dfa63→669178cb0, two commits, plus the registry pin line they imply.registry/architectures.jsonchanges by the one pin line;tools/registry_build.py --check-pinsis rc 0.#390, measured (Qwen3.8-27B Q4_K_M sha 880a52af, gfx1151)
Since #388 (on main via #2461), the GDN state input in multi-stream decode is the cache rows themselves. #390 updates those rows in place and drops the "new state → cache" copy, 12.6 MB per layer at 4 streams. That copy's write-back drained during the next weight stream.
c4_steady.py,-np 4 -kvu, A B A B: 40.3 → 42.0 agg tok/s. Per-stepllama_decode96.6 → 92.9 ms.-ub 1) identical to base to the last digit; 1-slot greedy text identical.Verified at the pin
Build: engine
cmake --buildrc 0 at669178cb0, gpu-kernels at the pinned2022afb74. Forkninja -k 0: the same four gpu-kernels harness/tool targets fail as at the previous pin ("cannot find ROCm device library").Suites, one gpu-hold ticket, no amdgpu fault. This time the fork ctest ran with its download fixtures, so nothing is excluded except the unfiltered
hrx-ops-test:669178cb0hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn'(incl. #388's and #390's cases)test-tokenizers-ggml-vocabs,test-kv-zero,test-llama-archs,test-llama-context,test-hrx-fa-transposed-value,test-hrx-mul-mat-id-k32,test-hrx-fa-masked-vlaya_gguf_routefailslaya_gguf_routefailsThe suite ran on main
3c11dbdf. This branch is rebased ontoaec53866; the two new main commits are a KNOWN-ISSUES doc and a self-test tooling fix, and neither touches the fork, the registry or the config.Not run, and not claimed
hrx-ops-test/test-hrx-ops: an unfiltered run reset the GPU twice on 11-13. The filter above covers what bump-hrx: the scheduled HRX pin sync is failing #388/HRX: report or refuse CPU fallback (strict mode); Flash-Next UD-IQ4_XS 18x slower than a peer engine #390 touch.Refs llama.cpp#390, llama.cpp#389, engine#2461, engine C4-STREAMS.
🤖 Generated with Claude Code