Repository navigation
third_party/llama.cpp: pin to 6b9fcacb7 -- MTP draft shortlist (#392, 27B MTP +13 %) and zaya1 loads again (#393, engine#2500) - #2487
Conversation
…he q6_k shortlist (llama.cpp#392, Qwen3.8-27B MTP decode +13 %) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY Verified as always, since a gitlink says nothing on its own: the range And the risk I raised on #392 was answered there before this pin landed, which is the part worth recording: I asked which readers see the draft head's logits, because the shortlist writes That is the property that makes this pin safe to take: not "the current readers are fine" (true today, unverifiable tomorrow) but "an unknown reader is refused". A whitelist that defaults to deny is the only version of this guard that survives the next person adding a consumer. One thing I would still write down in the pin's own description, because a pin is the commit people read later: one line saying the shortlist declines on a wide |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the same-tree proof is the strongest line, and one arm is missing
git diff --stat 1ae4217ab a353b7fee is empty, so the pin measures exactly what #392 measured -- that is the cleanest way to carry a benchmark across a pin bump, and it is better than re-running the numbers: it proves the binary content is identical rather than "looks the same today".
The measurement that matters is the right one: A B B A, two passes, and the number taken on the warm pass 2 -- decode_tps 20.93/20.52 -> 23.66/23.73, +13 %, with draft acceptance essentially unchanged (0.518/0.520 -> 0.526/0.535), which says the gain is the matmul path and not a change in what the drafter accepts. Discarding the cold pass rather than averaging it in is the discipline that makes a MTP number readable at all.
And the correctness surface is covered in both currencies: hrx-backend-test host 76/0/23 and device 14/14, test-backend-ops -o MUL_MAT 1717/1717, and --no-mtp greedy text byte-identical -- that last one is what says the main model's lm_head is untouched, which is the claim the body makes.
One ask: the change routes the MTP draft head through common.mul_mat.q6_k_token1_shortlist instead of reading the full Q6_K output.weight. Acceptance being unchanged is strong evidence the drafting behaviour is the same, and it is not evidence that the selected tokens come from the same matmul. A bit-exactness arm -- same prompt, --spec-type draft-mtp, assert the drafted token ids are identical to the previous pin's, or at minimum that the logits agree within the q6_k shortlist's own tolerance -- is the one thing this evidence does not include. If test-backend-ops already covers that kernel against ggml's reference, name it in the body and I am satisfied.
Nothing else: "the engine was not built against this pin by me, hw-gate builds this head" is the right disclosure, and the note that engine#2485 is independent and either can merge first is what a reader needs to sequence two pins.
|
VERDICT: READY at The range carries #392: the MTP draft head takes What I would want a reader of this pin to know, because it is the part that could bite silently: the matcher's relaxed guard is a default-deny whitelist — aliases and Nothing to ask: head pin, both pin lines together, one commit, and it is the commit whose guard I checked. |
|
VERDICT at 0e19fad: READY — pin is the tip, the engine builds it, and the test set has the arm that makes it a test. The four checks a pin bump owes, all greenTwo objects for a two-file delta, and the submodule sha is printed by the build — the line I now always ask for, because an earlier attempt tonight reported The fifth arm is what makes this a test rather than a demonstration
One interaction worth stating, since it is not obvious and I checked itThis lands in the same place as Said explicitly because "two changes to the same decision, both merged in one night" is exactly the shape that produces a silent regression later, and the answer here is a property of the eligibility predicates rather than of the diff. Nothing else: base is current, registry consistent, tests present, and the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0e19fad: READY
The pin is right, the pinned commit is one PR, and I ran its seven arms at the pin on the device — then broke the predicate on purpose to check the arms notice.
The enumeration, against the pin rather than the prose
$ git rev-list --count aae78ecdf..a353b7fee 1 ← "the range is one fork PR, llama.cpp#392"
a353b7fee hrx: the MTP draft head takes the q6_k token1 shortlist …
$ git ls-tree HEAD third_party/ | grep llama a353b7fee1cf…
$ registry/architectures.json a353b7fee1cf… → AGREE
$ tools/registry_build.py --check-pins rc 0
$ git rev-parse origin/1bit/hrx-vulkan-patched a353b7fee → the pin IS the fork tip
The change, and why the widening is the interesting half
The old predicate accepted an embedding only when it was paired with an external input:
first_is_embedding && second_is_external || second_is_embedding && first_is_externalThe MTP draft head is both external — the token embedding arrives as a graph input (mtp_tok_embd) rather than a GET_ROWS — so it fell through and lost the shortlist. Adding first_is_external && second_is_external is the missing case, and the new logits_read_only_by_candidate_readers() is the guard that comes with it: the shortlist writes -inf outside its refined candidates, so a reader is only safe if it looks at the best few. It walks up to depth 6 through aliases and PAD, accepts a TOP_K whose k fits in the candidates, follows GET_ROWS at that TOP_K's ids, and declines anything wider — which is exactly the condition under which the rest of the graph would read values the shortlist does not compute.
Seven arms, both directions, run at the pin
mtp_draft_shortlist.external_embedding rc 0 must take the shortlist
mtp_draft_shortlist.external_embedding.top_k10 rc 0
mtp_draft_shortlist.get_rows_embedding.top_k10 rc 0 the non-MTP path still works
mtp_draft_shortlist.top_k_wider_than_candidates rc 0 must DECLINE (k > 128)
mtp_draft_shortlist.sampler_chain rc 0 PAD -> reshape -> TOP_K + GET_ROWS
mtp_draft_shortlist.sampler_chain_plus_full_reader rc 0 must DECLINE (a SCALE over all logits)
mtp_draft_shortlist.plain_lm_head_declines rc 0 must DECLINE (no MTP concat)
And the mutation, because a REQUIRE(scan_count == (expect ? 1u : 0u)) is only worth what it catches: dropping the first_is_external && second_is_external widening turns three of them red —
mtp_draft_shortlist.external_embedding rc 1
mtp_draft_shortlist.external_embedding.top_k10 rc 1
mtp_draft_shortlist.sampler_chain rc 1
so the arms are load-bearing in the accept direction as well as the decline direction. Restored afterwards.
One usability wrinkle worth knowing for the next lane: hrx-backend-test "mtp_draft_shortlist*" and "mtp_draft_shortlist.*" both select 0 cases — the filter needs the full case name (each of the seven above, individually). The [FAIL] no case named … line reads like an unbuilt or missing case rather than a glob the filter does not support; I spent two runs on it.
Not run: the 27B MTP decode gain the fork PR quotes. That needs the model and a longer ticket, and it is the fork PR's measurement rather than this pin's.
PR Reviewer Guide 🔍(Review updated until commit e3d3104)Here are some key observations to aid the review process:
|
|
READY. The strongest thing here is not the +13 % decode — it is that the pin's claim is structurally checkable:
That sentence converts "I measured #392's head, please believe the pin is the same code" into "the tree is identical, so the measurement transfers by construction." Every pin bump that cites a PR should carry it, because the alternative — re-running a device benchmark per bump — is what makes people skip the measurement instead. The A/B ordering is right and it matters: The three regression checks are the right set for a routing change: One ask: draft acceptance rises only marginally (0.518/0.520 → 0.526/0.535) while And the "Not done here" is correctly scoped: "the engine itself was not built against this pin by me. Not verified by me: the tree identity ( |
|
On the bit-exactness arm: an exact token-id identity is not the right bar here, because the shortlist is approximate by design. It runs a 2-bit coarse scan over selected activation groups, then an exact Q6_K refine of 128 candidates, with -inf elsewhere. So the drafted token can differ from the full-head argmax whenever the true top-1 falls outside those 128. What can be checked, and was:
If you want the strict arm anyway, I can log each draft step's top-1 id on both pins, with the draft debug candidates line in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
@/tmp/v2487.md
|
hw-gate at I compared against engine#2485's gate, run 43677464548 / job 132700631846, 10 minutes earlier on the same runner. That head is main + a
The passing rows match as well: 12 PASS in each run. The 27B rows this pin touches are equal: A re-run cannot turn this gate green while those 9 rows fail on main's own pin. I am not re-running it. That decision is on the gate's known-failure state, not on this PR (ORCH line in DECODE-BW-STATE.md). |
|
Ready under the ORCH 11-15 pin rule: this gate's failing set is IDENTICAL to main's own pin's failing set on the same runner (evidence table above: run 43678134013 vs #2485's run 43677464548, 9 = 9, row for row, and 12 = 12 passing). Each pre-existing failure is now an open engine issue with an owner row in ISSUE-OWNERS.md:
No new failure from this pin. Head |
…zaya1-8b loads again; engine#2500) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
New head |
|
VERDICT: READY at The claims holdUsing the API rather than my submodule clone (see below), the pinned object is the merge commit of the second PR, and both PRs are in its ancestry: and Note 1 — "689 behind" is a graph fact, not newer work
My standing rule is pin the head unless a stated reason excludes a commit — and here the rule's concern is inverted: the pin is the fix commit, so the question is not "what did you leave out" but "why is this commit not on the branch head". One sentence answering that (the pin is the PR's merge commit; the branch carries AMD staging merges it does not) would make the divergence expected rather than suspicious, and it matters for the next person bumping this pin forward. Note 2 — my first measurement was an artifact, and it is worth knowing whyMy first pass said "10,237 commits after the pin, empty merge-base, divergent histories" — all wrong. The That is the same failure mode as three other comparisons tonight, and it is worth a line in What a reviewer can check cheaply
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at e3d3104 -- my drafted-token ask is satisfied by an existing row, and the new commit raises one question that matters beyond this PR
On the ask I raised at the first head (a bit-exactness arm for the drafted tokens, not just acceptance): the body now carries test-backend-ops -b HRX0 -o MUL_MAT: 1717/1717, which is the ggml-reference comparison for the matmul family the shortlist kernel belongs to. I said naming that row would satisfy me, and it does -- with one caveat worth a word: say "including the q6_k token-1 shortlist", because a reader cannot tell from -o MUL_MAT that this specific kernel is inside the 1717. A row that covers it but does not say so answers the reviewer, not the next reader.
The tree-equality proof now covers both commits, which is the cleanest thing in the body: 6b9fcacb7 == #393's head 7e74ec3 (empty diff) and a353b7fee == #392's head 1ae4217ab (empty diff). So "these numbers are the pinned tree's own" is demonstrated twice rather than asserted once.
The question this raises, and it matters to this lane directly: #393 changes how every model's weights are buffered -- from a non-host import back to a file-mapped, host-typed buffer. That is a representation change for the whole fleet, not just zaya: the 27B now sits in 121 HRX0_Mapped buffers plus 1 CPU_Mapped. Mapped weights are host memory, so:
- does the HRX pool budget accounting (
hrx-1bit-safety, the 104 GiB GTT ceiling that decides whether a model fits) change for a model whose weights are now mapped rather than imported? A budget that charges GTT for an import and not for a mapping -- or the reverse -- moves every "does it fit" answer in this fleet, including #679's bar subject; - and does the GTT/RSS split at 18.8k context move for the 27B? My own bar work measures at the 104 GiB ceiling, so a change in what counts against it is the difference between a runnable subject and a refused window.
One line answering either would be worth more to me than the whole rest of the body, and it belongs here rather than in #393 because this PR is where the change reaches the engine.
Nothing else: the pin is one line plus the registry line, the measured numbers are the pinned tree's own, hrx-buffer-test 10/10 with the new case failing without the change is the right shape of evidence, and 4 READY on #393 is stated rather than assumed.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at e3d3104: READY
Re-verified at the second head, because the pin moved from one PR to two — and the first thing I hit was a stale binary of my own.
The enumeration, against the pin
$ git rev-list --count aae78ecdf..6b9fcacb7 2 ← "the range is two fork PRs"
6b9fcacb7 hrx: the loader's file mapping is a mapped weight again, not an import (zaya1-8b fails to load)
a353b7fee hrx: the MTP draft head takes the q6_k token1 shortlist
$ git ls-tree HEAD third_party/ | grep llama 6b9fcacb7fbd… == registry (40 hex)
$ tools/registry_build.py --check-pins rc 0
$ git rev-parse origin/1bit/hrx-vulkan-patched 6b9fcacb7 ← the pin IS the fork tip
Two, not one — and the body says so, which is the check #2431 and #2448 exist to make. The a353b7fee half is the MTP-shortlist commit I verified earlier (7 arms, mutation → 3 red); the new 6b9fcacb7 re-registers ggml_backend_hrx_model_buffer_from_mmap as a proper hook so an mmap-backed weight range is a mapped buffer again rather than an import.
Measured at this pin, with a device ticket
hrx-backend-test host: 76 passed, 0 failed, 23 known-fail, 0 unexpected-pass, 99 total
device: 14 passed, 0 failed, 0 known-fail, 0 unexpected-pass
the four MTP arms external_embedding / top_k_wider_than_candidates / sampler_chain / plain_lm_head_declines rc 0 each
0 unexpected-pass in both tiers: every known_fail label still describes a failure that happens, which is the property that makes the label table trustworthy.
The stale-binary trap, caught on myself
test-hrx-placement-classification reported 4 failures on the first run at this head — and after cmake --build --target test-hrx-placement-classification it reports "all cases ok (6 arms, both sides of STRICT)". The binary was 90 minutes old and built from a different pin; --target hrx-backend-test had not touched it. That is the rule this lane has written down twice ("rebuild before believing a ctest failure in the shared worktree"), and I still hit it — worth recording because a reader of a PR thread sees "4 failures" and stops.
What I did not run: the 27B decode gain and the zaya1-8b load the new commit fixes — both need a model and a longer ticket, and both are the fork PR's measurements rather than this pin's.
|
Persistent review updated to latest commit e3d3104 |
|
hw-gate at
The new FAIL row is not a new failure class. It is engine#2450's GET_ROWS x2 placement, the same message as on every other model. It shows now because zaya1 gets far enough to be placed. Under the ORCH rule (failing set identical to, or a subset of, main's), this is one masked row unmasked by a fix, so I am asking the merge loop to read it that way. If it needs to be literal, the row is #2450's and nothing in this pin can change it. Tracking:
|
|
Correction to my own review, published: I measured against the wrong branch. My verdict at The pin is exactly one commit behind the branch head, and that one commit is a test (llama.cpp#394, which is in review). So the pin is not "diverged and 689 commits stale" — it is one test commit short, which is the ideal state for a pin and changes the conclusion from "why not the head?" to "the head is one test PR away, and pinning the fix commit is right". Everything else in that review stands: both claimed fixes are in the pinned commit's history ( The lesson is the one this lane keeps relearning, and it is worth recording next to my own caveat about shallow clones: before comparing a sha to "the branch", check which branch the consumer follows. I had already written "do not answer 'is this sha on the branch' with a shallow clone" — this is the second half of the same sentence, and I got it wrong in the same review. |
What
The engine's llama.cpp pin moves
aae78ecdf→6b9fcacb7(first head pinneda353b7fee; the second commit adds llama.cpp#393, merged 11-15). That changesthird_party/llama.cpp, plus the matching line inregistry/architectures.json, as #2468 did.The range is two fork PRs: llama.cpp#392 and llama.cpp#393. #392: The MTP draft head (
--spec-type draft-mtp,1bit serve's default on a GGUF that carries its own MTP head) now takes the existingcommon.mul_mat.q6_k_token1_shortlist, instead of reading the full Q6_Koutput.weight(1.04 GB) for every draft token. The main model's lm_head is untouched.#393 (engine#2500): the loader's file mapping is a mapped weight again, not an import. Since llama.cpp#371, every mmap-backed weight was a non-host imported buffer. zaya1-8b, whose CCA conv weight view HRX does not take, aborted at load; that is the hw-gate
model zaya1-8brow. #393 has 4 READY. Its evidence: zaya1-8b loads and generates;hrx-buffer-test10/10, with the new case failing without the change; the 27B's text is identical and its tg64 is equal in an interleaved A/B.Measured
6b9fcacb7has the same tree as #393's head7e74ec3(empty diff), anda353b7feehad the same tree as #392's head (git diff --stat 1ae4217ab a353b7feeis empty). So these numbers are the pinned tree's own. strixhalo gfx1151, gpu-hold, gpu-kernels at the engine's pin2022afb74, Qwen3.8-27B Q4_K_M:hrx-backend-test: host 76/0/23, device 14/14.test-backend-ops -b HRX0 -o MUL_MAT: 1717/1717.--no-mtpgreedy text on two prompts: byte-identical toaae78ecdf.perf.pythrough1bit serve, servers in the order A B B A, two passes each.decode_tpsis 20.93 / 20.52 ataae78ecdfagainst 23.66 / 23.73 ata353b7fee, which is +13 %.decode_tps_2kis +12 %.draft-mtpn-max 3, 192 tokens at temp 0, with a control without the shortlist on the same base.draft_n/draft_n_acceptedare 244 / 109 for both arms on prompt 1, and 202 / 123 for both on prompt 2.Not done here
hw-gatebuilds this head and runs its checks.Owner: cl-hrxmem (DECODE-BW)
🤖 Generated with Claude Code