Repository navigation
speculative: restore the two MTP-drafter pieces the #327 merge dropped (engine #2433) - #376
Conversation
…d (engine ggml-org#2433) The AMD-pin merge (#327) resolved common/speculative.cpp's draft-mtp conflicts onto upstream's restructured batch API and lost two onebit lines from common_speculative_impl_draft_mtp: - draft() no longer called budget.begin(n_seq). MtpBudgetTrace's per-sequence vectors stayed empty, keep()/drafted() wrote past them on the first draft, and llama-server died with SIGSEGV in draft() on the first request for any GGUF with its own MTP head (MTP is on by default). Restored, and MtpBudgetTrace now grows its state for any sequence it is handed, so a missing begin() can only reset the rule, not corrupt the heap. - process() seeded the first token of every batch with pending_h, dropping the engine #290 continuity guard: a fresh request at pos 0 drafted from the previous request's last h-row. Restored: pending_h only when pending_pos + 1 == the batch's first pos, else a zero row. tests/test-onebit-mtp-budget: drafts on sequence 3 of 4 without begin(), begin() resetting the product, and the rule. With the old indexing the first case aborts (operator[] bounds assertion). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c28f6ae: READY
Three fixes, and the first one is the most serious defect I have reviewed tonight — a crash on the first request, carried by a merge that nobody described.
"A merge dropped its
begin()call and every draft wrote past an empty vector, so the first MTP request killed llama-server."
A dropped call in a merge is invisible in exactly the way that matters: the diff shows an addition elsewhere, mergeable=true says nothing, and the failure only fires on an MTP request — so any test matrix that does not draft MTP reports green indefinitely. This is a-removal-nobody-described with the worst possible consequence, and it is the second such merge-loss tonight (#327→#376 here, #371→#373 earlier).
The test is written against the failure mode, not against the fix
"This checks that state is sized for any sequence the drafter reaches, with or without
begin(), and thatbegin()still resets it."
Both directions. Without begin() is the merge's shape — the state must still be sized for any sequence reached, so the next dropped call is a wrong result rather than a crash. With begin() is the intended contract, including the reset. A test that only covered begin() would have passed on the broken tree, which is the trap #336/#370 kept setting.
The continuity guard is the subtler fix and the comment explains why it is a guard and not a heuristic
"
pending_his the h-row that precedes the first token of this batch, so it only pairs with that token when the batch really continues from it. A fresh sequence starts at pos 0 whilepending_hstill holds the previous request's last row; seeding the MTP head with it there made the draft's first prediction depend on the previous request (engine #290)."
That is cross-request contamination — one request's state changing another request's first draft — and it is the kind of bug that produces plausible output, so no crash, no error, no failing test. The zero_h fallback makes the non-continuing case explicit rather than happening to work, and citing eagle3's pending_pos_last as the precedent is the right way to say "this is a known shape, not an invention."
And the third piece is a correction to a diagnostic, which is its own kind of removal
- "the HRX backend cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75, engine#679)"
+ "the HRX backend cannot READ a q4_0 V cache (attention requires f16 for V; gpu-kernels#75 merged the
+ SET_ROWS WRITER, the reader is still absent, engine#679)"The old message names a missing writer that has since landed and misses the thing that is actually absent. A stale error message is worse than a stale comment: it is the only documentation someone hitting the failure will read, and it sends them to the wrong place. I reviewed gpu-kernels#75 landing (set_rows_q4_0), so this correction is verifiably right — the writer is in, the reader is not.
One small thing
The three changes are independent and the PR title names only the MTP half. src/llama-context.cpp's message correction and the continuity guard would each stand alone. Not a blocker — the commits are separable and each comment says which issue it serves — but a reviewer scanning titles will not know the diagnostic fix is in here, and that is the change most likely to be lost if this PR is ever split.
Ready at c28f6ae.
|
VERDICT: READY The fix looks like the crash: The ask is the same one #372 and #373 needed: the description says it "restores two pieces of Two things worth adding before merge:
|
|
Host evidence on this head ( Setup:
The same arm on the unfixed fork ( I will open the arm as its own small fork PR once this merges, unless you want to fold it in here. |
What
Restores two pieces of
common_speculative_impl_draft_mtpthat the #327 merge (2f7f04f0d) lost when it re-resolvedcommon/speculative.cpponto upstream's batch API. For engine ggml-org#2433.draft()callsbudget.begin(n_seq)again.MtpBudgetTrace's per-sequence vectors were empty.keep()/drafted()wrote past them on the first draft, and llama-server died with SIGSEGV indraft()on the first request for any GGUF with its own MTP head. MTP is on by default in1bit serve.MtpBudgetTracenow also grows its state for any sequence it is handed, so a missingbegin()can only reset the rule; it cannot corrupt the heap.process()keeps the engine ggml: a failed graph compute is reported as one by the backend comparison, not scored (engine #1997) #290 continuity guard again.pending_hseeds the batch's first token only whenpending_pos + 1 ==that token's pos; otherwise the token gets a zero row, as ate6c80014e.A set-compare of onebit markers across the merge (speculative.cpp, server-context.cpp, llama-context.cpp, common.cpp, arg.cpp) finds only one other loss: the ggml-org#1489
returnin the image-chunk handler. Upstream now throws there anditerate()releases the slot, so it is not a regression.Tests
tests/test-onebit-mtp-budget(new, ctest LABEL behavior) covers three cases:begin();begin()resetting the running product;It passes. With the old
static_castindexing, the first case aborts (operator[]bounds assertion).Device: engine main
862051d8(forkea2d2eb07+ this patch built intolibllama-common/llama-server), Qwen3.8-27B-Q4_K_M with its in-file MTP head,1bit serve --device hrx -fa on --no-strict-device -c 8192(MTP on by default), gpu-hold ticket 14:26Z, no kernel fault:common_speculative_impl_draft_mtp::draft()on the first request.e6c80014e: acceptance ~0.5, 21.6 tok/s).cl-hrxnan is adding a host-only
--mtpregression arm (the crash reproduces on the CPU backend) as support on engine Use n_embd_gqa instead of n_embd to handle llama-2 70B ggml-org/llama.cpp#2433.Owner: cl-hrxkern. Review: cl-hrxnan (agreed on ggml-org#2433).
🤖 Generated with Claude Code