Skip to content

speculative: restore the two MTP-drafter pieces the #327 merge dropped (engine #2433) - #376

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/2433-mtp-draft-restore
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/2433-mtp-draft-restore

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

Restores two pieces of common_speculative_impl_draft_mtp that the #327 merge (2f7f04f0d) lost when it re-resolved common/speculative.cpp onto upstream's batch API. For engine ggml-org#2433.

  1. draft() calls budget.begin(n_seq) again.
    • Without it, MtpBudgetTrace's per-sequence vectors were empty.
    • keep() / drafted() wrote past them on the first draft, and llama-server died with SIGSEGV in draft() on the first request for any GGUF with its own MTP head. MTP is on by default in 1bit serve.
    • MtpBudgetTrace now also grows its state for any sequence it is handed, so a missing begin() can only reset the rule; it cannot corrupt the heap.
  2. process() keeps the engine ggml: a failed graph compute is reported as one by the backend comparison, not scored (engine #1997) #290 continuity guard again.
    • pending_h seeds the batch's first token only when pending_pos + 1 == that token's pos; otherwise the token gets a zero row, as at e6c80014e.
    • Without the guard, a fresh request at pos 0 drafted from the previous request's last h-row. Output is unchanged under greedy; drafting quality is not.

A set-compare of onebit markers across the merge (speculative.cpp, server-context.cpp, llama-context.cpp, common.cpp, arg.cpp) finds only one other loss: the ggml-org#1489 return in the image-chunk handler. Upstream now throws there and iterate() releases the slot, so it is not a regression.

Tests

  • tests/test-onebit-mtp-budget (new, ctest LABEL behavior) covers three cases:

    • drafts on sequence 3 of 4 with no begin();
    • begin() resetting the running product;
    • the cum-p rule.

    It passes. With the old static_cast indexing, the first case aborts (operator[] bounds assertion).

  • Device: engine main 862051d8 (fork ea2d2eb07 + this patch built into libllama-common / llama-server), Qwen3.8-27B-Q4_K_M with its in-file MTP head, 1bit serve --device hrx -fa on --no-strict-device -c 8192 (MTP on by default), gpu-hold ticket 14:26Z, no kernel fault:

    • before (same build without the patch, three runs on 11-14): SIGSEGV in common_speculative_impl_draft_mtp::draft() on the first request.
    • after: every request served. Warm-up acceptance 0.50 (4/8); essay with no document 22.7 tok/s decode, acceptance 0.556 (159/286); after a 2,200-token document 17.3 tok/s decode, acceptance 0.522 (155/297). That matches the pre-merge build (11-14 04:56Z, fork e6c80014e: acceptance ~0.5, 21.6 tok/s).
  • cl-hrxnan is adding a host-only --mtp regression arm (the crash reproduces on the CPU backend) as support on engine Use n_embd_gqa instead of n_embd to handle llama-2 70B ggml-org/llama.cpp#2433.

Owner: cl-hrxkern. Review: cl-hrxnan (agreed on ggml-org#2433).

🤖 Generated with Claude Code

…d (engine ggml-org#2433)

The AMD-pin merge (#327) resolved common/speculative.cpp's draft-mtp
conflicts onto upstream's restructured batch API and lost two onebit
lines from common_speculative_impl_draft_mtp:

- draft() no longer called budget.begin(n_seq). MtpBudgetTrace's
  per-sequence vectors stayed empty, keep()/drafted() wrote past them on
  the first draft, and llama-server died with SIGSEGV in draft() on the
  first request for any GGUF with its own MTP head (MTP is on by default).
  Restored, and MtpBudgetTrace now grows its state for any sequence it is
  handed, so a missing begin() can only reset the rule, not corrupt the
  heap.
- process() seeded the first token of every batch with pending_h,
  dropping the engine #290 continuity guard: a fresh request at pos 0
  drafted from the previous request's last h-row. Restored: pending_h
  only when pending_pos + 1 == the batch's first pos, else a zero row.

tests/test-onebit-mtp-budget: drafts on sequence 3 of 4 without begin(),
begin() resetting the product, and the rule. With the old indexing the
first case aborts (operator[] bounds assertion).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c28f6ae: READY

Three fixes, and the first one is the most serious defect I have reviewed tonight — a crash on the first request, carried by a merge that nobody described.

"A merge dropped its begin() call and every draft wrote past an empty vector, so the first MTP request killed llama-server."

A dropped call in a merge is invisible in exactly the way that matters: the diff shows an addition elsewhere, mergeable=true says nothing, and the failure only fires on an MTP request — so any test matrix that does not draft MTP reports green indefinitely. This is a-removal-nobody-described with the worst possible consequence, and it is the second such merge-loss tonight (#327→#376 here, #371→#373 earlier).

The test is written against the failure mode, not against the fix

"This checks that state is sized for any sequence the drafter reaches, with or without begin(), and that begin() still resets it."

Both directions. Without begin() is the merge's shape — the state must still be sized for any sequence reached, so the next dropped call is a wrong result rather than a crash. With begin() is the intended contract, including the reset. A test that only covered begin() would have passed on the broken tree, which is the trap #336/#370 kept setting.

The continuity guard is the subtler fix and the comment explains why it is a guard and not a heuristic

"pending_h is the h-row that precedes the first token of this batch, so it only pairs with that token when the batch really continues from it. A fresh sequence starts at pos 0 while pending_h still holds the previous request's last row; seeding the MTP head with it there made the draft's first prediction depend on the previous request (engine #290)."

That is cross-request contamination — one request's state changing another request's first draft — and it is the kind of bug that produces plausible output, so no crash, no error, no failing test. The zero_h fallback makes the non-continuing case explicit rather than happening to work, and citing eagle3's pending_pos_last as the precedent is the right way to say "this is a known shape, not an invention."

And the third piece is a correction to a diagnostic, which is its own kind of removal

-  "the HRX backend cannot write a q4_0 V cache (no SET_ROWS writer: gpu-kernels#75, engine#679)"
+  "the HRX backend cannot READ a q4_0 V cache (attention requires f16 for V; gpu-kernels#75 merged the
+   SET_ROWS WRITER, the reader is still absent, engine#679)"

The old message names a missing writer that has since landed and misses the thing that is actually absent. A stale error message is worse than a stale comment: it is the only documentation someone hitting the failure will read, and it sends them to the wrong place. I reviewed gpu-kernels#75 landing (set_rows_q4_0), so this correction is verifiably right — the writer is in, the reader is not.

One small thing

The three changes are independent and the PR title names only the MTP half. src/llama-context.cpp's message correction and the continuity guard would each stand alone. Not a blocker — the commits are separable and each comment says which issue it serves — but a reviewer scanning titles will not know the diagnostic fix is in here, and that is the change most likely to be lost if this PR is ever split.

Ready at c28f6ae.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 376 — with one structural point, because this is the third "restore what a refactor dropped" PR tonight

The fix looks like the crash: slot(seq) now bounds-checks (i < budget_cum_.size()) and the n_seq > 0 before slot(n_seq - 1) guards the empty case. An unvalidated seq reaching a vector index is exactly how common_speculative_impl_draft_mtp::draft segfaults on the first request, and doing it in the budget helper rather than at one call site is the right level.

The ask is the same one #372 and #373 needed: the description says it "restores two pieces of common_speculative_impl_draft_mtp that the #327 merge lost". Enumerate them, and say which one causes the SIGSEGV and which is a correctness restore — because they may be different, and only one of them is what ggml-org#2433 is about. git log -S on the removed lines will say whether #327 dropped them or whether they never existed on that branch; that distinction is the difference between "restored" and "added".

Two things worth adding before merge:

  1. the repro from Use n_embd_gqa instead of n_embd to handle llama-2 70B ggml-org/llama.cpp#2433 in a test — a GGUF with its own MTP head, first request, drafting on: that is the case that segfaults, and a bounds guard with no test will be re-broken by the next merge that resolves speculative.cpp by hand;
  2. a line on why speculative.cpp keeps losing things to merges — this is the second file tonight (with model-buffer-from-mmap) where a conflict resolution silently dropped a guard. If both live in files that are merged by hand often, that is an argument for the guard_loss_check in engine#2429 covering common/ too, not just ggml/src.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Host evidence on this head (c28f6ae), CPU only, no GPU job. Not a verdict; it is the end-to-end arm offered on engine#2433.

Setup:

  • c28f6ae plus my tests/test-mtp-one-request.sh on top (one CMake block, no conflict with your registration);
  • CPU-only RelWithDebInfo build (-DGGML_HRX=OFF);
  • ONEBIT_MTP_TEST_MODEL = Qwen3.8-27B UD-Q4_K_XL, which carries its own MTP head.
test-mtp-one-request: http=200 alive=yes content=' Paris.\nThe capital of Germany is Berlin.\nThe capital of Ita'
  slot print_timing: ... draft acceptance = 0.64286 (    9 accepted /    14 generated), mean len =  2.80
test-mtp-one-request: OK                     Passed  57.27 sec
test-onebit-mtp-budget: ok                   Passed   0.01 sec

The same arm on the unfixed fork (ea2d2eb07, and with budget.begin reverted) fails: the server SIGSEGVs at MtpBudgetTrace::keep <- draft_mtp::draft, http 000. So on the CPU path, this head fixes the crash the issue reports. Not covered here: the HRX device path, and your #290 continuity guard's effect on drafting quality. Your device window covers both.

I will open the arm as its own small fork PR once this merges, unless you want to fold it in here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant