Skip to content

llama-context: a DFlash draft with no target context fails to create instead of aborting (27B --dflash load) - #397

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/dflash-fit-probe
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
1bit/dflash-fit-probe

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

1bit serve --dflash aborts at load on engine main: Qwen3.8-27B UD-Q4_K_XL + DFlash2 q8_0, --device hrx. Found by the 27B re-measure (orchestrator 11-15).

src/models/dflash.cpp:944: GGML_ASSERT(cparams.ctx_other != nullptr) failed
  llama_model_dflash::graph ... llama_context::graph_reserve ... resolve_fused_ops ... sched_reserve
  llama_context::llama_context ... llama_init_from_model
  common_get_device_memory_data_impl ... common_params_fit_impl ... common_fit_apply ... server_context_impl::load_model

Why

  • llama-server's memory fit measures the draft as an extra model. The extra model came in with the ci : switch threads to 1 ggml-org/llama.cpp#2138 merge (b1fd5e8).
  • That probe builds the draft's context with no ctx_other.
  • q4_1/f16 model is slow ggml-org/llama.cpp#681's branch in the llama_context constructor lets EAGLE3 / DFLASH / QWEN4EXP drafts through with a warning, so fitting can measure them.
  • A DFlash draft without its own tok_embd / output takes them from the target through ctx_other. It asserts on that while sched_reserve builds its graph.
  • GGML_ASSERT aborts, so the fitter's catch (std::runtime_error) and fit-the-target-alone fallback (common/fit.cpp:306) never run.

Fix

  • The constructor now throws for a DFLASH draft that has no tok_embd / output and no ctx_other, the same way Gemma4Assistant does a few lines above.
  • llama_init_from_model turns the exception into nullptr, and the fitter logs "fitting without it".
  • The real draft context (common_speculative_init) passes ctx_other and is unchanged.
  • EAGLE3 and QWEN4EXP keep q4_1/f16 model is slow ggml-org/llama.cpp#681's warn-and-continue; their graphs do not read the target here.

Evidence (strixhalo, gfx1151, inside gpu-hold tickets)

Constructor, the drafter alone (no target):
llama-bench -m Qwen3.8-27B-DFlash2-q8_0.gguf -p 8 -n 0 --device HRX0 -ngl 99

libllama result
without this change rc 134, GGML_ASSERT(cparams.ctx_other != nullptr)
with it rc 1, failed to initialize the context: DFlash draft without tok_embd/output needs ctx_other (normal during memory fitting)

Serve: 1bit serve -m Qwen3.8-27B-UD-Q4_K_XL.gguf --device hrx --dflash Qwen3.8-27B-DFlash2-q8_0.gguf, with engine main b61d83b4, this fork at 6b9fcac + this change (same tree), gk 2022afb74.

Not run:

  • the full HRX ctest;
  • a host-only unit test. test-llama-archs skips DFLASH, and building a DFlash fixture without the target's tensors is more than this change needs.

The change is host-side C++ in one constructor branch; no kernel is touched.

🤖 Generated with Claude Code

… not aborts (engine 27B --dflash)

llama-server's memory fit measures the speculative draft as an extra model
(common_fit_params, from the ggml-org#2138 merge). That probe builds the draft's
context with no ctx_other. ggml-org#681 lets EAGLE3/DFLASH/QWEN4EXP drafts through
with a warning so fitting can measure them, but the DFlash graph takes the
target's tok_embd/output through ctx_other and asserts on it
(src/models/dflash.cpp:944) while sched_reserve builds it. GGML_ASSERT aborts,
so the fitter's catch-and-fit-alone never runs, and `1bit serve --dflash` on
Qwen3.8-27B + DFlash2 q8_0 exits before it is ready.

A DFlash draft without its own tok_embd/output now throws in the context
constructor, as Gemma4Assistant already does. llama_init_from_model returns
nullptr and the fitter fits the target alone. The real draft context
(common_speculative_init) passes ctx_other and is unchanged. EAGLE3 and
QWEN4EXP keep ggml-org#681's warn-and-continue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 06f8140 — and I checked the three things that decide whether "abort becomes a throw" actually improves anything. All three hold.

The throw is caught, which is the whole point

common/fit.cpp:303   try {
common/fit.cpp:306   } catch (const std::runtime_error & e) {

Without that catch this would turn GGML_ASSERT's abort into an uncaught throw — std::terminate, no better and harder to read. With it, the fitter falls back to fitting the target alone, exactly as the comment claims.

The scope matches the defect, measured

GGML_ASSERT(cparams.ctx_other …) appears at six sites, all in src/models/dflash.cpp (:944, :1080, :1193, :1290, :1375, :1541) and nowhere in eagle3.cpp or qwen4exp.cpp. So narrowing to arch == LLM_ARCH_DFLASH inside the block that already covers all three is not an arbitrary slice — it is precisely where the asserts live, and the siblings need nothing.

It follows the file's own precedent, one arch over

src/llama-context.cpp:242   if (params.ctx_other == nullptr) {
src/llama-context.cpp:244       throw std::runtime_error("Gemma4Assistant requires ctx_other to be set (this warni…

Same exception type, same shape, same place. New code that looks like the line above it is new code nobody has to argue about later.

Placement

Nested inside if (model.arch == EAGLE3 || DFLASH || QWEN4EXP) { if (tok_embd == nullptr || output == nullptr) { … } } — i.e. before graph_reserve/sched_reserve can reach dflash.cpp's assert. A guard that fires later would be a race with the abort, not a replacement for it.

One note, not an ask

There is no arm for this path (+7 lines, no test change), and there cannot easily be one: it needs a DFlash draft GGUF and a fitter run with an extra model. So the evidence is the 27B re-measure that aborted before the change. Worth one line saying that explicitly — "exercised by 1bit serve --dflash with a DFlash extra model during fitting; no unit arm, the model is not in the test tree" — because the next reader of llama-context.cpp will assume an assert-shaped guard has a test behind it, and this one is justified by a measured abort instead.

@bong-water-water-bong
bong-water-water-bong merged commit 8aa770f into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/dflash-fit-probe branch November 15, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant