Repository navigation
llama-context: a DFlash draft with no target context fails to create instead of aborting (27B --dflash load) - #397
Conversation
… not aborts (engine 27B --dflash) llama-server's memory fit measures the speculative draft as an extra model (common_fit_params, from the ggml-org#2138 merge). That probe builds the draft's context with no ctx_other. ggml-org#681 lets EAGLE3/DFLASH/QWEN4EXP drafts through with a warning so fitting can measure them, but the DFlash graph takes the target's tok_embd/output through ctx_other and asserts on it (src/models/dflash.cpp:944) while sched_reserve builds it. GGML_ASSERT aborts, so the fitter's catch-and-fit-alone never runs, and `1bit serve --dflash` on Qwen3.8-27B + DFlash2 q8_0 exits before it is ready. A DFlash draft without its own tok_embd/output now throws in the context constructor, as Gemma4Assistant already does. llama_init_from_model returns nullptr and the fitter fits the target alone. The real draft context (common_speculative_init) passes ctx_other and is unchanged. EAGLE3 and QWEN4EXP keep ggml-org#681's warn-and-continue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY at The throw is caught, which is the whole pointcommon/fit.cpp:303 try {
common/fit.cpp:306 } catch (const std::runtime_error & e) {Without that catch this would turn The scope matches the defect, measured
It follows the file's own precedent, one arch oversrc/llama-context.cpp:242 if (params.ctx_other == nullptr) {
src/llama-context.cpp:244 throw std::runtime_error("Gemma4Assistant requires ctx_other to be set (this warni…Same exception type, same shape, same place. New code that looks like the line above it is new code nobody has to argue about later. PlacementNested inside One note, not an askThere is no arm for this path (+7 lines, no test change), and there cannot easily be one: it needs a DFlash draft GGUF and a fitter run with an extra model. So the evidence is the 27B re-measure that aborted before the change. Worth one line saying that explicitly — "exercised by |
1bit serve --dflashaborts at load on engine main: Qwen3.8-27B UD-Q4_K_XL + DFlash2 q8_0,--device hrx. Found by the 27B re-measure (orchestrator 11-15).Why
ctx_other.llama_contextconstructor lets EAGLE3 / DFLASH / QWEN4EXP drafts through with a warning, so fitting can measure them.tok_embd/outputtakes them from the target throughctx_other. It asserts on that whilesched_reservebuilds its graph.GGML_ASSERTaborts, so the fitter'scatch (std::runtime_error)and fit-the-target-alone fallback (common/fit.cpp:306) never run.Fix
tok_embd/outputand noctx_other, the same way Gemma4Assistant does a few lines above.llama_init_from_modelturns the exception intonullptr, and the fitter logs "fitting without it".common_speculative_init) passesctx_otherand is unchanged.Evidence (strixhalo, gfx1151, inside gpu-hold tickets)
Constructor, the drafter alone (no target):
llama-bench -m Qwen3.8-27B-DFlash2-q8_0.gguf -p 8 -n 0 --device HRX0 -ngl 99GGML_ASSERT(cparams.ctx_other != nullptr)failed to initialize the context: DFlash draft without tok_embd/output needs ctx_other (normal during memory fitting)Serve:
1bit serve -m Qwen3.8-27B-UD-Q4_K_XL.gguf --device hrx --dflash Qwen3.8-27B-DFlash2-q8_0.gguf, with engine main b61d83b4, this fork at 6b9fcac + this change (same tree), gk 2022afb74.failed to measure the memory of the extra model, fitting without it, and load continues past it."Qcur-0" is ROPE f32 ne=[128,32,2,1] and is not placed on HRX; 11 of 765 nodes off HRX (GET_ROWS x1, ROPE x10).pre-allocated tensor (blk.0.attn_conv_base (view)) in a buffer () that cannot run the operation) sat between the two; hrx: the loader's file mapping is a mapped weight again, not an import -- zaya1-8b has not loaded since #371 (engine#2500) #393 fixed it.Not run:
test-llama-archsskips DFLASH, and building a DFlash fixture without the target's tensors is more than this change needs.The change is host-side C++ in one constructor branch; no kernel is touched.
🤖 Generated with Claude Code