Name and Version
./llama-cli --version
version: 9591 (62061f9)
built with GNU 16.1.1 for Linux x86_64
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Command line
OMP_PLACES="cores" OMP_PROC_BIND=spread BONSAI_SPECULATIVE=1 ./llama-server -m ./bonsai27b/Ternary-Bonsai-27B-Q2_0.gguf --alias "Qwen3.6-27B-MTP-GGUF" -c 32768 -ngl 99 --spec-draft-model ./bonsai27b/Ternary-Bonsai-27B-dspark-Q4_1.gguf --spec-type draft-dspark --spec-draft-n-max 4 -ngld 99 --jinja --chat-template-file ./bonsai27b/chat_template_v2.jinja -np 1 --host 0.0.0.0 --port 8090 --no-mmap -b 1024 -ub 512 --ctx-checkpoints 4 --webui-mcp-proxy
Problem description & steps to reproduce
System Specs
OS: CachyOS
CPU: Core i9-12900HK
RAM: 64GB DDR5-4800
GPU0: Intel Xe Graphics
GPU1: RTX 3080 Ti (Laptop) 16GB
Problem Description
When running draft-dspark speculative decoding on hardware with constrained VRAM (e.g., a 16GB GPU laptop setup), setting a high context window (-c 32768) on the main model causes a fatal cudaMalloc out-of-memory error during the graph reservation step.The scheduler dynamically catches the batch structure and forces the drafting array to bloat synchronously with the master model context parameter:
draft-dspark: raising draft ctx n_batch 1024 -> 32772 (full-context staging + block)
Because the fork lacks explicit CLI flags to step down the staging bounds of the speculative model alone, a massive ~5.88 GB compute buffer allocation is requested all at once, leading to an immediate boot crash. Offloading the draft layer to the CPU via --spec-draft-device none stops the crash but reduces generation performance from an exceptional 65+ T/S down to ~15 T/S due to heavy PCIe lane transfer bottlenecks.
Steps to Reproduce
Load a baseline dense model alongside the custom ternary weights (Q2_0) using a 32K context profile.Direct the execution stream to target draft-dspark fully inside GPU layer maps (-ngl 99, -ngld 99).Run the following command structure:
OMP_PLACES="cores" OMP_PROC_BIND=spread BONSAI_SPECULATIVE=1 ./llama-server \
-m ./bonsai27b/Ternary-Bonsai-27B-Q2_0.gguf \
-c 32768 \
-ngl 99 \
--spec-draft-model ./bonsai27b/Ternary-Bonsai-27B-dspark-Q4_1.gguf \
--spec-type draft-dspark \
--spec-draft-n-max 4 \
-ngld 99 \
-b 1024 -ub 512
First Bad Commit
No response
Relevant log output
Logs
0.02.287.240 I srv load_model: draft-dspark: raising draft ctx n_batch 1024 -> 32772 (full-context staging + block)
0.02.287.264 W llama_context: n_ctx_seq (32768) > n_ctx_train (4096) -- possible training context overflow
0.02.290.271 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 5888.53 MiB on device 0: cudaMalloc failed: out of memory
0.02.290.275 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 6174568960
0.02.290.276 E graph_reserve: failed to allocate compute buffers
0.02.290.470 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
Name and Version
./llama-cli --version
version: 9591 (62061f9)
built with GNU 16.1.1 for Linux x86_64
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server
Command line
Problem description & steps to reproduce
System Specs
Problem Description
When running draft-dspark speculative decoding on hardware with constrained VRAM (e.g., a 16GB GPU laptop setup), setting a high context window (-c 32768) on the main model causes a fatal cudaMalloc out-of-memory error during the graph reservation step.The scheduler dynamically catches the batch structure and forces the drafting array to bloat synchronously with the master model context parameter:
draft-dspark: raising draft ctx n_batch 1024 -> 32772 (full-context staging + block)Because the fork lacks explicit CLI flags to step down the staging bounds of the speculative model alone, a massive ~5.88 GB compute buffer allocation is requested all at once, leading to an immediate boot crash. Offloading the draft layer to the CPU via
--spec-draft-device nonestops the crash but reduces generation performance from an exceptional 65+ T/S down to ~15 T/S due to heavy PCIe lane transfer bottlenecks.Steps to Reproduce
Load a baseline dense model alongside the custom ternary weights (Q2_0) using a 32K context profile.Direct the execution stream to target draft-dspark fully inside GPU layer maps (-ngl 99, -ngld 99).Run the following command structure:
First Bad Commit
No response
Relevant log output
Logs