Eval bug: Error on loading Gemma 4 family with the new updates

[RyanDoesProgramming]

Name and Version

version: 0.3.0-dev (build 10744, commit 1f3d318)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

5070 ti+i9 14900k

Models

Gemma 4 family

Problem description & steps to reproduce

When I use MTP on gemma 4 family whether its the e4b or the 26b MOE

First Bad Commit

I didnt have this issue on previous versions

Relevant log output

Logs

[New LWP 219412]
[New LWP 219411]
[New LWP 219410]
[New LWP 219409]
[New LWP 219408]
[New LWP 219407]
[New LWP 219406]
[New LWP 219405]
[New LWP 219404]
[New LWP 219403]
[New LWP 219402]
[New LWP 219401]
[New LWP 219400]
[New LWP 219399]
[New LWP 219398]
[New LWP 219397]
[New LWP 219396]
[New LWP 219395]
[New LWP 219394]
[New LWP 219393]
[New LWP 219392]
[New LWP 219391]
[New LWP 219390]
[New LWP 219389]
[New LWP 219388]
[New LWP 219387]
[New LWP 219386]
[New LWP 219385]
[New LWP 219384]
[New LWP 219383]
[New LWP 219382]
[New LWP 219381]
[New LWP 219380]
[New LWP 219379]
[New LWP 219378]
[New LWP 219374]
[New LWP 219373]
[New LWP 219372]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
⚠️ warning: 56   ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x000076121d4a061c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
⚠️ warning: 49   ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x000076121d51cd4f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠️ warning: 30   ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x000076121db356f3 in ggml_print_backtrace () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libggml-base.so.0
#5  0x000076121db358a6 in ggml_abort () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libggml-base.so.0
#6  0x000076121cbde1e7 in llama_model_base::load_hparams(llama_model_loader&) () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama.so.0
#7  0x000076121cae0a57 in llama_model_load(gguf_context*, void (*)(ggml_tensor*, void*), void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, _IO_FILE*, llama_model_params&) () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama.so.0
#8  0x000076121cae1c65 in llama_model_load_from_file_impl(gguf_context*, void (*)(ggml_tensor*, void*), void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, _IO_FILE*, llama_model_params) () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama.so.0
#9  0x000076121cae2a09 in llama_model_load_from_file () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama.so.0
#10 0x000076121d042340 in common_get_device_memory_data_impl(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama-common.so.0
#11 0x000076121d042b19 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level)::{lambda(std::vector<llama_device_memory_data, std::allocator<llama_device_memory_data> >&)#1}::operator()(std::vector<llama_device_memory_data, std::allocator<llama_device_memory_data> >&) const () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama-common.so.0
#12 0x000076121d043a66 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, common_fit_extra_model const*, ggml_log_level) () from /home/ryan-norouzi/Ai/Projects/Ai_wrapper/llama.cpp/build/bin/libllama-common.so.0

[tinek78]

It's not just Gemma4, since the latest MTP PR (#28123) NONE of the models I tried (Qwen 3.6 9B, Gemma4 12B, Qwen3.8 27B) to load with MTP heads integrated actually load MTP, they all error out with
"llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
common_speculative_init_result: failed to create MTP context"

Loading MTP as a sidecar also doesn't work, but loading dflash2 as sidecar works without issues.

All of that on CUDA (RTX 5090) with plenty of free vRAM, so definitely not a lack of vRAM being a problem here.

[byNick33]

I already reproduced this with Qwen 3.8 27B and Gemma4 31B unsloth quants with the following message:
/opt/llama.cpp/src/llama-model.cpp:1227: GGML_ASSERT(hparams.n_layer_nextn < hparams.n_layer_all) failed
The reason it fails to load with MTP lies in commit 9d81721 (pr #28159) and was fixed in commit 73159c3 (pr #28183).

[tinek78]

Confirmed, all works good now, as far as I can see.