Eval bug: llama-server produces intermittent garbage output with Talkie architecture, llama-cli works fine

[JoeEnderman]

Name and Version

version: 9438 (d749821)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 5900x + 7900 XTX

Models

Talkie 1930 13b IT HF Q8

https://huggingface.co/mradermacher/talkie-1930-13b-it-hf-GGUF

Talkie Q8_O

https://huggingface.co/niklassheth/talkie-1930-13b-it-GGUF/tree/main

Problem description & steps to reproduce

llama-cli -cnv produces coherent output every time with the Talkie 1930 13B architecture (merged in #22596), but llama-server produces repetitive garbage on both /v1/chat/completions and /completion endpoints. The garbage is intermittent. The same prompt sometimes produces coherent output and sometimes produces degenerate repetition.

CLI works when I do this: llama-cli
-m talkie-1930-13b-it-hf.Q8_0.gguf
-ngl 999
-c 2048
--jinja
-cnv
-p "Hello, what year is it?"

And it always puts out clean responses like this: It is 1883.

When I run llama-server though I cannot get verified working output every single time.

llama-server
-m talkie-1930-13b-it-hf.Q8_0.gguf
-ngl 999
-c 2048
--jinja
--host 0.0.0.0
--port 11434
--parallel 1
--metrics

I tried curl:

curl http://localhost:11434/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "talkie",
"messages": [{"role": "user", "content": "Hello, what year is it?"}],
"max_tokens": 100
}'

And OpenWebUI, and the raw llama ui endpoints. I will get output like this:

? 2? 4? 2? 2? 4? 2? 2? 2? 2? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4? 4?

Or:

NRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNRNARNARNRARNRNRNRNERNRNERNERNRNRNERNNRNERNRNRNERNRNRNRNRNARNARARNERNRNRNERNRNERNRNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERNERN

Or:

an? a?? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a? a

I have tested both ggufs listed above as well as I quantized my own with the current build to see if that made a difference. All three behave identically.

I also tried disabling flash-attn, disabling prompt cache (--cache-ram 0), disabling warmup (--no-warmup), and passing an explicit --chat-template-file. None made a difference.

First Bad Commit

No response

Relevant log output

0.00.022.031 I device_info:
0.00.022.164 I - Vulkan0 : AMD Radeon RX 7900 XTX (RADV NAVI31) (24576 MiB, 22454 MiB free)
0.00.022.169 I - CPU : AMD Ryzen 9 5900X 12-Core Processor (32009 MiB, 32009 MiB free)
0.00.022.214 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.022.273 I srv init: using 23 threads for HTTP server
0.00.022.356 I srv start: binding port with default address family
0.00.023.710 I srv llama_server: loading model
0.00.023.714 I srv load_model: loading model '/home/joeenderman/AI_Models/talkie-Q8_0-self.gguf'
0.00.023.737 I common_init_result: fitting params to device memory ...
0.00.023.738 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.09.612.672 I srv load_model: initializing slots, n_slots = 1
0.09.660.235 W common_speculative_init: no implementations specified for speculative decoding
0.09.660.238 I slot load_model: id 0 | task -1 | new slot, n_ctx = 2048
0.09.660.298 I srv load_model: prompt cache is disabled - use --cache-ram N to enable it
0.09.660.300 I srv load_model: for more info see #16391
0.09.660.300 I srv load_model: context checkpoints enabled, max = 32, min spacing = 256
0.09.660.329 W srv init: --cache-idle-slots requires --kv-unified, disabling
0.09.661.716 I init: chat template, example_format: '<|system|>You are a helpful assistant<|end|><|user|>Hello<|end|><|assistant|>Hi there<|end|><|user|>How are you?<|end|><|assistant|>'
0.09.662.353 I srv init: init: chat template, thinking = 0
0.09.662.376 I srv llama_server: model loaded
0.09.662.382 I srv llama_server: server is listening on http://0.0.0.0:11434
0.09.662.387 I srv update_slots: all slots are idle

[yann-amd]

Your testing already eliminated flash-attn, prompt cache, warmup, and chat template — that rules out the typical server-specific variables. Given Talkie's unique embedding skip connection (#22596) and known outlier activations, the intermittent nature is the key constraint here.

One diagnostic that would cleanly split the remaining hypotheses: try your curl with "temperature": 0 (force greedy decoding). If greedy gives consistent clean output, the degeneration is likely a sampling interaction — CLI -cnv and the server API may end up with different effective sampling defaults when not explicitly specified. If garbage persists under greedy, it points to something in the server's eval path handling this architecture specifically.

[JoeEnderman]

Tried temperature: 0 as suggested. All three runs produced identical garbage: nothing but ? tokens, every time. With default temperature it was intermittent (sometimes coherent, sometimes garbage), but greedy decoding is broken 100% of the time.

[JoeEnderman]

Setting -ub 1 (batch size 1) on the server produces correct output. Default batch size produces garbage. The bug appears to be in how the server batches prompt evaluation with Talkie's embedding skip connections.

[yann-amd]

Your testing nails it down. temperature=0 going from intermittent to 100% garbage rules out sampling entirely, the logits themselves are corrupted. And -ub 1 fixing it points squarely at the batched evaluation path, exactly as you diagnosed.

Talkie's embedding skip connection (#22596) adds a scaled copy of the initial embedding back into each layer's output. That add operation's tensor indexing likely assumes a specific layout that holds for single-token eval but breaks when multiple tokens share a batch. The corruption pattern fits: batch dim misalignment in the skip connection would produce garbled logits without necessarily crashing.

To narrow it further: try -ub 2, then 4, 8 to find the exact threshold. Also worth testing -ngl 0 (CPU-only) to check whether this is Vulkan-specific or a general graph construction issue. cc @niklassheth since the skip connection plumbing in #22596 is probably where the batch handling needs a look.

[niklassheth]

I wasn't able to reproduce a difference between llama-server and llama-cli, but I was able to reproduce the garbage output on Vulkan. It looks like the ffn_down kernel can hit an fp16 overflow due to large outlier activations. The best fix would be to store the values in bf16 instead of fp16, which is proposed in #22571 but that PR only has a CUDA implementation.

The easiest fix for now is to set GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1, this does cause a ~20% slowdown for prompt processing. I also have a patch generated by Codex with the bf16 Vulkan fix, I'm no kernel expert but it looks reasonable.

Click here to show the GGML Vulkan patch

diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vecq.comp b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vecq.comp
index 6fe3e2dc0..249beee84 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vecq.comp
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vecq.comp
@@ -36,7 +36,8 @@ void iter(inout FLOAT_TYPE temp[NUM_COLS][NUM_ROWS], const uint first_row, const
         const uint b_qs_idx = tid % (32 / K_PER_ITER);
         const uint b_block_idx_outer = b_block_idx / 4;
         const uint b_block_idx_inner = b_block_idx % 4;
-        cache_b_ds = vec2(data_b[b_block_idx_outer].ds[b_block_idx_inner]);
+        const uint32_t b_ds = data_b[b_block_idx_outer].ds[b_block_idx_inner];
+        cache_b_ds = vec2(bf16_to_fp32(b_ds & 0xffffu), bf16_to_fp32(b_ds >> 16));
 
 #if QUANT_R == 2
         // Assumes K_PER_ITER == 8
diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mmq_funcs.glsl b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mmq_funcs.glsl
index 59931b04b..873a4e754 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/mul_mmq_funcs.glsl
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/mul_mmq_funcs.glsl
@@ -426,7 +426,8 @@ void block_b_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
         const uint ib_inner = ib % 4;
 
         if (iqs == 0) {
-            buf_b[buf_ib].ds = FLOAT_TYPEV2(data_b[ib_outer].ds[ib_inner]);
+            const uint32_t b_ds = data_b[ib_outer].ds[ib_inner];
+            buf_b[buf_ib].ds = FLOAT_TYPEV2(vec2(bf16_to_fp32(b_ds & 0xffffu), bf16_to_fp32(b_ds >> 16)));
         }
 
         const ivec4 values = data_b[ib_outer].qs[ib_inner * 2 + iqs];
diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/quantize_q8_1.comp b/ggml/src/ggml-vulkan/vulkan-shaders/quantize_q8_1.comp
index 7ea29a07e..47e34e933 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/quantize_q8_1.comp
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/quantize_q8_1.comp
@@ -111,9 +111,9 @@ void quantize(const uint wgid) {
 #endif
 
 #ifndef QBLOCK_X4
-        data_b[ib].ds = f16vec2(vec2(d, sum * d));
+        data_b[ib].ds = fp32_to_bf16(d) | (fp32_to_bf16(sum * d) << 16);
 #else
-        data_b[ibx4_outer].ds[ibx4_inner] = f16vec2(vec2(d, sum * d));
+        data_b[ibx4_outer].ds[ibx4_inner] = fp32_to_bf16(d) | (fp32_to_bf16(sum * d) << 16);
 #endif
     }
 }
diff --git a/ggml/src/ggml-vulkan/vulkan-shaders/types.glsl b/ggml/src/ggml-vulkan/vulkan-shaders/types.glsl
index f84d6f873..b4fa9c84a 100644
--- a/ggml/src/ggml-vulkan/vulkan-shaders/types.glsl
+++ b/ggml/src/ggml-vulkan/vulkan-shaders/types.glsl
@@ -211,32 +211,32 @@ struct block_q1_0
 
 struct block_q8_1
 {
-    f16vec2 ds;
+    uint32_t ds;
     int8_t qs[32];
 };
 
 struct block_q8_1_packed16
 {
-    f16vec2 ds;
+    uint32_t ds;
     int16_t qs[16];
 };
 
 struct block_q8_1_packed32
 {
-    f16vec2 ds;
+    uint32_t ds;
     int32_t qs[8];
 };
 
 // 4 blocks in one to allow 16-byte/128-bit alignment and loads
 struct block_q8_1_x4
 {
-    f16vec2 ds[4];
+    uint32_t ds[4];
     int32_t qs[32];
 };
 
 struct block_q8_1_x4_packed128
 {
-    f16vec2 ds[4];
+    uint32_t ds[4];
     ivec4 qs[8];
 };

[JoeEnderman]

GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 alone: no improvement on RX 7900 XTX / RADV NAVI31

bf16 shader patch alone (no -ub 1): significant improvement — primary responses often clean, but garbage still appears intermittently, particularly on context-reused requests

-ub 1 remains the only fully reliable workaround

This suggests the fp16 overflow is real and the patch is on the right track, but there may be additional code paths in the Vulkan backend hitting the same class of issue that aren't covered by these four shader files.

tested on build d749821

[yann-amd]

Good results, and the partial fix confirms the fp16 overflow diagnosis.

GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 not helping is expected — both paths share the same fp16 shared-memory bottleneck: disabling IDP does bypass the entire q8_1 pipeline (quantize_q8_1.comp → mul_mmq.comp). The fallback is the general tiled GEMM mul_mm.comp. However, mul_mm.comp declares its shared-memory tile buffers as FLOAT_TYPEV2 (mul_mm.comp, the shared FLOAT_TYPEV2 buf_b[BN * SHMEM_STRIDE] line), and on fp16-capable devices like the 7900 XTX, FLOAT_TYPEV2 resolves to f16vec2 (set in vulkan-shaders-gen.cpp, the FLOAT_TYPE definition when fp16=true). So when load_b_to_shmem (mul_mm_funcs.glsl) loads f32 activation values into buf_b, they're truncated to fp16 — any value exceeding 65504 overflows to inf/NaN, same as the mmq path. Both matmul paths hit the same fp16 precision ceiling.

On why the bf16 patch is partial — possible secondary overflow in the mmq shared memory path: @niklassheth's patch correctly fixes the quantize_q8_1.comp storage — using bf16 for the q8_1 scale/sum avoids the overflow at that stage. But in mul_mmq.comp, the B-matrix cache is declared as shared block_b_cache buf_b[BN * BK_STEP] (mul_mmq.comp), where block_b_cache (mul_mmq_shmem_types.glsl) contains:

struct block_b_cache {
    int32_t qs[8];
    FLOAT_TYPEV2 ds;
};

On fp16-enabled devices, FLOAT_TYPEV2 = f16vec2, so the q8_1 scale values that survived bf16 quantization are then re-stored as fp16 in shared memory during the matmul phase. If those scale values exceed 65504, they overflow here — a second truncation point downstream of the fix.

By contrast, mul_mat_vecq.comp (the -ub 1 vector path) stores ds into vec2 cache_b_ds — always fp32, in registers, never in fp16 shared memory (mul_mat_vecq.comp). This is why -ub 1 eliminates the issue entirely: the vector path has no fp16 precision bottleneck for activation scale values at any stage.

The fix likely needs the mmq shared memory ds to use fp32 precision — either by changing block_b_cache.ds to vec2, or by widening FLOAT_TYPEV2 to vec2 for the matmul paths on affected quant types. The mul_mm.comp fallback path would also need buf_b to be vec2 (or clamp values before storing) if we want IDP-disable to work as a workaround.

The hypothesis is: bf16 in quantize_q8_1 fixes the first overflow point, but the mmq shared memory's FLOAT_TYPEV2 ds field creates a second overflow point. -ub 1 avoids both because the vector path uses fp32 registers throughout.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.