[osabnis]

Overview

Fix incorrect results with -fa off in the OpenVINO backend, improve decode performance at long context for models with grouped-query attention (GQA), and handle the incompatibility between transposed V caches and stateful execution.

This change:

• Fixes incorrect KV-cache slicing for the transposed V cache used by -fa off.
• Avoids replicating K/V for every query head during GQA.
• Allows the GPU plugin to fuse the attention scale, mask, and softmax with the MatMuls.
• Handles models whose V-cache dimensions differ between layers.
• Falls back to stateless execution when a transposed V cache is used with stateful execution.
• Keeps the flat [1, 1, ctx, n_heads_kv * head_size] KV-cache layout with -fa off; -fa on continues to use the 4-D layout from the KV-cache change dd04231.

Fix -fa off V-cache slicing

With -fa off, llama.cpp stores the V cache in transposed form: each row contains one value element, with tokens laid out along the row. try_make_kv_sliced_tensor() previously assumed that each row represented one token and therefore sliced the cache to its first n_kv rows.

That is incorrect for the transposed V cache: when n_kv < kv_size, the slice selects the wrong data and causes a large accuracy loss at depth.

GgmlOvDecoder::is_kvcache_token_rows() now detects caches whose SET_ROWS writes rows narrower than the cache row. KV slicing is skipped for these caches, so the full transposed V cache is used.

This fixes the accuracy loss with -fa off when the KV cache is not full.

Speed up GQA decode

For models with grouped KV heads, the previous implementation replicated K and V for every query head before the KQ and KQV MUL_MAT operations. This unnecessarily copies the entire K/V cache for every layer and token.

The new implementation groups the query heads belonging to each KV head along the rows using a reshape, then performs one MatMul per KV head. This avoids K/V replication while producing the same grouped-attention computation.

The resulting 4-D attention scores are passed through SOFT_MAX, which applies the scale, tiled causal mask, and softmax together. This allows the GPU plugin to fuse these operations with the MatMuls. Keeping the mask as a separate Add operation made prefill slower.

On GPU, the attention mask is clamped to -30000, matching the existing behavior in flash_attn_ext. This avoids NaNs in the fused kernel when an entire block contains -inf.

Handle different V-cache sizes across layers

Some models use different V head sizes in different layers. For example, Gemma-4 E2B uses a V head size of 512 in full-attention layers and 256 in sliding-window layers.

The transposed V cache is therefore allocated for the widest layer. When reading the cache, the implementation now uses only the n_heads * head_size rows required by the current layer.

Fall back to stateless execution

The stateful KV-cache path does not support a transposed V cache.

Following Suhas Somashekar's approach in ggml-org/llama.cpp#30350, the decoder now detects a transposed V cache and, when GGML_OPENVINO_STATEFUL_EXECUTION=1, falls back to stateless execution and logs a warning.

The conversion check in set_rows.cpp remains as a safety net.

This behavior is also documented in docs/backend/OPENVINO.md.

Keep the flat KV-cache layout with -fa off

The 4-D KV-cache layout introduced by dd04231 breaks MUL_MAT attention in Phi-3.5 mini and Phi-4 mini with oneDNN:

could not construct a memory descriptor using strides

When the graph writes a transposed V cache, the KV cache therefore continues to use the flat [1, 1, ctx, n_heads_kv * head_size] layout with -fa off.

-fa on continues to use the 4-D layout introduced by dd04231.

The root cause of the Phi-3.5 mini / Phi-4 mini oneDNN failure is left for a follow-up change.

Validation

Arc B390 (Panther Lake) GPU, Q4_K_M, OpenVINO master.

Accuracy

KL divergence against the CPU backend, decode at context 1024 (-b 1), -fa off:

Model	Before	This PR
Llama-3.2 1B	5.39	0.0406
SmolLM2 1.7B	20.5	0.0728

Perplexity on wikitext-2, -c 1024, 4 chunks:

Model	-fa off	-fa on
Llama-3.2 1B	14.8066	14.8127
SmolLM2 1.7B	9.3999	9.4038
Phi-4 mini	10.0698	10.0684
Qwen3.5 4B	11.1087	11.1084
Phi-3.5 mini	5.8312	5.8313
Gemma-4 E2B	170.7457	170.7349

With GGML_OPENVINO_STATEFUL_EXECUTION=1 and -fa off, Llama-3.2 1B, SmolLM2 1.7B, Qwen3.5 4B, Phi-4 mini, and Phi-3.5 mini all print the fallback warning and produce the same perplexity as stateless execution.

test-backend-ops: 8024/8024 passed.

Performance

llama-bench -fa 0, decode (-n 64, tokens/s). Console runs, mean of two rounds, on the fork before the GQA attention change 2e4dbf0 was merged (ab759cf41), without the stateful fallback.

The base implementation produces incorrect results at depth because of the V-cache slicing issue. For reference, the table also includes the base implementation with KV slicing disabled, which produces correct results.

Model	Depth	Base	Base, no slicing	This PR
Llama-3.2 1B	1024	71.2	69.3	87.7
Llama-3.2 1B	4096	40.3	36.9	64.9
Phi-4 mini	1024	22.6	22.0	28.1
Phi-4 mini	4096	12.9	11.7	21.1
Qwen3.5 4B	1024	24.2	24.2	25.7
Qwen3.5 4B	4096	19.2	18.6	23.4

Decode at depth 0 changes by at most 7% (Llama-3.2 1B, 91.5 → 97.8).

Prefill (-p 512) changes by -10% to +20% depending on model and depth, with no consistent direction. This is within the run-to-run variation observed on this machine: -fa 1 prefill, which does not use the changed code, varies by up to 10% between the same two builds.

SmolLM2 1.7B and Phi-3.5 mini do not use grouped KV heads and are unchanged from the base implementation when KV slicing is disabled. -fa 1 decode is unchanged within 2%.

On the final branch, rebased onto 8619618 and including the stateful fallback, an SSH check at depth 4096 (median of 3) gives:

Model	-fa off	-fa on
Llama-3.2 1B	61.5	71.0
Phi-4 mini	20.6	25.1
Qwen3.5 4B	23.0	24.4
Phi-3.5 mini	11.8	20.9

Before dd04231, -fa off and -fa on decoded at the same speed at this depth. The remaining gap is primarily because -fa on uses the 4-D KV-cache layout introduced by dd04231, while -fa off intentionally retains the flat layout for compatibility with the transposed V cache.

The first two commits were also A/B tested against dev_backend_openvino using three interleaved rounds at depths 0 and 4096. Performance was within 1% on Llama-3.2 1B, Phi-4 mini, and Qwen3.5 4B. The third commit only applies with -fa off.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES. AI was used for testing the changes and merging the two PRs together.