Eval bug: Vulkan graph optimizer silently corrupts output for models with view-aliased state

[Eric-A-Stalee]

Name and Version

$ ./llama-cli --version
version: 0.1.2-dev (build 10511, commit f7aadef)
built with GNU 16.2.1 for Linux x86_64

Built from #27342 (unmerged). The bug is in ggml-vulkan.cpp, present on master. The PR does not touch ggml-vulkan.cpp.

Operating systems

Linux, Windows

GGML backends

Vulkan

Hardware

• AMD Radeon RX 7800 XT (gfx1101, RADV, Mesa 26.2.1, Linux 6.18)
• AMD Strix Halo/Ryzen AI Max+ 395 (gfx1151, RADV, Linux)
• NVIDIA Quadro RTX 5000 (Windows, driver 572.61; Vulkan affected, CUDA clean)

Models

Qwen3.8-27B: bartowski/Qwen3.8-27B-GGUF IQ2_XXS (NVIDIA run); 2-bit and 3-bit K/IQ quants of the same model (AMD runs).
Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF Q4_K_M (DFlash 2, #27342).

Any quant works. The bug has to do with the model's recurrent-state graph, not the quant.

Problem description & steps to reproduce

The Vulkan graph optimizer is unsafe for models with view-aliased state. It can select a wrong token in greedy decoding. All runs below use temperature 0 and seed 0, so the same input must give the same text on every run, and a speculative run must give the same text as the plain run of the target. One wrong token changes all the text that follows. Nothing is written to the log. With speculative decoding the wrong path accepts draft tokens that the model did not select, so its acceptance rate and speed are not valid measurements (faster on one GPU, slower on another pre/post patch; very relevant for #27342).

Cause: is_src_of in ggml_vk_graph_optimize, file ggml/src/ggml-vulkan/ggml-vulkan.cpp, line 17665 at commit f7aadef09. The currently implemented check does not identify two views of one tensor as dependent. The optimizer can then move a node across a read or a write of the same memory. Qwen3.8 keeps its recurrent state in views. The DFlash 2 verify graph from #27342 makes this occur on AMD and NVIDIA Vulkan. CUDA on the same NVIDIA GPU gives correct output. DFlash 2 is a TRIGGER not a CAUSE. The CAUSE is ggml-vulkan.cpp.

Fix: compare the view_src base of each source and destination. Tested on AMD and NVIDIA: 12/12 runs give the reference text. PR to follow.

--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -17422,15 +17422,24 @@ static void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph *
     };
 
     auto const &is_src_of = [](const ggml_tensor *dst, const ggml_tensor *src) -> bool {
+        auto const &base = [](const ggml_tensor * tensor) {
+            return tensor->view_src ? tensor->view_src : tensor;
+        };
         for (uint32_t s = 0; s < GGML_MAX_SRC; ++s) {
             if (dst->src[s] == src) {
                 return true;
             }
+            // A source view of dst may read storage written through a different view by src.
+            if (dst->src[s] && base(dst->src[s]) == base(src)) {
+                return true;
+            }
+            // Moving dst forward may overwrite storage still read through a view by src.
+            if (src->src[s] && base(dst) == base(src->src[s])) {
+                return true;
+            }
         }
         // implicit dependency if they view the same tensor
-        const ggml_tensor *dst2 = dst->view_src ? dst->view_src : dst;
-        const ggml_tensor *src2 = src->view_src ? src->view_src : src;
-        if (dst2 == src2) {
+        if (base(dst) == base(src)) {
             return true;
         }
         return false;

Steps:

1. Build the spec : add DFlash2 support (local convolution + candidate selector) #27342 branch with -DGGML_VULKAN=ON.
2. Download the target model and the drafter model listed above.
3. Run the reference command 12 times. Start a new process for each run. Save the generated text of each run.
4. Run the bug command 12 times in the same way.
5. Set GGML_VK_DISABLE_GRAPH_OPTIMIZE=1. Run the bug command 12 times in the same way.
6. Compare the generated texts. The attached run_repro_cli.sh with prompt.txt in same dir does steps 3 thru 6. It exits with code 1 when the bug occurs.
   Usage: BIN_DIR=<build/bin> TARGET=<target.gguf> DRAFTER=<drafter.gguf> ./run_repro_cli.sh, with prompt.txt in the same directory

Expected result: all three conditions give the same text in every run.

Actual result:

• Reference: 12/12 runs give the same text.
• Bug: the text is different multiple times from the reference at Temperature ZERO.
• Workaround: 12/12 runs give the reference text.

Measured:

• RX 7800 XT: 2-bit target, CLI, 3 runs each: reference 3/3 identical; bug 3/3 identical to each other but diverging from the reference at character 279; workaround 3/3 matching the reference over its full 2260 characters. (this is to satisfy the 'reproduce the issue using llama-completion with -fit off' request)
• RTX 5000: Vulkan, 12 runs each: bug 4 distinct outputs, 0/12 matching the reference; workaround 12/12 matching; patched build 12/12 matching. CUDA on the same GPU: 12/12 matching without any change.
• RX 7800 XT: server, 12 runs each: bug diverges from the reference at character 899 on every run; workaround 12/12 matching; patched build 12/12 matching; reference unchanged by the patch.
• Strix Halo (Ryzen AI Max+ 395): server: 15 runs, 12 distinct outputs.

Patch: vulkan-graph-alias-dependency-fix.patch, PR to follow.

Evidence: FINAL-WRITEUP.md (NVIDIA Vulkan vs CUDA, AMD Vulkan; Patch vs Pre-Patch).

The investigation was AI-assisted. Every number above was measured by me on my own hardware, and the patch was validated on NVIDIA and AMD. From my capability testing, DFlash 2 triggering this bug led to active sabotage of a model's ability to give a correct answer. This is not a statistically minor issue.

First Bad Commit

e68aa10 introduced this check. #17475 (removal of the large-graph skip in the optimizer) exposed this graph shape to it.

Relevant log output

Nothing is logged: no error, no warning. The output is silently wrong.
Results per GPU are in the Measured list above.

[Eric-A-Stalee]

With the fix, I get 12/12 accuracy vs the reference (no speculation, temp 0) for some n= draft lengths but not others, on the models I tested. Where it fails it is now consistently the same wrong text rather than randomly wrong. Before, with speculative decoding (DFlash 2), three fresh runs (at a fixed n=) gave THREE uniquely hashed different outputs, compared to ONE for the plain no speculative decoding run. But the issue occurred with MTP as well (it just didn't diverge multiple times, instead it consistently generated text different from no speculative decoding). And, early testing, indicates possibly with no speculative decoding as well...just much lower occurrence.

So the PR (#27812/fix) helps isolate the issue further, and improves behavior, and the optimizer bug itself looks resolved.

But different draft lengths can still produce different outputs which deviate from the target model, for the same given inputs at temperature 0, consistently (which should be impossible for a same machine and parameters).

Unfortunately it appears the issue -might- not be limited solely to speculative decoding but rather that it amplifies something in the underlying decoding process.

More to come.