Vulkan: GPU firmware crashes on Imagination PowerVR C-series (Pixel 11 Pro, Tensor G6, subgroupSize 128)

[JulianPscheid]

Summary

On a Google Pixel 11 Pro (Imagination PowerVR C-Series CXTP-48-1536 MC1, VK_DRIVER_ID_IMAGINATION_PROPRIETARY, Vulkan 1.4.317, subgroupSize=128, maxComputeWorkGroupInvocations=1024, maxComputeSharedMemorySize=32768), the Vulkan backend crashes the GPU firmware during prompt processing.

Every compute pipeline compiles. The first 128-token batch returns from llama_decode. About 9 seconds after the context is created, Android logs a PowerVR subsystem restart, the compositor wedges, and the device usually reboots. No tokens are produced.

The same app and models work on a Pixel 8 Pro (ARM Mali/Immortalis).

I don't expect a fix in llama.cpp; the fault is almost certainly in Imagination's driver. This is filed so the next person who tries Vulkan on a Tensor G6 finds it, and in case the subgroup-128 detail suggests a workaround to someone who knows the backend.

Android's GPU crash record

adb shell dumpsys dropbox --print | grep -A12 SubsystemRestart gives a per-crash reason. Across builds:

llama.cpp	flash attn	submissions	OS build	crash reason
b10621	on	default	2026-07	GUILTY_OVERRUNING
b10731	on	default	2026-07	[0]:[addr=0x8056288700]:[FW_PAGEFAULT]
b10731	off	default	2026-07	[0]:[addr=0x805730fd80]:[FW_PAGEFAULT]
b10731	on	serialised	2026-07 and 2026-09	GUILTY_OVERRUNING
b10731	on	default	2026-09	[0]:[addr=0x8047852d80]:[FW_PAGEFAULT]

GUILTY_OVERRUNING reads as a job that never terminated and hit the GPU watchdog. FW_PAGEFAULT reads as an MMU fault on an unmapped GPU address. The fault address is stable within a process (allocations move between runs), so the access is deterministic rather than a race.

b10621 -> b10731 changed the failure class but fixed nothing.

Ruled out, all measured on the device

• llama.cpp version: b10621 and b10731 (the latter includes vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps #27726's warptile clamp for subgroup > 64).
• Flash attention: LLAMA_FLASH_ATTN_TYPE_DISABLED for this vendor. No change.
• Submission structure: GGML_VK_SERIALIZE_SUBMISSIONS=1. No change except the crash class. Since every submit then waits for completion, this also shows that the first 128-token chunk finishes on the GPU and a later one faults. It is not a cross-submission race.
• Memory: reproduced on a fresh boot with 14 GB available and zero swap.
• Model architecture: a hybrid model (Qwen 3.5 2B, gated delta net + SSM) and a plain transformer (Gemma 4 E2B) fail identically.
• Quantisation: Q4_0 crashes exactly like Q4_K_M and UD-Q4_K_XL. This matters because a user on Imagination's own forum found that on the Pixel 10's D-series DXT-48-1536, k-quant compute shaders return wrong results while Q4_0, Q5_0, Q8_0, F16 and F32 are correct (https://forums.imgtec.com/t/powervr-dxt-48-1536-incorrect-results-from-k-quant-vulkan-compute-shaders-in-llama-cpp/4291). That is a different bug. The D-series computes wrong answers; the C-series crashes its firmware.
• Driver: the September 2026 Pixel OTA (CD1A.260905.001.A1, new vendor image) behaves identically. The driver still reports 1.4.317.

Configuration

Vulkan only, all layers offloaded, n_batch=128, n_ctx=7168-7424, prompt of about 2,850 to 3,000 tokens, type_k and type_v left at F16.

Related

• Bug: Vulkan backend not work on an Imagination GPU on RISC-V Platform #8437: Vulkan on an Imagination GPU (RISC-V, warp size 1). Different symptom, stale.
• Vulkan: wrong matmul results on Adreno (subgroupSize=128) — warptile WM exceeds BM (implicit WM <= BM invariant violated for subgroupSize > 64) #25734: Adreno subgroupSize=128, wrong matmul results, fixed by vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps #27726.
• Vulkan backend produces all-zero outputs on PowerVR GPU (Pixel 10 Pro) pytorch/executorch#17299: PowerVR D-series on the Pixel 10 Pro produced all-zero outputs from ExecuTorch's Vulkan backend, root-caused to a command-sequencing defect where dispatches after the first in a command buffer lost their writes.

Repro

So far only reproduced from an app embedding libllama, not from llama-bench. I can run a standalone repro on the device if someone suggests the exact invocation. Each attempt costs a reboot.

[ZitouniNidhal]

Great writeup. subgroupSize=128 on PowerVR’s driver is likely triggering a driver SPIR-V compiler bug during subgroup shuffles/broadcasts in ggml-vulkan.cpp, leading to the firmware pagefaults and watchdog hangs.

To run a standalone repro via ADB without your app wrapper:

adb push llama-bench /data/local/tmp/
adb push model.gguf /data/local/tmp/
adb shell "chmod +x /data/local/tmp/llama-bench && /data/local/tmp/llama-bench -m /data/local/tmp/model.gguf -p 1024 -n 0 -ngl 99"

A couple quick things to test:

GGML_VK_DISABLE_F16=1 (rules out PowerVR FP16 ALU driver bugs)
If you temporarily disable subgroup ops or hardcode subgroup size in ggml-vulkan.cpp, does prompt processing succeed? If that fixes it, we could add a driver vendor check for VK_DRIVER_ID_IMAGINATION_PROPRIETARY to bypass subgroup matmul paths on C-series hardware.

[JulianPscheid]

I built test-backend-ops and llama-bench for Android arm64 and ran them on the device. There turned out to be several separate problems rather than one, so this is a correction to my original report as much as an update.

@ZitouniNidhal you were right that it is a driver shader compiler bug, though not in the way either of us expected.

1. Dequant mul_mat_vec pipelines fail to compile (fixed, PR linked below)

vkCreateComputePipelines returns VK_ERROR_UNKNOWN for every dequant mul_mat_vec shader built with the subgroup-only reduction: Q2_K through Q6_K, TQ2_0, the IQ types, MXFP4, NVFP4. ggml rethrows and the process aborts on the first generated token. Legacy quants compile fine.

Not the subgroup width: 32, 64 and 128 all fail, as does dropping the full-subgroups flag and the required-subgroup-size pNext. Shared-memory reduction compiles and is correct. PR: #28341

This one aborts cleanly. It is not what crashes the GPU firmware.

2. FLASH_ATTN_EXT is wrong, and can hang the GPU

test-backend-ops -o FLASH_ATTN_EXT by head size:

hsk	passed	failed
40	0	226
64	2038	0
72	745	102
80	92	22
256	passes at kv=1024/nb=32; fails at kv=1024/nb=64 and at kv=2048

Failures are ERR ≈ 1.0, so the output is unrelated to the reference rather than slightly off.

The head-size-72 run also wedged the GPU. dumpsys dropbox recorded powervr - [988]:[surfaceflinger]:[GUILTY_OVERRUNING], the compositor froze, and the test process hung until its timeout. Head sizes 96 and 256 were only partly covered because I stopped the sweep after that.

3. FW_PAGEFAULT at large context (still open, and this is the one from the original report)

With the pipeline fix applied and flash attention disabled, inference at n_ctx=7168 still kills the firmware:

crash reason: powervr - [0]:[addr=0x80652c9f80]:[FW_PAGEFAULT]

llama-bench -p 128 runs clean end to end. -p 7168 page-faults. GGML_VK_MAX_NODES_PER_SUBMIT=8 does not prevent it, so it is not submission length. An unmapped GPU read rather than a watchdog kill, so it is a different failure from #2. I have not isolated it further yet.

4. Results are not reproducible run to run

Same binary, same input, three consecutive runs of test-backend-ops -o GET_ROWS: 102, 99, 100 of 115 passing. MUL_MAT -p type_a=iq2_xxs: 14 then 15 of 27. All failures are IQ types.

The k-quants are stable by contrast, 26/26 on Q4_K across four runs, so this is not general flakiness in the harness.

Filed separately as #28343.

5. Unrelated, small

mul_mat_vec_bf16_f32_f32 is created unconditionally at ggml-vulkan.cpp:5374 regardless of whether the device supports bf16. This device reports bf16: 0 and its compiler rejects that shader, so anything requesting a bf16 mul_mat_vec aborts. Filed as #28342.

Reproducing

cmake -S . -B build-android -G Ninja \
  -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-31 \
  -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_COMMON=ON
adb push build-android/bin/test-backend-ops /data/local/tmp/
adb shell /data/local/tmp/test-backend-ops test -b Vulkan0 -o MUL_MAT -p type_a=q4_K,

adb shell dumpsys dropbox --print | grep -A12 SubsystemRestart is what gives the firmware crash reason.

I would suggest keeping this issue for #3, since that is the firmware crash the title describes and the only one still unexplained.

[ZitouniNidhal]

This now looks like a separate address/memory-range problem rather than a submission or watchdog issue.

Since:

• -p 128 completes cleanly
• -p 7168 reproducibly triggers FW_PAGEFAULT
• FLASH_ATTN_EXT is disabled
• GGML_VK_MAX_NODES_PER_SUBMIT=8 does not change the result

I would focus on isolating the first context length where the fault appears and determining which Vulkan operation introduces the invalid GPU read.

A useful next step would be a binary search over prompt/context length (e.g. 1024, 2048, 4096, 5120, 6144, 7168) with the same model and binary. If there is a stable threshold, that strongly suggests a size-dependent buffer/offset/addressing issue rather than a random kernel failure.

I would then instrument the Vulkan backend to log the tensor/buffer ranges and the operation immediately before each submission around the failing threshold. In particular, check for offsets or buffer ranges that exceed the allocated VkBuffer size once the KV/context tensors grow.

If the same operation can be reproduced in test-backend-ops with dimensions matching the failing sequence length, that would give us a minimal driver repro.

For a permanent llama.cpp workaround, if this is confirmed to be a PowerVR address-range issue, I would prefer a vendor-specific safe path (chunking the affected operation or splitting the large buffer access) for VK_DRIVER_ID_IMAGINATION_PROPRIETARY, rather than globally reducing context size.

That should preserve correctness and avoid the firmware crash while keeping the normal path unchanged for other drivers.

[JulianPscheid]

Update with new measurements. The context-length lead is dead, and there is a worse problem underneath it.

@ZitouniNidhal on your suggestions:

GGML_VK_DISABLE_F16=1 does not help. Still garbage output.

The vendor check for VK_DRIVER_ID_IMAGINATION_PROPRIETARY is already up as #28341. It fixes the pipeline compile failure. It does not make this device produce correct results.

On the binary search over context length: I ran it, and there is no threshold. Standalone llama-bench at my app's settings (-b 128 -ub 128 -fa off) completes cleanly at p=2048, p=4096 and p=7168, and generates cleanly at depth 7168. My earlier "p=7168 page-faults" was llama-bench's default batching (-b 2048 -ub 512), which is a much larger graph, so that was my mistake. The app then crashed at n_ctx=6656, which is smaller than the standalone run that survives. Context size is not the trigger.

The part that matters more: this device computes wrong numbers without crashing.

llama-completion, standalone, -ngl 99 -c 6656 -b 128 -ub 128 -fa off --temp 0 --seed 1, prompt "The capital of France is Paris. The capital of Japan is Tokyo. Write three sentences about why the sky appears blue.":

stra。。肠肠肠爱ippo
   >>>是是
 2016 年 1 月 1 日，在 1999 年 1 月 1 日，在 1999 年 1 月 1 日

The GPU fault counter was identical before and after. No firmware fault, no hang. The device just returns junk.

The output is also different every time. 14 runs across 7 configurations, all greedy with a fixed seed, same binary, model and prompt:

baseline, GGML_VK_DISABLE_F16, GGML_VK_DISABLE_FUSION, GGML_VK_DISABLE_MMVQ, GGML_VK_DISABLE_ASYNC, shared-memory dmmv reduction, hybrid dmmv reduction.

Every one produced garbage, and no two runs matched. Greedy decoding should be byte-identical run to run, so this looks like a race or uninitialized memory rather than a wrong kernel. The hybrid reduction failing the same way also rules out #28341 as the cause.

I could not get a CPU-side comparison on this phone. CPU-only runs time out before emitting a token, so I have no -ngl 0 baseline to show you.

One thing from outside llama.cpp that may be relevant. Chrome's Dawn force-disables VulkanUseZeroInitializeWorkgroupMemoryExtension for all Imagination devices, with the comment "Pixel 10 workgroup zero init does not always work as expected" (crbug.com/479242793). Dawn also enables VulkanPolyfilSwitchWithIf for this driver ("polyfill for case switch with large ranges"), enables VulkanIncompletePipelineCacheWorkaround on Pixel 10, and denies maximal reconvergence on IsAndroidImgTec(). Workgroup memory that is not reliably zeroed would fit non-deterministic compute results.

test-backend-ops does not catch any of this. run_whole_graph() defaults to false and the comparator runs the graph one node at a time, so every individual op I tested passes, including all k-quant MUL_MAT and the hybrid and recurrent ops, while the assembled model produces garbage. What does catch it in about ten seconds is a greedy canary: --temp 0 -n 16 on a short prompt, and check that the output is coherent English.

I am not asking anyone to fix this driver. I have disabled the vendor in my own app. I am reporting it in case you would rather llama.cpp refuse VK_DRIVER_ID_IMAGINATION_PROPRIETARY, or at least warn on it, than emit silently wrong tokens, which is what it does today. Chrome, PlayCanvas and Construct 3 have all ended up blocking this vendor for WebGPU for related reasons.