vulkan: use CPU writes in ggml_backend_vk_cpy_tensor_async if the context is idle

[jeffbolznv]

Overview

I've again run into a perf issue where downloading inputs from the CPU/previous split is fairly expensive due to overhead of waiting on a fence (roughly 50us each in my testing). A while back we had all of these running through ggml_vk_buffer_write_2d and using memcpy to host-visible vidmem. But I think when we enabled cpy_tensor_async they started using GPU copies and a synchronize call for each download. This is relatively worse in recent qwen/gemma4 models which download 2 inputs per token, each doing a full CPU-GPU sync.

Fix is to detect that the backend is idle and use a CPU copy if so. I had also considered trying to amortize the synchronize calls in ggml-backend, but there was a recent change that tried to do that (#20793) and it had to be reverted, so handling this in the vulkan backend seems safer.

before

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 128 -p 0 -r 10 --prio 1 -m c:\models\DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf -m c:\models\DeepSeek-R1-Distill-Llama-8B-Q6_K.gguf -m c:\models\DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf -m c:\models\Llama-3.2-1B.Q2_K.gguf -m c:\models\Llama-3.2-1B.Q3_K_S.gguf -m c:\models\llama-3.2-3b-instruct-q5_k_m.gguf -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen2.5-7B-Instruct-1M-Q2_K.gguf  -m c:\models\gpt-oss-20b-mxfp4.gguf -m c:\models\Phi-3-mini-4k-instruct-q4.gguf -m c:\models\llama-2-7b.Q4_0.gguf -m c:\models\llama-3.2-3b-instruct-q8_0.gguf -m c:\models\Mistral-22B-v0.2-Q4_K_M.gguf -m c:\models\nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-Q4_K_S.gguf -m C:\models\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -m c:\models\unsloth\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q4_K_M.gguf -m c:\models\unsloth\gemma-4-E4B-it-GGUF\gemma-4-E4B-it-Q4_K_M.gguf -m c:\models\unsloth\gemma-4-E2B-it-GGUF\gemma-4-E2B-it-Q4_K_M.gguf -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\gpt-oss-20b-mxfp4.gguf -m c:\models\granite-3.0-3b-a800m-instruct-Q8_0.gguf -m c:\models\Moonlight-16B-A3B-Instruct-Q4_K_M.gguf -m c:\models\Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2v
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  -1 |   1 |           tg128 |       242.93 ± 13.39 |
| llama 8B Q6_K                  |   6.14 GiB |     8.03 B | Vulkan     |  -1 |   1 |           tg128 |        200.36 ± 5.45 |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | Vulkan     |  -1 |   1 |           tg128 |        136.67 ± 4.01 |
| llama 1B Q2_K - Medium         | 546.50 MiB |     1.24 B | Vulkan     |  -1 |   1 |           tg128 |       836.98 ± 57.05 |
| llama 1B Q3_K - Small          | 604.50 MiB |     1.24 B | Vulkan     |  -1 |   1 |           tg128 |        842.88 ± 5.20 |
| llama 3B Q5_K - Medium         |   2.16 GiB |     3.21 B | Vulkan     |  -1 |   1 |           tg128 |       408.89 ± 14.55 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           tg128 |       282.78 ± 25.82 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  -1 |   1 |           tg128 |        321.72 ± 8.87 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     |  -1 |   1 |           tg128 |       369.78 ± 20.11 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     |  -1 |   1 |           tg128 |       375.09 ± 16.69 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  -1 |   1 |           tg128 |        283.44 ± 9.99 |
| llama 3B Q8_0                  |   3.18 GiB |     3.21 B | Vulkan     |  -1 |   1 |           tg128 |       329.90 ± 21.00 |
| llama ?B Q4_K - Medium         |  12.42 GiB |    22.24 B | Vulkan     |  -1 |   1 |           tg128 |         97.25 ± 2.10 |
| deci 70B Q4_K - Small          |  26.66 GiB |    49.87 B | Vulkan     |  -1 |   1 |           tg128 |         52.14 ± 0.30 |
| qwen35moe 35B.A3B Q4_K - Medium |  20.60 GiB |    34.66 B | Vulkan     |  -1 |   1 |           tg128 |        204.40 ± 9.82 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    26.90 B | Vulkan     |  -1 |   1 |           tg128 |         68.09 ± 1.08 |
| gemma4 E4B Q4_K - Medium       |   4.62 GiB |     7.52 B | Vulkan     |  -1 |   1 |           tg128 |        228.04 ± 3.06 |
| gemma4 E2B Q4_K - Medium       |   2.88 GiB |     4.65 B | Vulkan     |  -1 |   1 |           tg128 |       312.78 ± 14.08 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           tg128 |       282.90 ± 25.72 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     |  -1 |   1 |           tg128 |       361.67 ± 42.49 |
| granitemoe 3B Q8_0             |   3.34 GiB |     3.37 B | Vulkan     |  -1 |   1 |           tg128 |       449.12 ± 16.76 |
| deepseek2 16B Q4_K - Medium    |   9.81 GiB |    15.96 B | Vulkan     |  -1 |   1 |           tg128 |       364.93 ± 27.93 |
| nemotron_h_moe 31B.A3.5B Q4_K - Medium |  22.88 GiB |    31.58 B | Vulkan     |  -1 |   1 |           tg128 |       276.10 ± 26.84 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           tg128 |        207.16 ± 2.31 |

after

Z:\github\jeffbolznv\llama.cpp\build\bin\RelWithDebInfo>llama-bench.exe -fa 1 -n 128 -p 0 -r 10 --prio 1 -m c:\models\DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf -m c:\models\DeepSeek-R1-Distill-Llama-8B-Q6_K.gguf -m c:\models\DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf -m c:\models\Llama-3.2-1B.Q2_K.gguf -m c:\models\Llama-3.2-1B.Q3_K_S.gguf -m c:\models\llama-3.2-3b-instruct-q5_k_m.gguf -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\Qwen2.5-7B-Instruct-1M-Q2_K.gguf  -m c:\models\gpt-oss-20b-mxfp4.gguf -m c:\models\Phi-3-mini-4k-instruct-q4.gguf -m c:\models\llama-2-7b.Q4_0.gguf -m c:\models\llama-3.2-3b-instruct-q8_0.gguf -m c:\models\Mistral-22B-v0.2-Q4_K_M.gguf -m c:\models\nvidia_Llama-3_3-Nemotron-Super-49B-v1_5-Q4_K_S.gguf -m C:\models\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -m c:\models\unsloth\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q4_K_M.gguf -m c:\models\unsloth\gemma-4-E4B-it-GGUF\gemma-4-E4B-it-Q4_K_M.gguf -m c:\models\unsloth\gemma-4-E2B-it-GGUF\gemma-4-E2B-it-Q4_K_M.gguf -m c:\models\Qwen_Qwen3-30B-A3B-Q2_K.gguf -m c:\models\gpt-oss-20b-mxfp4.gguf -m c:\models\granite-3.0-3b-a800m-instruct-Q8_0.gguf -m c:\models\Moonlight-16B-A3B-Instruct-Q4_K_M.gguf -m c:\models\Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf -m c:\models\Qwen3.6-35B-A3B-UD-Q3_K_M.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | fp4: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2v
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  -1 |   1 |           tg128 |       242.83 ± 15.12 |
| llama 8B Q6_K                  |   6.14 GiB |     8.03 B | Vulkan     |  -1 |   1 |           tg128 |        204.30 ± 6.55 |
| qwen2 14B Q4_K - Medium        |   8.37 GiB |    14.77 B | Vulkan     |  -1 |   1 |           tg128 |        135.75 ± 4.38 |
| llama 1B Q2_K - Medium         | 546.50 MiB |     1.24 B | Vulkan     |  -1 |   1 |           tg128 |       915.72 ± 37.46 |
| llama 1B Q3_K - Small          | 604.50 MiB |     1.24 B | Vulkan     |  -1 |   1 |           tg128 |        874.34 ± 6.93 |
| llama 3B Q5_K - Medium         |   2.16 GiB |     3.21 B | Vulkan     |  -1 |   1 |           tg128 |       406.72 ± 21.85 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           tg128 |       298.06 ± 12.48 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  -1 |   1 |           tg128 |       327.94 ± 17.33 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     |  -1 |   1 |           tg128 |       372.11 ± 18.69 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     |  -1 |   1 |           tg128 |       368.99 ± 27.47 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  -1 |   1 |           tg128 |       280.98 ± 11.92 |
| llama 3B Q8_0                  |   3.18 GiB |     3.21 B | Vulkan     |  -1 |   1 |           tg128 |       337.24 ± 17.31 |
| llama ?B Q4_K - Medium         |  12.42 GiB |    22.24 B | Vulkan     |  -1 |   1 |           tg128 |         97.37 ± 1.33 |
| deci 70B Q4_K - Small          |  26.66 GiB |    49.87 B | Vulkan     |  -1 |   1 |           tg128 |         52.47 ± 0.16 |
| qwen35moe 35B.A3B Q4_K - Medium |  20.60 GiB |    34.66 B | Vulkan     |  -1 |   1 |           tg128 |        207.31 ± 5.64 |
| qwen35 27B Q4_K - Medium       |  15.65 GiB |    26.90 B | Vulkan     |  -1 |   1 |           tg128 |         68.96 ± 0.36 |
| gemma4 E4B Q4_K - Medium       |   4.62 GiB |     7.52 B | Vulkan     |  -1 |   1 |           tg128 |        233.96 ± 4.75 |
| gemma4 E2B Q4_K - Medium       |   2.88 GiB |     4.65 B | Vulkan     |  -1 |   1 |           tg128 |       341.76 ± 11.51 |
| qwen3moe 30B.A3B Q2_K - Medium |  10.15 GiB |    30.53 B | Vulkan     |  -1 |   1 |           tg128 |       291.51 ± 12.12 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     |  -1 |   1 |           tg128 |        382.98 ± 3.04 |
| granitemoe 3B Q8_0             |   3.34 GiB |     3.37 B | Vulkan     |  -1 |   1 |           tg128 |       470.98 ± 19.94 |
| deepseek2 16B Q4_K - Medium    |   9.81 GiB |    15.96 B | Vulkan     |  -1 |   1 |           tg128 |       364.50 ± 49.00 |
| nemotron_h_moe 31B.A3.5B Q4_K - Medium |  22.88 GiB |    31.58 B | Vulkan     |  -1 |   1 |           tg128 |       283.05 ± 17.44 |
| qwen35moe 35B.A3B Q3_K - Medium |  15.45 GiB |    34.66 B | Vulkan     |  -1 |   1 |           tg128 |        207.09 ± 7.09 |

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, paired with codex.