sched, ggml-meta : pipeline the host KV delivery under split mode tensor

[Piggidragon]

Overview

Pipeline the host-to-device delivery of a host-resident KV cache under -sm tensor, the one case GenerelSchwerz#39 left on the ordered path.

Stacked on stack/kv-transport-base, which merges GenerelSchwerz#39 (pipelined transport)
and #5 (host cache split by head). GitHub allows only one base branch, so the two parents meet on that
branch and this PR targets it. The diff below is only the three commits that belong here. Merge
GenerelSchwerz#39 and #5 first.

The change itself is the last three commits:

1. ggml-meta : add events and a ranged write for a head-split host cache
   • A meta event is one event per simple device.
   • set_tensor_async / set_tensor_2d_async accept part of the window of the head-split copy (whole cells from an offset, per stream), as one 2d copy per stream per device.
   • ggml_backend_meta_init_transfer builds a meta backend without a communicator for the transfer streams.
   • The meta graph compute rotates the compute containers of buffers that are only sources, so tensors bound into the ring do not pile up.
2. sched : pipeline the host KV delivery under split mode tensor
   • A meta backend is eligible when every simple backend is.
   • The headroom check uses the fullest simple device.
   • A device of the meta type that is not the ggml meta backend stays ordered (the existing test-alloc case).
   • test-llama-archs adds Meta -nkvo -np 2 -kvpd 1.
3. docs : measure the pipelined transport under split mode tensor
   • Measurements and gates are in docs/kv-transport-pipelining.md#tensor-parallelism.
   • LLAMA_KV_SM works for the A/B and server gate scripts.

Testing

RTX 4070 (gen4 x16) + RTX 3060 (gen3 x4), CUDA + NCCL, Qwen3.8-27B-UD-IQ2_M, -sm tensor -nkvo --kv-cpu-pinned -ctk q8_0 -ctv q8_0.

llama-bench tg128, A/B/A/B, -kvpb 512:

context	ordered	pipelined	gain
4,096	15.84	21.95	+38.6%
16,384	7.08	8.78	+24.0%
32,768	4.08	4.89	+19.7%

At 16,384 the ordered copy is 104 ms against 31 ms of compute. That is the 3060's x4 link, so the gain is bounded by the slower link, not by the scheduling.

Output is identical at depth 0, 1 and 4 in every gate:

• server exactness gate (LLAMA_KV_SM=tensor): all 8 tasks
• parallel gate, 8 concurrent sequences over streams: 125cb9c2082d36cf
• greedy completion: Qwen3.8 64e86551f7ef1638, the same hash as a device-resident cache; gemma-4 5525e3f5ac7337d7 (-ts 50,50) and 4f8986fb3655a567 (-ts 55,45)

Existing paths are unchanged: the parallel gate still gives 17f946c340db110b with -sm none and db661b7a08686b97 with -sm layer at depth 0 and 1. test-llama-archs -s 1 passes on 2, 3 and 4 CUDA devices, and test-alloc passes.

Not done here

• Each device allocates the whole ring slot although it holds only its share of the heads. The meta compute buffers work the same way; sizing per device is left as future work.
• Not measured on devices with equal links.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - implemented by an agent on my instruction, see the Assisted-by: commit trailers.

🤖 Generated with Claude Code

https://claude.ai/code/session_0145V8xvxxUQMUVbg6ZiCZMg

🤖 Generated with Claude Code