DSV4 prefill: expert transfer volume and overlap (~3x term)

[ro99]

Summary

Strata's DSV4 prefill moves more expert bytes than the reference and overlaps
them worse, but the direct profile in experiment 0114 shows this is roughly a
3x term, not the dominant one. Filing it so it is tracked rather than
repeatedly re-prioritised above the two kernel defects that are 66x and 286x.

Measured

Reference, one ~3,035-token prefill, rank 0 (experiment 0114):

• expert H2D 28.85 GB at 5.73 GB/s, 5.032 s
• copies arrive as paired 2.10 MB + 0.13 MB, about 11,850 pairs, i.e. one per
  expert per layer — the routed set is streamed in full each request
• GPU is ~90% busy across the request span, so their transfer is well overlapped

Strata at 2,612 tokens (experiment 0113):

• expert demand H2D 92.12 GB, demand wait 14.98–41.88 s across runs
• effective ~4.3 GB/s on wait
• expert_prefetch_predictions defaults to 0, so advisory prefetch is off
• the resident arena is not page-locked; Dsv4ResidentWeightStore::pin()
  exists but is blocked on VRAM reserve accounting

What this means

Our transfer rate is within 1.3x of the reference. The gap is volume (92 GB
against their 57.7 GB across both ranks) and overlap (demand-miss stalls rather
than prefetch). Fixing both is worth on the order of 10 s of a 145 s prefill.

Sub-items

• Enable and evaluate expert prefetch (expert_prefetch_predictions > 0,
  the equivalent of their LVLLM_GPU_PREFETCH_WINDOW=1). Never attempted.
• Page-lock the resident arena. Blocked on ~72 MB of device-side mapping
  against the 21.0 GiB kDsv4RankLocalPerDeviceVramCeiling; the real task
  is accounting for the ~3 GiB between that ceiling and the card's 24 GiB,
  not nudging the constant.
• Explain the expert-upload variance: 14.98 s vs 41.88 s of demand wait on
  identical 2,612-token arms. Experiment 0106 ruled out NUMA page placement
  (the tiled arena is explicitly bound, shard 0→node 0, shard 1→node 1, and
  verified stable in live numa_maps). Cause still unknown, and it makes
  single-arm wall-clock comparisons unreliable.
• Understand why we move 1.6x more expert bytes than the reference for a
  comparable prompt.
• Mean-locality: both reference GPUs are NUMA-affine to node 1, so roughly
  half of every expert upload is deterministically remote. Full locality is
  impossible under the ~129 GB per-node capacity; a capacity-aware policy
  biasing bytes toward node 1 is a distinct hypothesis aimed at mean
  transfer cost.

Priority

Below the attention kernel (286x, experiment 0114) and the expert
dequantise-then-GEMM restructure (66x). Worth revisiting once those land, since
transfer becomes a larger share of what remains.

References

• docs/experiments/0114-reference-stack-prefill-profile-2026-08-16.md
• docs/experiments/0113-dsv4-inline-bf16-host-passes-2026-08-16.md
• docs/experiments/0106 (NUMA placement falsified)