[executorch][cuda] Lower off-graph KV in a cell layout

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [executorch][muse] Move off-graph KV to the batching path and test batching in OSS CI #23375
• [executorch][muse] Add run_solo_batching #23374
• [executorch][muse] Add export_solo_batching.py #23373
• [executorch][cuda] Let CudaExecutor honor a program's prefill lower bound #23372
• [executorch][llm] Share the batching executors' session logic under extension/llm/batching/util #23371
• [executorch][cuda] Add CudaExecutor #23370
• [executorch][cuda] Add step_plan for the CUDA batching executor #23369
• [executorch][llm] Add Buck targets for extension/llm/batching #23368
• [executorch][cuda] Add CudaCellCache #23367
• -> [executorch][cuda] Lower off-graph KV in a cell layout #23366
• [executorch][cuda] Extract CudaKVPool from CudaSequenceKVCache #23365

Adds a second layout to LowerOffGraphKVPass for batching many sequences over
one forward, the CUDA counterpart of MLX's batched-cell cache.

A manifest with "layout": "cell" (and max_cells) lowers every
kvcache::update_and_attend to:

index_copy_(pool, cells[:T], k)   # per layer, BSHD [1, max_cells, H, D]
triton::sdpa(q, pool, mask_w[:, :, :T, :], kv_len=read_len)

• Every layer keeps its history in one pool of max_cells rows. A window only
  selects which mask the layer reads.
• cells ([max_write] int64), read_len ([1]) and one
  mask_w{window} ([1, 1, max_write, max_cells] bool) per distinct window
  are runtime-bound constants, declared once per program at a fixed shape. The
  runtime cache writes them before each forward, so a captured CUDA graph keeps
  their addresses and serves any mix of sequences without recapture. Unlike
  the pools they are small, so they carry zero bytes at compile time (Inductor
  reads small constants while compiling); like the pools they are never
  serialized.
• Placement and visibility are the neutral CellCache's; the mask is always
  explicit, and read_len bounds the sweep so stale columns past it are never
  read.

The sequence layout is unchanged and stays the default.

Compiling these buffers needs two Inductor options while a cell-layout method
compiles:

• joint_graph_constant_folding off: the buffers are constants the graph only
  reads, so Inductor would fold slices of them at compile time, reading storage
  that does not exist yet and baking the result in.
• always_keep_tensor_constants on: read_len is small enough that Inductor
  would otherwise inline it as a literal.

The batching executor sends one-token steps to a static decode method and
wider ones to a prefill method dynamic from two tokens. A spike confirmed
AOTI compiles triton::sdpa with a symbolic query length starting at 2 (the
2..4 small-query split-K branch included) and matches eager SDPA for T=2..64,
so no sdpa change is needed.

Differential Revision: D123051478

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23366

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Pending, 4 Unclassified Failures

As of commit d0b6945 with merge base c16dd0a ():

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 134
• Test CUDA Builds / unittest-cuda / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_offgraph_kv.py::OffGraphKVCellCompileTest::test_decode_and_prefill_from_two_tokens_lower
• Test CUDA Builds / unittest-cuda-runtime / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  [ FAILED ] CudaKVCacheTest.CacheOutlivesTheStreamItLastSteppedOn
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_offgraph_kv.py::OffGraphKVCellCompileTest::test_decode_and_prefill_from_two_tokens_lower

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.