[executorch][cuda] Add CudaCellCache

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [executorch][muse] Move off-graph KV to the batching path and test batching in OSS CI #23375
• [executorch][muse] Add run_solo_batching #23374
• [executorch][muse] Add export_solo_batching.py #23373
• [executorch][cuda] Let CudaExecutor honor a program's prefill lower bound #23372
• [executorch][llm] Share the batching executors' session logic under extension/llm/batching/util #23371
• [executorch][cuda] Add CudaExecutor #23370
• [executorch][cuda] Add step_plan for the CUDA batching executor #23369
• [executorch][llm] Add Buck targets for extension/llm/batching #23368
• -> [executorch][cuda] Add CudaCellCache #23367
• [executorch][cuda] Lower off-graph KV in a cell layout #23366
• [executorch][cuda] Extract CudaKVPool from CudaSequenceKVCache #23365

The runtime half of the cell layout: CudaCellCache is
cache::CellCache + CudaKVCache, the CUDA counterpart of MLXCellCache, built
over the CudaKVPool from the previous diff.

• The neutral CellCache owns the cell table: placement, owner bitsets, and the
  BatchControl verbs (seq_new/clone/rm, rewind, declare_step).
• The pool owns one growable K/V pool per layer (every layer, window or not,
  keeps its history there), plus the step buffers the cell lowering declares:
  cells, read_len, and one mask_w{window} per distinct window.
• prepare_step(width, stream) places the declared tokens and writes the
  placement, extent and per-window masks into those buffers on the step's
  stream. Positions follow from the declaration (every token continues its
  sequence), so nothing is read back from the device. The pool grows before
  placement, to the furthest placement could reach, so a failed allocation
  leaves the table untouched.
• The step buffers never move, so a captured decode graph serves any mix of
  sequences. Only pool growth recaptures.
• Only mask rows [0, width) and columns [0, read_len) are written; the lowered
  program bounds its sweep by read_len (guarded by a test in the lowering
  diff).

Registered for batched-cell and for batched, as the batch layout this
backend serves.

The mutex is recursive because the base's verbs call each other virtually
(seq_clone takes its id through seq_new).

Differential Revision: D123051479

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23367

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 5 Unclassified Failures

As of commit 7db538f with merge base c16dd0a ():

NEW FAILURE - The following job has failed:

• Lint / lintrunner (gh)
  >>> Lint for extension/cuda/runtime_api.h:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-non-quantized / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 134
• Test CUDA Builds / test-model-cuda-e2e-Voxtral-Mini-3B-2507-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 134
• Test CUDA Builds / unittest-cuda / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_offgraph_kv.py::OffGraphKVCellCompileTest::test_decode_and_prefill_from_two_tokens_lower
• Test CUDA Builds / unittest-cuda-runtime / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  [ FAILED ] CudaKVCacheTest.CacheOutlivesTheStreamItLastSteppedOn
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_offgraph_kv.py::OffGraphKVCellCompileTest::test_decode_and_prefill_from_two_tokens_lower

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.