[sglang] Persist TP, prefill, NEXTN, and DeepEP graphs on main

.gitignore

@@ -362,4 +362,7 @@ marimo/_lsp/
 __marimo__/
 
 # Streamlit
-.streamlit/secrets.toml
+.streamlit/secrets.toml
+tests/deepep_matrix_work/
+deepep_fabric_archive/
+hook_archive/

README.md

@@ -82,11 +82,30 @@ Foundry ships engine integrations under `foundry/python/foundry/integration/`. P
 | Engine | Single GPU | DP | TP | EP |
 |---|:---:|:---:|:---:|:---:|
 | vLLM | ✅ | ✅ | 🚧 | ✅ |
-| SGLang | ✅ | ✅ | 🚧 | ✅ |
+| SGLang | ✅ | ✅ | ✅ | ✅ |
 | TensorRT-LLM | 🚧 | 🚧 | 🚧 | 🚧 |
 
 ✅ validated end-to-end (SAVE → LOAD → query)  ·  🚧 not yet
 
+SGLang-main validation is pinned to
+`9b853e6832e71a3058212df02a025232a453e146`; the current evidence uses immutable
+Foundry `eb1ec00b7004339c5150ce6c3ba057ceda047833`. Plain TP uses one explicit
+padded symmetric-memory decode shape; the Full prefill backend may additionally
+persist multiple token buckets. Qwen3.5-122B-A10B-GPTQ-Int4 TP=4 restored
+decode-8 and prefill-16/64 graphs on every H100 rank in Modal run
+`eb1ec00b7004339c5150ce6c-1784907539317556437`. Combined online-FP8 TP=2/EP=2,
+NEXTN, DeepEP low-latency, and Full prefill-16/64 was validated with
+Qwen3.5-35B-A3B in run
+`eb1ec00b7004339c5150ce6c-1784906247386993159`.
+
+NEXTN is the GPU-validated speculative algorithm. DeepEP on the pinned Hopper
+stack requires FA3 and the documented NVSHMEM transport settings; a separate
+legacy NVL CUDA-IPC path is available for no-fabric hosts. Multi-shape decode,
+other speculative algorithms, other fabrics, breakable/`tc` prefill
+persistence, PP, LoRA, PDMux, diffusion, and recapture remain outside the
+validated scope. Broader TP collective support is tracked in
+[issue #6](https://github.com/foundry-org/foundry/issues/6).
+
 The adapted vLLM / SGLang / TensorRT-LLM forks will be released alongside this repo at `foundry-org/vllm`, `foundry-org/sglang`, `foundry-org/TensorRT-LLM`.
 
 ### Performance

RELEASE.md

@@ -1,10 +1,48 @@
-# Foundry 0.0.2
+# Release Notes
+
+## Unreleased
+
+- **SGLang-main tensor parallel and Full prefill.** One explicit padded
+  torch-symmetric-memory decode shape is validated end-to-end with Qwen3-8B
+  TP=2 and Qwen3.5-122B-A10B-GPTQ-Int4 TP=4 on H100. The TP4 regression also
+  persisted Full prefill buckets 16/64: fresh LOAD restored three graphs per
+  rank at offset `59301167104`, with exact outputs, collectives, catalogs, and
+  SAVE/SAVE2 fingerprints in run
+  `eb1ec00b7004339c5150ce6c-1784907539317556437`. Multi-shape decode and
+  alternate-collective support remain tracked in
+  [upstream issue #6](https://github.com/foundry-org/foundry/issues/6).
+- **SGLang graph-mode composition.** With SGLang
+  `9b853e6832e71a3058212df02a025232a453e146`, Qwen3.5-35B-A3B online FP8
+  combined TP=2/EP=2, NEXTN (3 steps, top-k 1, 4 draft tokens), DeepEP
+  low-latency, and Full prefill 16/64 in one fresh LOAD. Run
+  `eb1ec00b7004339c5150ce6c-1784906247386993159` at immutable Foundry
+  `eb1ec00b7004339c5150ce6c3ba057ceda047833` restored four graphs and three
+  sessions per rank, reproduced rank-local offsets and archives, and retained
+  exact token, speculative-acceptance, DeepEP-node, bootstrap, NVSHMEM-module,
+  and no-recapture evidence. NEXTN is the only GPU-validated speculative
+  algorithm; Full is the only validated prefill-persistence backend.
+- **No-fabric DeepEP IPC.** The VMM-IPC bridge transports shareable file
+  descriptors with `SCM_RIGHTS`, allowing vLLM and SGLang DeepEP NVL buffers to
+  work without fabric/IMEX. This extends
+  [upstream PR #3](https://github.com/foundry-org/foundry/pull/3). The verified
+  regression is BF16 on 2×H100; GB300/aarch64 FP8 and RDC relinking reported in
+  [upstream issue #1](https://github.com/foundry-org/foundry/issues/1) remain
+  outside that validation scope.
+- **A100/FA2 load parity.** CUDA generator registration is skipped on LOAD when
+  every graph has zero RNG increment, preventing unused seed/offset tensors
+  from drifting the tracked workspace cursor. This incorporates
+  [upstream PR #4](https://github.com/foundry-org/foundry/pull/4) and the root
+  cause from [issue #2](https://github.com/foundry-org/foundry/issues/2).
+
+---
+
+## Foundry 0.0.2
 
 SGLang graduates to a fully validated engine. This release brings the SGLang
 integration to parity with vLLM across single GPU, data parallel, and expert
 parallel — with a self-contained recipe and no vLLM build dependency for EP.
 
-## Highlights
+### Highlights
 
 - **SGLang single GPU / DP / EP all validated end-to-end.**
   SAVE → LOAD → query verified on single-GPU Qwen3-1.7B / 4B / 14B,
@@ -23,12 +61,13 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   `data_parallel_controller.py`). All save/load logic lives in the integration
   layer; the edits are inert unless `--foundry-graph-extension-config-path` is set.
 
-## Engine integrations
+### Engine integrations
 
-- **SGLang** — integration for SGLang v0.5.13. Working configurations: single
+- **SGLang** — integration for SGLang v0.5.13. Working configurations in this
+  historical release: single
   GPU, data parallel (DP), expert parallel (EP, DeepEP low-latency + DP-attention with fa3). Self-contained recipes under `recipe/sglang/` (shared TOML pair +
   per-config serve scripts, mirroring `recipe/vllm/`) for Qwen3-1.7B (single),
-  Qwen3-1.7B (DP), and Qwen3-30B-A3B-FP8 (EP). TP attention stays unsupported
+  Qwen3-1.7B (DP), and Qwen3-30B-A3B-FP8 (EP). TP attention was not yet supported
   (NCCL all-reduce vs the VMM region) — EP uses DP-attention instead. EP requires
   SGLang's pinned kernels — DeepEP `9af0e0d` (not vLLM's `29d31c0`),
   `sgl-deep-gemm >= 0.1.2`, and `flash-attn-3` (fa3 sidesteps a FlashInfer
@@ -45,7 +84,7 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   on LOAD, with `init_nvshmem_for_loaded_modules` run once on LOAD before any
   NVSHMEM-kernel graph replays.
 
-## Fixes
+### Fixes
 
 - **Per-rank VMM device binding (DP/TP/EP).** `set_allocation_region` binds to
   the current CUDA device, so the integration now calls `set_device(gpu_id)`
@@ -63,26 +102,22 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   capture loop never sets), and the FlashInfer per-bs pre-pass gated off for fa3
   while still populating `decode_cuda_graph_metadata` post-load for replay.
 
-## Docs
+### Docs
 
 - New `docs/sglang/` set (overview, direct-edits, hooks, memory-lifecycle,
   save-load-workflow, memory-consistency) and a self-contained `recipe/sglang/`
   README with install, run, performance, and troubleshooting.
 - Top-level README parallelism status table updated to mark SGLang single GPU,
   DP, and EP as validated.
 
----
-
-## Previous Releases
-
 ## Foundry 0.0.1
 
 First public release of Foundry — a CUDA-graph persistence library that
 captures an entire model's CUDA graphs (plus their device context: modules,
 workspaces, VMM layout) once and replays them at startup, eliminating compile,
 warmup, and capture from cold-start time.
 
-## Highlights
+### Highlights
 
 - **Deterministic memory layout — zero patching on graph load.**
   Foundry indirects memory allocation to the same reserved memory region
@@ -99,7 +134,7 @@ warmup, and capture from cold-start time.
   On load, the template is rebuilt once per group and instances are reconstructed
   on demand, keeping load fast and asynchronous.
 
-## Engine integrations
+### Engine integrations
 
 - **vLLM** — compatible with vLLM v0.21. Working configurations: single GPU,
   data parallel (DP), expert parallel (EP, DeepEP low-latency). End-to-end
@@ -111,31 +146,31 @@ warmup, and capture from cold-start time.
   integration layer (`foundry/integration/<engine>/`); engine forks contain
   only minimal hook calls.
 
-## Verified kernel & comm support
+### Verified kernel & comm support
 
 - **cuBLAS NVJET** kernels (Hopper+).
 - **torch.compile** modules.
 - **NVSHMEM / DeepEP** validated.
 - **DeepGEMM FP8 MoE** validated.
 
-## Dependency
+### Dependency
 
 - **PyTorch 2.11.0** (compatible with 2.9 – 2.11).
 - **CUDA 12+**, CMake 4.0+, Boost.
 
-## Documentation
+### Documentation
 
 - Integration design notes and per-engine recipes under `docs/` and `recipe/`.
 - vLLM recipe README covers save/load workflow, archive layout, and required
   env settings.
 
-## Repository hygiene
+### Repository hygiene
 
 - Open-source pre-commit hooks (ruff, ruff-format, clang-format,
   markdownlint, actionlint, DCO sign-off).
 - Smoke tests covering re-export imports and archive round-trip.
 
-## Roadmap
+### Roadmap
 
 - Adapted vLLM and SGLang forks published alongside the release.
 - Tensor parallel support.

ROADMAP.md

@@ -30,8 +30,17 @@
     - [x] Quantized MoE with DeepGemm
 - [x] Drop-in integration layer for CUDA graph persistence in vLLM
 - [x] Sync with latest SGLang release
-    - [ ] EP on SGLang
-    - [ ] TP on SGLang
+    - [x] EP on SGLang
+    - [x] Single-shape symmetric-memory TP on SGLang main
+        - [x] Qwen3-8B TP=2 on 2×H100
+        - [x] Qwen3.5-122B-A10B TP=4 on 4×H100
+        - [x] Full-backend prefill persistence with 16/64-token buckets
+    - [x] NEXTN speculative graph persistence on TP=2
+    - [x] DeepEP low-latency graph persistence on TP=2/EP=2
+    - [x] Combined TP=2/EP=2 + NEXTN + DeepEP + Full prefill
+    - [ ] Multi-shape and alternate-collective TP ([upstream issue](https://github.com/foundry-org/foundry/issues/6))
+    - [ ] Other speculative algorithms and breakable/`tc` prefill persistence
+    - [ ] PP, LoRA, PDMux, diffusion, and recapture workflows
 
 ## Stage 5: Disaggregated and Large-Scale Serving
 

csrc/CUDAGraph.cpp

@@ -1528,15 +1528,30 @@ GraphLoadResult CUDAGraph::load(const std::string& json_path, MempoolId_t pool)
   }
 
   const json::array& generators_array = root.at("generators").as_array();
+  bool has_rng_generator = false;
   for (const auto& gen_val : generators_array) {
     const json::object& gen_obj = gen_val.as_object();
-    uint64_t state_id = gen_obj.at("id").to_number<uint64_t>();
-    uint64_t seed = gen_obj.at("seed").to_number<uint64_t>();
-    uint64_t wholegraph_increment = gen_obj.at("wholegraph_increment").to_number<uint64_t>();
+    auto increment_it = gen_obj.find("wholegraph_increment");
+    // Missing metadata is treated conservatively: registering may expose an
+    // allocation mismatch, but skipping could silently leave RNG kernel
+    // arguments pointing at the SAVE process's seed/offset tensors.
+    if (increment_it == gen_obj.end() || increment_it->value().to_number<uint64_t>() != 0) {
+      has_rng_generator = true;
+      break;
+    }
+  }
 
-    auto state = global_generator_state_registry.get_state_from_id(state_id, seed);
-    state->register_graph(reinterpret_cast<at::cuda::CUDAGraph*>(graph.get()));
-    graph->captured_generator_states_[state] = wholegraph_increment;
+  if (has_rng_generator) {
+    for (const auto& gen_val : generators_array) {
+      const json::object& gen_obj = gen_val.as_object();
+      uint64_t state_id = gen_obj.at("id").to_number<uint64_t>();
+      uint64_t seed = gen_obj.at("seed").to_number<uint64_t>();
+      uint64_t wholegraph_increment = gen_obj.at("wholegraph_increment").to_number<uint64_t>();
+
+      auto state = global_generator_state_registry.get_state_from_id(state_id, seed);
+      state->register_graph(reinterpret_cast<at::cuda::CUDAGraph*>(graph.get()));
+      graph->captured_generator_states_[state] = wholegraph_increment;
+    }
   }
 
   const json::object& allocator_events = root.at("allocator_events").as_object();

csrc/CUDAGraphParallel.cpp

@@ -1658,6 +1658,7 @@ std::shared_ptr<PendingGraphLoads> start_graph_builds_impl(
           go["seed"] = gens[g].seed;
           go["wholegraph_increment"] = gens[g].wholegraph_increment;
           gens_array.push_back(go);
+          pending->has_rng_generators |= gens[g].wholegraph_increment != 0;
         }
         pending->entries[i].generators_meta = boost::json::value(std::move(gens_array));
       }
@@ -1690,6 +1691,15 @@ std::shared_ptr<PendingGraphLoads> start_graph_builds_impl(
 
       auto gen_it = pre_root.find("generators");
       if (gen_it != pre_root.end()) {
+        const boost::json::array& generators = gen_it->value().as_array();
+        for (const auto& gen_val : generators) {
+          const boost::json::object& generator = gen_val.as_object();
+          auto increment_it = generator.find("wholegraph_increment");
+          if (increment_it == generator.end() || increment_it->value().to_number<uint64_t>() != 0) {
+            pending->has_rng_generators = true;
+            break;
+          }
+        }
         pending->entries[i].generators_meta = std::move(gen_it->value());
         pre_root.erase(gen_it);
       }
@@ -2030,11 +2040,12 @@ std::shared_ptr<PendingGraphLoads> start_graph_builds_impl(
 // SAVE mode.
 // ============================================================================
 
-// Per-entry finish. Idempotent on build_complete_ wait (shared_future).
+// Per-entry finish. shared_future::get() is repeatable and propagates any
+// background build exception to every caller.
 GraphLoadResult finish_one_graph_load_impl(std::shared_ptr<PendingGraphLoads> pending, size_t index,
                                            ReconstructTensorFn reconstruct_fn) {
   if (pending->build_complete_.valid()) {
-    pending->build_complete_.wait();
+    pending->build_complete_.get();
   }
 
   if (index >= pending->entries.size()) {
@@ -2045,9 +2056,14 @@ GraphLoadResult finish_one_graph_load_impl(std::shared_ptr<PendingGraphLoads> pe
 
   auto& entry = pending->entries[index];
 
-  // Register deferred generators (before allocator replay to match
-  // SAVE mode timing where generators are created before graph capture).
-  if (pending->registry && !entry.generators_meta.is_null()) {
+  // register_graph lazily creates seed/offset tensors before allocator replay.
+  // They must stay in the deterministic region when a captured kernel consumes
+  // Philox state, because their pointers are embedded in that kernel's args.
+  // When the entire pending set has zero RNG increment, no captured node
+  // references those tensors; skip registration entirely. This avoids the
+  // caching-allocator segment growth that caused a 2 MiB A100/FA2 cursor drift
+  // without replacing deterministic pointers with ordinary CUDA allocations.
+  if (pending->has_rng_generators && pending->registry && !entry.generators_meta.is_null()) {
     const boost::json::array& gen_array = entry.generators_meta.as_array();
     for (const auto& gen_val : gen_array) {
       const boost::json::object& gen_obj = gen_val.as_object();