Consolidate TP support and validate vLLM and SGLang

.gitignore

@@ -362,4 +362,7 @@ marimo/_lsp/
 __marimo__/
 
 # Streamlit
-.streamlit/secrets.toml
+.streamlit/secrets.toml
+tests/deepep_matrix_work/
+deepep_fabric_archive/
+hook_archive/

README.md

@@ -85,7 +85,35 @@ Foundry ships engine integrations under `foundry/python/foundry/integration/`. P
 | SGLang | ✅ | ✅ | 🚧 | ✅ |
 | TensorRT-LLM | 🚧 | 🚧 | 🚧 | 🚧 |
 
-✅ validated end-to-end (SAVE → LOAD → query)  ·  🚧 not yet
+✅ officially validated end-to-end  ·  🚧 not official/general
+
+An experimental SGLang dense-TP recipe is validated with Qwen3-8B at TP=2 on
+2×H100. Both ranks restore 36 graphs at their saved allocation offsets, and
+deterministic LOAD responses match a non-Foundry baseline using the same pinned
+settings. This is not general TP support: upstream
+[Discussion #5](https://github.com/orgs/foundry-org/discussions/5) reports a
+successful internal torch symmetric-memory prototype but no official
+implementation; support remains tracked in
+[issue #6](https://github.com/foundry-org/foundry/issues/6).
+
+SGLang-main also has an opt-in torch symmetric-memory TP=2 path. Foundry records
+the process-local communication buffer and pointer-table operands, restores the
+per-shape warmups, and relocates those operands to the fresh LOAD communicator.
+The checked-in 2×H100 proof validates one self-contained batch-1 CUDA graph per
+rank at `final_alloc_offset=68555898880`: baseline, two SAVE passes, and fresh
+LOAD return byte-identical deterministic responses with no recapture or runtime
+errors. Larger batches fall back to eager execution. Multi-shape symmetric
+capture remains rejected because later graphs depend on mutable FlashInfer state
+owned by earlier shapes.
+
+An experimental vLLM dense-TP recipe is validated with Qwen3-8B at TP=2 on
+2×H100. Both ranks restore 64 graphs at the same per-rank offset produced by
+repeated deterministic SAVE passes, without recapture. Sequential and
+concurrent LOAD responses match a non-Foundry baseline using identical pinned
+NCCL settings. As with SGLang, this is a pinned result rather than general TP
+support; upstream reports an unpublished torch symmetric-memory prototype
+([Discussion #5](https://github.com/orgs/foundry-org/discussions/5),
+[issue #6](https://github.com/foundry-org/foundry/issues/6)).
 
 The adapted vLLM / SGLang / TensorRT-LLM forks will be released alongside this repo at `foundry-org/vllm`, `foundry-org/sglang`, `foundry-org/TensorRT-LLM`.
 

RELEASE.md

@@ -1,10 +1,55 @@
-# Foundry 0.0.2
+# Release Notes
+
+## Unreleased
+
+- **Experimental vLLM dense tensor parallel.** Qwen3-8B TP=2 is validated
+  end-to-end on 2×H100 with NCCL P2P/IPC. Deterministic SAVE passes produce
+  identical per-rank offsets and graph inventories; fresh LOAD restores 64
+  graphs per rank without capture and returns byte-identical sequential and
+  concurrent factual completions. General TP remains open in
+  [upstream issue #6](https://github.com/foundry-org/foundry/issues/6);
+  [Discussion #5](https://github.com/orgs/foundry-org/discussions/5) reports an
+  internal torch symmetric-memory prototype but no official implementation.
+- **Experimental SGLang dense tensor parallel.** Qwen3-8B TP=2 is validated
+  end-to-end on 2×H100 with NCCL P2P/IPC. Fresh LOAD restores 36 graphs per
+  rank at the recorded allocation offset and returns byte-identical
+  temperature-0 outputs versus a non-Foundry baseline using identical pinned
+  settings. General TP remains open in
+  [upstream issue #6](https://github.com/foundry-org/foundry/issues/6);
+  [Discussion #5](https://github.com/orgs/foundry-org/discussions/5) reports an
+  unpublished torch symmetric-memory prototype. SGLang-main now has a validated
+  TP=2 symmetric-memory option for one batch-1 CUDA graph: SAVE records the
+  external communication operands and full graph JSON, while LOAD rebuilds
+  required warmup state and relocates those operands to the live communicator.
+  A 2×H100 baseline/SAVE/SAVE/LOAD matrix returned byte-identical outputs at
+  `final_alloc_offset=68555898880`. Larger batches use eager execution;
+  multi-shape symmetric graph capture remains fail-closed.
+- **No-fabric DeepEP IPC.** The VMM-IPC bridge transports shareable file
+  descriptors with `SCM_RIGHTS`, allowing vLLM and SGLang DeepEP NVL buffers to
+  work without fabric/IMEX. This extends
+  [upstream PR #3](https://github.com/foundry-org/foundry/pull/3). The verified
+  regression is BF16 on 2×H100; GB300/aarch64 FP8 and RDC relinking reported in
+  [upstream issue #1](https://github.com/foundry-org/foundry/issues/1) remain
+  outside that validation scope.
+- **FA2 workspace-parity fix.** CUDA generator registration is skipped on LOAD
+  when every graph has zero RNG increment, preventing unused seed/offset tensors
+  from drifting the tracked workspace cursor. This incorporates
+  [upstream PR #4](https://github.com/foundry-org/foundry/pull/4) and the root
+  cause from [issue #2](https://github.com/foundry-org/foundry/issues/2);
+  this branch did not independently rerun the A100 hardware case.
+- **CUDA default-alignment VMM reservations.** `cuMemAddressReserve` accepts
+  `alignment=0` to request the driver default. Foundry now preserves the current
+  VMM cursor for that value instead of bit-masking it to address zero.
+
+---
+
+## Foundry 0.0.2
 
 SGLang graduates to a fully validated engine. This release brings the SGLang
 integration to parity with vLLM across single GPU, data parallel, and expert
 parallel — with a self-contained recipe and no vLLM build dependency for EP.
 
-## Highlights
+### Highlights
 
 - **SGLang single GPU / DP / EP all validated end-to-end.**
   SAVE → LOAD → query verified on single-GPU Qwen3-1.7B / 4B / 14B,
@@ -23,7 +68,7 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   `data_parallel_controller.py`). All save/load logic lives in the integration
   layer; the edits are inert unless `--foundry-graph-extension-config-path` is set.
 
-## Engine integrations
+### Engine integrations
 
 - **SGLang** — integration for SGLang v0.5.13. Working configurations: single
   GPU, data parallel (DP), expert parallel (EP, DeepEP low-latency + DP-attention with fa3). Self-contained recipes under `recipe/sglang/` (shared TOML pair +
@@ -45,7 +90,7 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   on LOAD, with `init_nvshmem_for_loaded_modules` run once on LOAD before any
   NVSHMEM-kernel graph replays.
 
-## Fixes
+### Fixes
 
 - **Per-rank VMM device binding (DP/TP/EP).** `set_allocation_region` binds to
   the current CUDA device, so the integration now calls `set_device(gpu_id)`
@@ -63,26 +108,22 @@ parallel — with a self-contained recipe and no vLLM build dependency for EP.
   capture loop never sets), and the FlashInfer per-bs pre-pass gated off for fa3
   while still populating `decode_cuda_graph_metadata` post-load for replay.
 
-## Docs
+### Docs
 
 - New `docs/sglang/` set (overview, direct-edits, hooks, memory-lifecycle,
   save-load-workflow, memory-consistency) and a self-contained `recipe/sglang/`
   README with install, run, performance, and troubleshooting.
 - Top-level README parallelism status table updated to mark SGLang single GPU,
   DP, and EP as validated.
 
----
-
-## Previous Releases
-
 ## Foundry 0.0.1
 
 First public release of Foundry — a CUDA-graph persistence library that
 captures an entire model's CUDA graphs (plus their device context: modules,
 workspaces, VMM layout) once and replays them at startup, eliminating compile,
 warmup, and capture from cold-start time.
 
-## Highlights
+### Highlights
 
 - **Deterministic memory layout — zero patching on graph load.**
   Foundry indirects memory allocation to the same reserved memory region
@@ -99,7 +140,7 @@ warmup, and capture from cold-start time.
   On load, the template is rebuilt once per group and instances are reconstructed
   on demand, keeping load fast and asynchronous.
 
-## Engine integrations
+### Engine integrations
 
 - **vLLM** — compatible with vLLM v0.21. Working configurations: single GPU,
   data parallel (DP), expert parallel (EP, DeepEP low-latency). End-to-end
@@ -111,31 +152,31 @@ warmup, and capture from cold-start time.
   integration layer (`foundry/integration/<engine>/`); engine forks contain
   only minimal hook calls.
 
-## Verified kernel & comm support
+### Verified kernel & comm support
 
 - **cuBLAS NVJET** kernels (Hopper+).
 - **torch.compile** modules.
 - **NVSHMEM / DeepEP** validated.
 - **DeepGEMM FP8 MoE** validated.
 
-## Dependency
+### Dependency
 
 - **PyTorch 2.11.0** (compatible with 2.9 – 2.11).
 - **CUDA 12+**, CMake 4.0+, Boost.
 
-## Documentation
+### Documentation
 
 - Integration design notes and per-engine recipes under `docs/` and `recipe/`.
 - vLLM recipe README covers save/load workflow, archive layout, and required
   env settings.
 
-## Repository hygiene
+### Repository hygiene
 
 - Open-source pre-commit hooks (ruff, ruff-format, clang-format,
   markdownlint, actionlint, DCO sign-off).
 - Smoke tests covering re-export imports and archive round-trip.
 
-## Roadmap
+### Roadmap
 
 - Adapted vLLM and SGLang forks published alongside the release.
 - Tensor parallel support.

ROADMAP.md

@@ -26,12 +26,18 @@
 
 - [x] Sync with latest vLLM release
     - [x] EP on vLLM with DeepEP
-    - [ ] TP on vLLM
+    - [ ] Official TP on vLLM
+        - [x] Experimental NCCL TP=2 recipe on 2×H100
+        - [ ] Evaluate/publish the torch symmetric-memory prototype ([upstream discussion](https://github.com/orgs/foundry-org/discussions/5))
     - [x] Quantized MoE with DeepGemm
 - [x] Drop-in integration layer for CUDA graph persistence in vLLM
 - [x] Sync with latest SGLang release
-    - [ ] EP on SGLang
-    - [ ] TP on SGLang
+    - [x] EP on SGLang
+    - [ ] Official TP on SGLang
+        - [x] Experimental NCCL TP=2 recipe on 2×H100
+        - [x] Reproduce torch symmetric-memory fresh-LOAD corruption on 2×H100
+        - [x] Relocate process-local symmetric-memory operands for a TP=2 batch-1 graph
+        - [ ] Generalize symmetric-memory replay to multiple graph shapes ([upstream discussion](https://github.com/orgs/foundry-org/discussions/5))
 
 ## Stage 5: Disaggregated and Large-Scale Serving
 

csrc/CUDAGraph.cpp

@@ -1528,15 +1528,30 @@ GraphLoadResult CUDAGraph::load(const std::string& json_path, MempoolId_t pool)
   }
 
   const json::array& generators_array = root.at("generators").as_array();
+  bool has_rng_generator = false;
   for (const auto& gen_val : generators_array) {
     const json::object& gen_obj = gen_val.as_object();
-    uint64_t state_id = gen_obj.at("id").to_number<uint64_t>();
-    uint64_t seed = gen_obj.at("seed").to_number<uint64_t>();
-    uint64_t wholegraph_increment = gen_obj.at("wholegraph_increment").to_number<uint64_t>();
+    auto increment_it = gen_obj.find("wholegraph_increment");
+    // Missing metadata is treated conservatively: registering may expose an
+    // allocation mismatch, but skipping could silently leave RNG kernel
+    // arguments pointing at the SAVE process's seed/offset tensors.
+    if (increment_it == gen_obj.end() || increment_it->value().to_number<uint64_t>() != 0) {
+      has_rng_generator = true;
+      break;
+    }
+  }
 
-    auto state = global_generator_state_registry.get_state_from_id(state_id, seed);
-    state->register_graph(reinterpret_cast<at::cuda::CUDAGraph*>(graph.get()));
-    graph->captured_generator_states_[state] = wholegraph_increment;
+  if (has_rng_generator) {
+    for (const auto& gen_val : generators_array) {
+      const json::object& gen_obj = gen_val.as_object();
+      uint64_t state_id = gen_obj.at("id").to_number<uint64_t>();
+      uint64_t seed = gen_obj.at("seed").to_number<uint64_t>();
+      uint64_t wholegraph_increment = gen_obj.at("wholegraph_increment").to_number<uint64_t>();
+
+      auto state = global_generator_state_registry.get_state_from_id(state_id, seed);
+      state->register_graph(reinterpret_cast<at::cuda::CUDAGraph*>(graph.get()));
+      graph->captured_generator_states_[state] = wholegraph_increment;
+    }
   }
 
   const json::object& allocator_events = root.at("allocator_events").as_object();

csrc/CUDAGraphParallel.cpp

@@ -1658,6 +1658,7 @@ std::shared_ptr<PendingGraphLoads> start_graph_builds_impl(
           go["seed"] = gens[g].seed;
           go["wholegraph_increment"] = gens[g].wholegraph_increment;
           gens_array.push_back(go);
+          pending->has_rng_generators |= gens[g].wholegraph_increment != 0;
         }
         pending->entries[i].generators_meta = boost::json::value(std::move(gens_array));
       }
@@ -1690,6 +1691,15 @@ std::shared_ptr<PendingGraphLoads> start_graph_builds_impl(
 
       auto gen_it = pre_root.find("generators");
       if (gen_it != pre_root.end()) {
+        const boost::json::array& generators = gen_it->value().as_array();
+        for (const auto& gen_val : generators) {
+          const boost::json::object& generator = gen_val.as_object();
+          auto increment_it = generator.find("wholegraph_increment");
+          if (increment_it == generator.end() || increment_it->value().to_number<uint64_t>() != 0) {
+            pending->has_rng_generators = true;
+            break;
+          }
+        }
         pending->entries[i].generators_meta = std::move(gen_it->value());
         pre_root.erase(gen_it);
       }
@@ -2045,9 +2055,14 @@ GraphLoadResult finish_one_graph_load_impl(std::shared_ptr<PendingGraphLoads> pe
 
   auto& entry = pending->entries[index];
 
-  // Register deferred generators (before allocator replay to match
-  // SAVE mode timing where generators are created before graph capture).
-  if (pending->registry && !entry.generators_meta.is_null()) {
+  // register_graph lazily creates seed/offset tensors before allocator replay.
+  // They must stay in the deterministic region when a captured kernel consumes
+  // Philox state, because their pointers are embedded in that kernel's args.
+  // When the entire pending set has zero RNG increment, no captured node
+  // references those tensors; skip registration entirely. This avoids the
+  // caching-allocator segment growth that caused a 2 MiB A100/FA2 cursor drift
+  // without replacing deterministic pointers with ordinary CUDA allocations.
+  if (pending->has_rng_generators && pending->registry && !entry.generators_meta.is_null()) {
     const boost::json::array& gen_array = entry.generators_meta.as_array();
     for (const auto& gen_val : gen_array) {
       const boost::json::object& gen_obj = gen_val.as_object();