[sglang] No-fabric DeepEP EP: port the NVL CUDA-IPC path to the SGLang integration

.gitignore

@@ -362,4 +362,7 @@ marimo/_lsp/
 __marimo__/
 
 # Streamlit
-.streamlit/secrets.toml
+.streamlit/secrets.toml
+tests/deepep_matrix_work/
+deepep_fabric_archive/
+hook_archive/

docs/sglang/hooks.md

@@ -183,6 +183,13 @@ TP attention is unsupported (its NCCL all-reduce is incompatible with the VMM re
   captured stream (`deep_ep_cpp.Buffer(...)` → "operation not permitted when stream is
   capturing"). The hook forces it before the capture loop, unwrapping the
   `MaybeTboDeepEPDispatcher._inners` to reach a `DeepEPDispatcher`. Runs on SAVE and LOAD.
+- **DeepEP transport policy** (`_patch_deepep`, hooks). On Foundry SAVE/LOAD,
+  the integration wraps `deep_ep.Buffer.__init__` and defaults to
+  `use_fabric=True, num_nvl_bytes=0`. Setting `FOUNDRY_DEEPEP_NVL_IPC=1`
+  instead sets `use_fabric=False` and preserves SGLang's computed nonzero
+  `num_nvl_bytes`, selecting the VMM-backed CUDA-IPC NVL path. DeepEP
+  `29d31c0` or newer is required because it adds this keyword while retaining
+  the older positional constructor shape.
 - **SAVE-only warmup pass** (`_run_warmup_pass`, `capture()` patch). Reuses the upstream
   capture loop with graph capture neutered (run forwards only) to trigger every
   pre-capture lazy init — DeepGEMM per-shape JIT (`stream.synchronize()` is illegal in
@@ -203,7 +210,8 @@ TP attention is unsupported (its NCCL all-reduce is incompatible with the VMM re
   surfaced only on sglang EP.
 
 See [`../../recipe/sglang/README.md`](../../recipe/sglang/README.md) for the EP serve
-config and required kernel versions (deep_ep `9af0e0d`, sgl-deep-gemm ≥0.1.2, fa3).
+config and required kernel versions (deep_ep `29d31c0` or newer,
+sgl-deep-gemm ≥0.1.2, fa3).
 
 ## Patch idiom
 

docs/sglang/overview.md

@@ -19,7 +19,7 @@ serve script is `recipe/sglang/serve_qwen3-30ba3bfp8_ep.sh
 <ep_size> [--save|--load]` with: `--enable-dp-attention --moe-a2a-backend deepep
 --deepep-mode low_latency --moe-runner-backend deep_gemm --attention-backend fa3
 --disable-custom-all-reduce`. Required kernel builds in the env: `deep_ep` at sglang's
-pinned commit (`9af0e0d`, not vLLM's), `sgl-deep-gemm>=0.1.2` (0.1.0 lacks
+pinned commit (`29d31c0` or newer; it adds the `use_fabric` API), `sgl-deep-gemm>=0.1.2` (0.1.0 lacks
 `m_grouped_bf16_gemm_nt_masked`), `flash-attn-3`. `fa3` is required because the
 flashinfer ragged-prefill path has an off-by-one (`q.shape != qo_indptr`) under this
 config. DeepEP low-latency caps dispatch at
@@ -30,6 +30,12 @@ SAVE-only warmup pass (triggers DeepGEMM JIT + buffer creation outside the captu
 stream), `deepep_adapter` mode init on LOAD, the FlashInfer pre-pass gated off for
 fa3, and a C++ fix binding the CUDA context on the graph-build pool workers.
 
+For no-fabric H200 systems, use
+`recipe/experimental/serve_qwen3-30ba3bfp8_sglang_ipc_ep.sh`. With
+`FOUNDRY_DEEPEP_NVL_IPC=1`, Foundry preserves DeepEP's nonzero NVL buffer and
+sets `use_fabric=False`, while the RDMA buffer continues to use NVSHMEM's
+symmetric heap. Use identical SAVE and LOAD settings.
+
 **Per-rank device binding (DP/TP/EP).** Foundry's `set_allocation_region` binds the
 VMM region to the CUDA device current at call time. Upstream sets the device
 *inside* `init_torch_distributed`, which the integration wraps and front-runs, so

docs/vllm/direct-edits.md

@@ -55,6 +55,8 @@ class CompilationConfig:
 
 This is the earliest activation point in the parent process. As soon as a `CompilationConfig` is fully constructed, foundry's runtime patches are in place.
 
+> Note: vLLM folds `graph_extension_config_path` into `CompilationConfig.compute_hash()` (the torch.compile cache key) even though it never affects codegen. This is harmless **as long as every SAVE/LOAD phase passes the same path string** — run all phases from one canonical CWD (or always an absolute path). If pass 1 and pass 2 see the path via different mount aliases (`/home/...` vs `/data/...`), pass 2 misses pass 1's warm cache and inductor recompiles inside the cuda-graph capture window → `cudaErrorStreamCaptureInvalidated`. We keep vLLM unpatched and rely on consistent invocation instead; the EP recipe scripts are launched from the workspace root. (An optional `ignored_factors` exclusion would harden this at the source but is not applied.)
+
 ## 3. `vllm/v1/engine/core.py` (~6 lines)
 
 ```python

python/foundry/__init__.py

@@ -1,5 +1,11 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright contributors to the Foundry project
+# The ops wildcard must come FIRST: .graph's CUDAGraph (and the
+# allocation_region helpers) intentionally override the raw pybind names.
+# isort: off
+from .ops import *
+
+# isort: on
 from .allocation_region import (
     allocation_region,
     free_preallocated_region,
@@ -14,7 +20,6 @@
     graph,
     save_graph_manifest,
 )
-from .ops import *
 
 # Re-exports. Listed here so ruff's --fix doesn't strip them as unused.
 # (We also configure per-file-ignores for F401 in pyproject.toml as a backstop.)

python/foundry/integration/sglang/hooks.py

@@ -5,6 +5,7 @@
 from __future__ import annotations
 
 import functools
+import inspect
 import logging
 import os
 import time
@@ -20,6 +21,7 @@
 
 logger = logging.getLogger(__name__)
 _INSTALLED = False
+_DEEPEP_PATCHED = False
 
 
 def _ep_lazy_init_needed() -> bool:
@@ -92,11 +94,59 @@ def install_hooks(server_args) -> None:
     _patch_kernel_warmup()
     _patch_cuda_graph_capture()
     _patch_spawn_sites()
+    _patch_deepep()
 
     _INSTALLED = True
     logger.info("[Foundry] SGLang hooks installed")
 
 
+def _patch_deepep() -> None:
+    """Select Foundry's DeepEP memory transport for graph SAVE/LOAD.
+
+    DeepEP 29d31c0 added ``use_fabric`` while preserving the older Buffer
+    positional call shape.  Keep the default Foundry mode on the fabric
+    RDMA-only path; the IPC experiment retains DeepEP's NVL allocation and
+    uses the VMM CUDA-IPC bridge in ``libcuda_hook.so``.
+    """
+    global _DEEPEP_PATCHED
+    if _DEEPEP_PATCHED:
+        return
+
+    try:
+        from deep_ep import Buffer
+    except Exception:
+        return
+
+    original_init = Buffer.__init__
+    supports_use_fabric = "use_fabric" in inspect.signature(original_init).parameters
+
+    @functools.wraps(original_init)
+    def patched(self, group, num_nvl_bytes=0, num_rdma_bytes=0, *args, **kwargs):
+        if get_graph_extension_mode() != CUDAGraphExtensionMode.NONE:
+            if not supports_use_fabric:
+                raise RuntimeError(
+                    "[Foundry] SGLang DeepEP requires Buffer(use_fabric=...). "
+                    "Install DeepEP 29d31c0 or newer."
+                )
+            if os.environ.get("FOUNDRY_DEEPEP_NVL_IPC", "0") == "1":
+                kwargs["use_fabric"] = False
+            else:
+                num_nvl_bytes = 0
+                kwargs["use_fabric"] = True
+        return original_init(
+            self,
+            group,
+            num_nvl_bytes,
+            num_rdma_bytes,
+            *args,
+            **kwargs,
+        )
+
+    Buffer.__init__ = patched
+    _DEEPEP_PATCHED = True
+    logger.info("[Foundry] SGLang DeepEP transport patch installed")
+
+
 def _patch_init_torch_distributed() -> None:
     from sglang.srt.model_executor import model_runner as mr
 

python/foundry/integration/vllm/hooks.py

@@ -695,12 +695,22 @@ def _patch_deepep() -> None:
     def patched(self, *args, **kwargs):
         out = orig(self, *args, **kwargs)
         if isinstance(out, dict) and get_graph_extension_mode() != CUDAGraphExtensionMode.NONE:
-            # Force VMM-fabric buffers
-            # so the foundry cuMem hook tracks them, and disable the
-            # NVLink buffer (LL-mode uses RDMA) to avoid fabric-cuMemCreate
-            # collisions with preallocated region.
-            out["use_fabric"] = True
-            out["num_nvl_bytes"] = 0
+            if os.environ.get("FOUNDRY_DEEPEP_NVL_IPC", "0") == "1":
+                # Keep upstream's NVLink buffer (legacy cudaIpc sharing). The
+                # hook's VMM-IPC layer transports the fds via SCM_RIGHTS and
+                # maps peers at relocated VAs, which DeepEP absorbs through
+                # its buffer_ptrs_gpu device table - peer VAs are never baked
+                # into captured graphs. use_fabric stays off: fabric
+                # cuMemCreate needs IMEX, and exercising the IPC path is the
+                # point of this mode.
+                out["use_fabric"] = False
+            else:
+                # Default: force VMM-fabric buffers
+                # so the foundry cuMem hook tracks them, and disable the
+                # NVLink buffer (LL-mode uses RDMA) to avoid fabric-cuMemCreate
+                # collisions with preallocated region.
+                out["use_fabric"] = True
+                out["num_nvl_bytes"] = 0
         return out
 
     cls._make_all2all_kwargs = patched

recipe/experimental/README.md

@@ -0,0 +1,156 @@
+# Foundry recipe — experimental (DeepEP legacy CUDA-IPC NVL buffer)
+
+End-to-end SAVE / LOAD recipe for **Qwen3-30B-A3B** expert-parallel where DeepEP's
+intranode NVLink buffer stays on the **legacy CUDA-IPC** path
+(`cudaIpcGetMemHandle` / `cudaIpcOpenMemHandle`) under foundry, instead of the
+default fabric/NVSHMEM-only path used by `recipe/vllm/serve_qwen3-30ba3b_ep.sh`.
+
+This exercises foundry's **VMM-IPC translation layer**: DeepEP's NVL buffer is a
+foundry VMM (`cuMemCreate`) allocation that legacy IPC can't share, so the hook
+exports it as a POSIX fd and transports that fd to the peer rank via
+`SCM_RIGHTS` over a per-process unix socket, then maps it into the importer's
+own VA range. This is the path that lets foundry SAVE/LOAD work on machines
+**without fabric / IMEX**, where DeepEP would otherwise fail at `Buffer.sync`
+with `cuMemImportFromShareableHandle ... error 999`.
+
+```
+recipe/experimental/
+├── README.md                          # this file
+├── foundry_save.toml                  # SAVE config (workspace_root = "foundry_archive_ipc")
+├── foundry_load.toml                  # LOAD config (same workspace_root)
+├── serve_qwen3-30ba3b_ipc_ep.sh       # Qwen3-30B-A3B (MoE) vLLM EP, NVL/IPC
+├── sglang_foundry_save.toml            # SGLang SAVE config
+├── sglang_foundry_load.toml            # SGLang LOAD config
+└── serve_qwen3-30ba3bfp8_sglang_ipc_ep.sh # SGLang EP, DeepEP NVL/IPC
+```
+
+It is the standard `recipe/vllm` EP recipe plus one switch — `FOUNDRY_DEEPEP_NVL_IPC=1`
+— so read [`../vllm/README.md`](../vllm/README.md) first for installation, the
+two-pass SAVE workflow, the archive layout, and the shared EP flags. Only the
+IPC-specific deltas are documented here.
+
+## Required code
+
+The IPC path needs only **Foundry** changes beyond the standard install — no
+vLLM edit:
+
+- **Foundry** — `foundry/csrc/hook.cpp`: the SCM_RIGHTS VMM-IPC fd transport +
+  whole-chunk peer mapping (handles LOAD-mode buffers carved from the
+  preallocated chunk). **C++ change → rebuild required:**
+  `uv pip install -e . --no-build-isolation` in `foundry/`.
+- **Foundry** — `foundry/python/foundry/integration/vllm/hooks.py`: the
+  `FOUNDRY_DEEPEP_NVL_IPC` env knob in `_patch_deepep` (Python, no rebuild).
+- **Foundry** — `foundry/python/foundry/integration/sglang/hooks.py`: the
+  matching SGLang `deep_ep.Buffer` policy patch (Python, no rebuild).
+
+### Run all phases from one consistent path (no vLLM edit needed)
+
+vLLM folds the foundry TOML path (`graph_extension_config_path`) into its
+torch.compile cache key even though the path never affects codegen. If SAVE
+pass 1 and pass 2 see that path with *different spellings* — e.g. two mount
+aliases of the same dir (`/data/...` vs `/home/...`), or you move the TOML
+between passes — pass 2 misses pass 1's warm compile cache and inductor
+recompiles **inside** the cuda-graph capture window, where its combo-kernel
+benchmark does an illegal `torch.randn` → `cudaErrorStreamCaptureInvalidated`.
+
+The fix is operational, not code: **invoke the script the same way for every
+phase** (same shell / same `cd`, or always an absolute path), so SAVE pass 1,
+SAVE pass 2, and LOAD all pass the identical path string → identical cache hash
+→ pass 2 reuses pass 1's compiled kernels. Run from the workspace root, e.g.:
+
+```bash
+cd <workspace>          # one canonical path for all three phases
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --load
+```
+
+## Workflow
+
+Same two-pass SAVE → LOAD as the base recipe; `<ep_size>` is the first arg:
+
+```bash
+# 0. Fresh start (distinct workspace from the base recipe)
+rm -rf foundry_archive_ipc
+
+# 1. SAVE pass 1 — memory profile + capture
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+# wait for "Application startup complete", then Ctrl-C
+
+# 2. SAVE pass 2 — deterministic re-capture
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+# wait for "Application startup complete", then Ctrl-C
+
+# 3. LOAD — preallocate, re-import IPC buffers, replay graphs
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --load
+# leave running
+
+# 4. Query (separate shell)
+bash ../../../experimental/query.sh 12000 Qwen/Qwen3-30B-A3B
+```
+
+Uncomment `nvshmem_host_path` in both TOMLs first (EP still needs NVSHMEM for the
+DeepEP RDMA buffer; the IPC path only changes the NVL buffer).
+
+## SGLang variant
+
+The SGLang variant is
+`serve_qwen3-30ba3bfp8_sglang_ipc_ep.sh`. It uses the same Foundry VMM-IPC core
+but the SGLang integration wraps `deep_ep.Buffer.__init__` rather than vLLM's
+all-to-all kwargs builder. Use the SGLang-specific TOMLs and run the same
+`--save`/`--load` settings for every phase:
+
+```bash
+rm -rf foundry_archive_sglang_ipc
+bash serve_qwen3-30ba3bfp8_sglang_ipc_ep.sh 2 --save
+bash serve_qwen3-30ba3bfp8_sglang_ipc_ep.sh 2 --load
+```
+
+The SGLang variant requires DeepEP `29d31c0` or newer. This is the vLLM
+installer's validated revision and is API-compatible with SGLang's older
+`Buffer(group, num_nvl_bytes, num_rdma_bytes, ...)` call shape while adding
+`use_fabric`. NVSHMEM remains required for DeepEP's RDMA symmetric heap; only
+the NVL buffer uses CUDA IPC. Keep the model, EP size, low-latency token cap,
+chunked-prefill size, and IPC environment settings identical between SAVE and
+LOAD.
+
+## What `FOUNDRY_DEEPEP_NVL_IPC=1` does
+
+`_patch_deepep` (`foundry/python/foundry/integration/vllm/hooks.py`) normally
+forces `use_fabric=True, num_nvl_bytes=0` so the only cross-GPU buffer is the
+NVSHMEM symmetric heap. With `FOUNDRY_DEEPEP_NVL_IPC=1` it instead keeps
+upstream's nonzero `num_nvl_bytes` with `use_fabric=False`, so the DeepEP Buffer
+allocates the NVLink buffer via `cudaMalloc` + `cudaIpcGetMemHandle` on **both**
+SAVE and LOAD. The hook then:
+
+- **SAVE**: exports each VMM-backed NVL buffer as a POSIX fd, served to peers
+  over `\0foundry-vmm-ipc.<pid>` via `SCM_RIGHTS` (same-uid `SO_PEERCRED` check,
+  per-process token vs PID reuse).
+- **LOAD**: buffers are carved from the preallocated chunk (no individual
+  handle), so the hook exports the **whole chunk** fd + offset; the peer maps the
+  entire chunk once and returns an interior pointer.
+- Peer mappings land at a **relocated** VA (logged `[HOOK] INFO: VMM-IPC import
+  relocated`) — correct, because DeepEP resolves peers through its device-side
+  `buffer_ptrs_gpu` table (refreshed by `Buffer.sync`), never through addresses
+  baked into captured graphs.
+
+A healthy run logs two `VMM-IPC import relocated` lines per phase (one per peer)
+and **no** `error 999`.
+
+## Validation status
+
+SAVE → SAVE → LOAD → query verified on Qwen3-30B-A3B EP=2 (2× H200, no IMEX):
+per-rank `final_alloc_offset` identical across passes, LOAD replays at the saved
+offset, query returns coherent completions. LOAD reaches a serving server in
+~27 s; the IPC import itself is sub-second (inside the ~7.5 s `load_model`) and
+has no steady-state serving cost — peer addresses are resolved once at init via
+the device pointer table, not per token.
+
+## IPC-specific troubleshooting
+
+| Symptom | Likely cause |
+|---|---|
+| `[HOOK] ERROR: cuMemImportFromShareableHandle failed with error 999` at `deep_ep.cpp` `Buffer::sync` | Foundry hook not rebuilt with the SCM_RIGHTS transport — it's still packing a raw fd. Rebuild `foundry/` (`uv pip install -e . --no-build-isolation`). |
+| `operation failed due to a previous error during capture` on SAVE **pass 2** | vLLM compile-cache over-keying not applied (`graph_extension_config_path` still hashed) → recompile-in-capture. Apply the `compilation.py` `ignored_factors` edit, or set `FOUNDRY_DISABLE_COMBO_KERNELS=1` (opt-in belt-and-suspenders, wired in the experimental serve script under `experimental/expert-parallel/`). |
+| LOAD `illegal memory access` at replay with an NVL buffer present | Relocated peer import collided with the NVSHMEM heap hint — ensure the hook build includes the dedicated import-VA zone (`0x300000000000`). |
+| `error 999` only with `--deepep-mode auto`/`normal` | That's expected for non-LL modes here; this recipe pins `deepep_low_latency`. |

recipe/experimental/foundry_load.toml

@@ -0,0 +1,9 @@
+mode = "load"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc"
+scratch_space_size = "4096MB"
+# REQUIRED for the EP/IPC recipe — DeepEP low-latency still allocates its RDMA
+# buffer on the NVSHMEM symmetric heap (the NVL buffer is the cudaIpc path).
+# Point this at the libnvshmem_host.so built by vllm/tools/ep_kernels.
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_load_fp8.toml

@@ -0,0 +1,6 @@
+mode = "load"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc_fp8"
+scratch_space_size = "4096MB"
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_save.toml

@@ -0,0 +1,9 @@
+mode = "save"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc"
+scratch_space_size = "4096MB"
+# REQUIRED for the EP/IPC recipe — DeepEP low-latency still allocates its RDMA
+# buffer on the NVSHMEM symmetric heap (the NVL buffer is the cudaIpc path).
+# Point this at the libnvshmem_host.so built by vllm/tools/ep_kernels.
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_save_fp8.toml

@@ -0,0 +1,6 @@
+mode = "save"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc_fp8"
+scratch_space_size = "4096MB"
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"