Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a2f8e2e
test(sglang): define v0.5.15 graph contracts
cursoragent Jul 23, 2026
f4137ca
test(sglang): compare archive records structurally
cursoragent Jul 23, 2026
f271e0f
feat(sglang): port hooks to split CUDA graph runner
cursoragent Jul 23, 2026
78c399d
feat(sglang): version graph archives
cursoragent Jul 23, 2026
21b06e1
fix(sglang): validate usable graph archives
cursoragent Jul 23, 2026
5a57c88
docs(sglang): update v0.5.15 recipes
cursoragent Jul 23, 2026
820091a
docs(sglang): fix editable install guidance
cursoragent Jul 23, 2026
75972fc
fix(hook): preserve cursor for default VMM alignment
cursoragent Jul 23, 2026
2585c85
style(hook): format zero-alignment guard
cursoragent Jul 23, 2026
3a24fc3
fix(sglang): restore DeepEP allocation symmetry
cursoragent Jul 23, 2026
094819f
fix(sglang): propagate LOAD preallocation failures
cursoragent Jul 23, 2026
3f05720
docs(sglang): record H100 validation
cursoragent Jul 23, 2026
e11c93d
fix(sglang): isolate optional CUDA graph imports
cursoragent Jul 23, 2026
c488d5a
fix(sglang): require per-rank load offset
cursoragent Jul 23, 2026
3864a4c
docs(sglang): clarify per-rank allocation watermark
cursoragent Jul 23, 2026
fdb3ecc
feat(sglang): validate TP2 DP2 topology
cursoragent Jul 23, 2026
d33d996
feat(sglang): add TP2 DP2 serve recipe
cursoragent Jul 23, 2026
de92ab1
fix: harden VMM IPC across processes
cursoragent Jul 23, 2026
6394fc9
test(sglang): cover validated EP4 topology
cursoragent Jul 23, 2026
973e9a0
docs(sglang): record four-GPU validation
cursoragent Jul 23, 2026
7d70c6f
fix(sglang): clarify TP attention validation error
cursoragent Jul 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,32 @@ Foundry ships engine integrations under `foundry/python/foundry/integration/`. P
| Engine | Single GPU | DP | TP | EP |
|---|:---:|:---:|:---:|:---:|
| vLLM | ✅ | ✅ | 🚧 | ✅ |
| SGLang | ✅ | ✅ | 🚧 | ✅ |
| SGLang `0.5.15.post1` | ✅ | ✅ | ✅[^sglang-tp] | ✅ |
| TensorRT-LLM | 🚧 | 🚧 | 🚧 | 🚧 |

✅ validated end-to-end (SAVE → LOAD → query)  ·  🚧 not yet

The adapted vLLM / SGLang / TensorRT-LLM forks will be released alongside this repo at `foundry-org/vllm`, `foundry-org/sglang`, `foundry-org/TensorRT-LLM`.
[^sglang-tp]: SGLang conventional tensor parallelism is validated only for
dense TP=2 × DP=2 with PP=1 and EP=1. All other conventional TP topologies
remain unsupported.

The SGLang integration is latest-only for upstream `v0.5.15.post1`
(`0b3bb0c`), CUDA 13.0, and PyTorch 2.11/cu130 on H100. Full decode with
prefill graphs disabled is validated for single-GPU FlashInfer and regular
DP=2 FlashInfer on Qwen3-1.7B (52 graphs/rank), exact dense TP=2 × DP=2
FlashInfer on Qwen3-8B (36 graphs/rank), and DP-attention EP=2 and EP=4 with
DeepEP low-latency, DeepGEMM, and FA3 on Qwen3-30B-A3B-FP8 (20 graphs/rank).
All passed baseline/SAVE/fresh LOAD/query with text/token parity and no
recapture. EP=4 uses DP-attention/EP rather than conventional TP attention,
even though SGLang's process topology sets `tp_size=4`. Existing SGLang
archives are incompatible and require a fresh SAVE. See
[`recipe/sglang/README.md`](recipe/sglang/README.md).

For the current Foundry-only SGLang workflow, check out upstream
`sgl-project/sglang` at the tagged release and apply the documented four-file
patch. `foundry-org/sglang` should carry that same patch when its release branch
is published; the current instructions do not assume that branch already
exists. Adapted engine forks will be released alongside this repository.

### Performance

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@
- [x] Quantized MoE with DeepGemm
- [x] Drop-in integration layer for CUDA graph persistence in vLLM
- [x] Sync with latest SGLang release
- [ ] EP on SGLang
- [ ] TP on SGLang
- [x] EP on SGLang (DP-attention + DeepEP, validated at EP=2 and EP=4)
- [x] Exact dense TP=2 × DP=2 on SGLang (all other TP topologies unsupported)

## Stage 5: Disaggregated and Large-Scale Serving

Expand Down
Loading