Skip to content

[GPU] Qwen3-30B-A3B standalone FFN service (re-opened) - #578

Open
ronenkat wants to merge 39 commits into
vllm-project:mainfrom
ms-llmd:ffn-faas-qwen3-30b
Open

[GPU] Qwen3-30B-A3B standalone FFN service (re-opened)#578
ronenkat wants to merge 39 commits into
vllm-project:mainfrom
ms-llmd:ffn-faas-qwen3-30b

Conversation

@ronenkat

@ronenkat ronenkat commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a GPU implementation of the standalone FFN service for BF16
Qwen3-30B-A3B, supporting the existing DeepEPM2NAFDConnector over M2N. FFN
ranks run under torchrun, build a GPU-only NCCL/EP world, load only routed
experts, and reuse the existing GPU FFN runner and connector for eager
execution, FULL_DECODE_ONLY CUDA graphs, and two-lane DBO. The P2P connector
is out of scope for this PR.

  • Feedback is welcome!

Issue

Related issue(s): #552
Closing keyword, only if fully resolved: #552

Implementation

  • Registers the gpu-deepep-m2n backend and adds CPU-side launch, checkpoint,
    topology, and connector-option validation.
  • Initializes the F-only GPU world with torchrun coordinates, vLLM external
    launcher semantics, data-parallel rank selection, expert parallelism, and a
    rank-zero-hosted M2N rendezvous.
  • Loads only Qwen3 routed experts through vLLM's native loader, then attaches
    the externally owned model to GPUFFNRoutedExpertsModelRunner.
  • Reuses the existing M2N control plane, DeepEP buffers, bucket ladder, CUDA
    graph capture/replay, and two-lane DBO execution instead of duplicating the
    Worker-path data plane.
  • Adds host-side idle-receiver cancellation and clean resource teardown, with
    no vLLM Worker, EngineCore, scheduler, KV cache, or complete CausalLM on the
    FFN side.
  • Refactors shared GPU/NPU launcher validation, F-only engine arguments,
    environment invariants, parameter-storage accounting, and Qwen3 routed-expert
    construction.

Performance evaluation

Tests

  • FFN unit subset: 170 passed.
  • Full unit run: 1530 tests, 0 failures, 48 skips, and 1 pre-existing setup
    error.
  • GPU smoke on 4xH100: eager, graph capture/replay, and Attention-first failure
    passed; combined output matched the dense reference.
  • Qwen3 M2N E2E: eager, graph, graph-DBO, and graph-TP2 2A2F scenarios passed
    GSM8K-100 with clean FFN teardown.

jiaran-king and others added 30 commits October 25, 2026 16:46
@jiaran-king
fix(npu): apply V4 SwiGLU limits in layered W4A8
92d1a30 
Clamp reconstructed gate and up activations before SwiGLU and INT8 quantization. Keep the original unlimited ACLNN ABI and expose a separate clamp-aware entry point. This common correctness prerequisite contains no transport or scheduling changes.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
feat(npu): run standalone V4 FFN service over mature CAM
a26b392 
Load and retain only native routed W4A8 parameters in an F-only launcher, then compose the existing Async CAM runner, startup, graphs and two-stream execution. Keep Attention and the transport contract unchanged. Retain owners on failed or pending receive and document whole-pair process isolation.

Validation: 40 CPU service contracts, 131 pinned-stack unit contracts, 24 native numerical cases, finite synthetic eager/FULL CAM and original 43-layer A1/F4 eager versus FULL two-stream exact output comparison. Pending-receive isolation covered in both modes. No inference service benchmark or GSM8K rerun.
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
fix(npu): preserve FFN isolation exit when diagnostics fail
d136157 
Keep the unreleased-session exit in an outer finally, capture release proof before reporting, and make traceback, status output and HTTP cleanup best effort without turning failures into success. Preserve normal teardown when native resources are confirmed released.

Validate the CLI in real host child processes: 12 exit regressions and 52 CPU service cases pass. Reuse native and packing evidence for unchanged execution paths; no new device experiment. Clarify that the 12A/4F Attention TP4/DBO setup remains an unqualified integration example.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
docs(npu): qualify real standalone CAM 12A4F integration
7ebcf77 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
refactor(npu): isolate FFN adapters and retain native expert context
9151609 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
docs(npu): qualify retained FFN ownership and prefill parity
540ceb8 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
test(ffn): guard first imports and document CAM step scope
70cad14 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
refactor(ffn): separate CAM model resources from V4 loading hooks
d3c6198 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
docs(npu): qualify the FaaS model integration boundary
5ebffba 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
refactor(faas): colocate V4 checkpoint description and adapter
2e1cccb 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
refactor(faas): remove shared SwiGLU clamp fix from service scope
cfb72da 
Restore the existing layered operator, bindings and numerical tests. Keep the standalone service integration and its runtime attachment hooks. Remove the clamp API documentation from the service PR; preserve the isolated fix separately.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
docs(faas): focus documentation on current service contracts
0b97807 
Keep supported configuration, module ownership, launch and shutdown behavior. Remove historical validation, performance results and review-process narrative from product documentation.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
Merge main into PR vllm-project#490 standalone FFN service
e849f5c 
Resolve the operator documentation, runtime ownership paths and runtime test
conflicts while preserving current main behavior and standalone model attachment.
Adapt routed weights to the current layered executor interface, retaining W4A8
selection. Use StrEnum under the current Python 3.11 lint contract.

Validation: 79 CPU cases passed, with the NPU-dependent runtime module skipped;
33 affected service/runner/CLI cases passed after the enum change. Applicable
pre-commit checks and mypy passed. No new NPU or performance run.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@ronenkat @claude
[GPU] Standalone FFN service for Qwen3-30B-A3B over DeepEP M2N
f5ee720 
Add the gpu-deepep-m2n backend to the standalone FFN service
(python -m afd_plugin.ffn_faas serve): BF16 Qwen3MoeForCausalLM routed
experts over DeepEPM2NAFDConnector, eager, FULL_DECODE_ONLY graphs and
two-lane DBO, without a vLLM Worker, EngineCore, scheduler, KV cache or
complete CausalLM.

- Host protocol: FFNBackend.execute_step() returns bool and gains
  cancel_receive(); the host cancels an idle receive before joining.
  The NPU backend returns True and keeps a no-op cancel.
- GPUFFNModelRunner.execute_connector_driven_step(): the Worker loop
  body, now shared by AFDFFNWorker and the standalone backend.
- GPUFFNRoutedExpertsModelRunner.attach_model() and
  mark_receiver_exited() for an externally loaded model.
- ffn_faas/platforms/gpu: CPU-only adapter, settings and Qwen3
  checkpoint description; GPU environment (F-only NCCL world, AFD
  architecture, Triton MoE pin); FFN-only Qwen3 model shell on vLLM's
  native loader; DeepEPM2NFFNBackend.
- Tests: CPU unit tests, a 4-GPU service smoke in tests/gpu, and four
  Qwen3 M2N FFN-service E2E scenarios in the E2E runner.
- Docs: GPU standalone guide and FFN runtime design update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
[GPU] FFN service: host the M2N rendezvous under torchrun
626e909 
torchrun sets TORCHELASTIC_USE_AGENT_STORE=True in its workers, so every
tcp:// rendezvous joins the agent's store as a client. The F-only world
needs that, but the connector's M2N rendezvous must be hosted by F rank 0;
with the flag set nobody listened and both roles timed out. Drop the flag
once vLLM's F-only groups exist.

Also fix the new runner unit test (AFDControlPayload needs explicit flags)
and the GPU smoke harness (interleaved per-rank announcements, and kill
torchrun's workers, which are outside the launcher's process group).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
[GPU] FFN service guide: Attention-first failure behavior
8c114ae 
Measured by the GPU smoke test: the NCCL watchdog aborts the F ranks after afd_process_group_timeout_s, so no status file is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
refactor qwen3_moe
6fa7148 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@jiaran-king
Merge main into PR vllm-project#490 standalone FFN service
bd5dc94 
Keep standalone model attachment with the updated Async CAM diagnostics. Follow vllm-project#550 by removing obsolete FFN two-stream options and configuration keys.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@jiaran-king
refactor(ffn): reuse native engine CLI and AFD configuration
b26bdc3 
Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@ronenkat
Share FFN launcher environment checks
717ca64 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Share standalone FFN launch validation
288f323 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Share F-only FFN engine arguments
4b20b9e 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Share FFN model attachment validation
ad89f42 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Share standalone FFN environment invariants
b57c2ab 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Load Qwen3 FFN-only routed experts through vLLM
46772c2 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
Fix GPU worker unit-test configuration stubs
a607c3d 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat
sync with faas branch
25cb087 
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
[GPU] FFN service: type the loaded FFN-only model
3003fc9 
GPUFFNRoutedExpertsModelRunner.get_model() returns Any, so assigning it to
the Optional self.model attribute left the value typed as possibly None
when it was passed to parameter_storage_bytes, and the mypy-3.11
pre-commit hook failed on backend.py. Bind the loaded model to a typed
local first, then store it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
test(e2e): sweep FFN service torchrun workers on teardown
f86a41c 
torchrun starts its workers in their own sessions, outside the launcher
process group that the E2E harness owns and signals. A SIGTERM to
torchrun lets it stop the workers itself (a cancelled FFN service case
on 4xH100 left no workers behind), but when torchrun does not exit
within the harness grace period, the group SIGKILL reaches only torchrun
and its workers survive holding GPUs into the next case
(E2E-INV-003/E2E-INV-006). Give the FFN service scenarios an E2E run id
so the existing environment-matched FFN sweep also reaps the torchrun
workers, as the NPU async path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
Inline the NPU FFN attach guard
d1b4c80 
validate_external_model_attachment was extracted for the GPU and NPU
attach_model methods. The GPU attach_model was removed when the
standalone GPU service moved to loading through the routed runner's
native load_model, so the helper had a single NPU caller and a new
v1/worker module with no owning design page. Restore the guard inline in
AsyncNPUFFNModelRunner.attach_model and delete the module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
ronenkat and others added 9 commits November 7, 2026 07:00
@ronenkat
Make GPU FFN service model agnostic
b772817 
Resolve the model path in the CPU adapter and let vLLM model config plus the AFD registry select the FFN-only shell. Remove the duplicate Qwen3 checkpoint parser and derive expert progress from the loaded runner.

Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@jiaran-king
Merge main into PR vllm-project#490 native CLI alignment
47c75d8 
Retain standalone CLI/configuration reuse while adopting the Async CAM session ownership refactor and current runtime documentation.

Signed-off-by: jiaran-king <jiaranran2@gmail.com>
@ronenkat @claude
Merge codex/ffn-faas-framework: adopt the native engine CLI for GPU FFN
7e098ef 
The base's "reuse native engine CLI and AFD configuration" refactor
replaced FFNPlatformAdapter.add_arguments with create_parser (vLLM's
EngineArgs parser, external_launcher by default), moved topology and
connector options into --additional-config, renamed the receiver join
timeout to --receiver-shutdown-timeout and reduced the NPU FFNConfig to
launcher facts.

Conflicts (npu adapter, backend, environment, settings) take the base
behavior. The shared launch module now owns the native parser and the
F-only checks both adapters run (positional model, FFN-role AFD config,
WORLD_SIZE = --data-parallel-size = afd.num_ffn_ranks, DP rank from
RANK, external launcher) plus rank/shutdown validation. The NPU
environment keeps the shared base with the base's native device mapping.

The GPU backend moves to the same contract: options come from
--additional-config, --enable-dbo, and --enforce-eager or
--compilation-config FULL_DECODE_ONLY; TP 1 with expert parallel is
required; the tokenizer skip and EP weight filter are parser defaults;
the F-only VllmConfig is built with EngineArgs.from_cli_args. Removed
--host/--port/--attention-ranks/--ffn-ranks/--graph/--dbo/
--connector-extra-config/--afd-timeout. Unit tests, the GPU smoke test,
the E2E runner command and the GPU guide follow.

Also narrow the loaded FFN-only model to a torch module before counting
its parameter storage (mypy).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
test(ffn_faas): GPU FFN rank must honor --device-ids
1f783fc 
Add a regression test for the device-selection review finding: with
--device-ids 2,3 and LOCAL_RANK=1 the standalone GPU FFN rank binds
cuda:1 instead of GPU 3. vLLM only stores the resolved IDs in
parallel_config.assigned_physical_gpu_ids; its GPU Worker registers them
with set_assigned_physical_gpu_ids before mapping the local rank to a
device, and the F path skips that step and selects the device before the
config exists.

The test drives DeepEPM2NFFNBackend.prepare from the native CLI and stops
when the rank binds its GPU. It fails on this branch and passes once the
config is built first and the mapping is registered before selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
fix(ffn_faas): apply --device-ids before selecting the GPU
942c37b 
The standalone GPU FFN rank ignored --device-ids and bound
cuda:<LOCAL_RANK>, so FFN ranks could land on the Attention GPUs.
vLLM only stores the resolved IDs in
parallel_config.assigned_physical_gpu_ids; its GPU Worker registers them
with set_assigned_physical_gpu_ids before mapping the local rank to a
device. The F path selected the device before the config even existed.

Build the F-only config first, then register the assignment and map the
local rank through it, as the GPU Worker and the NPU adapter do. Building
the config reads no weights, so a bad deployment still fails before the
experts are loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
test(ffn_faas): drop the --device-ids device-selection unit test
dbccf65 
Remove test_device_ids_place_the_rank_on_its_assigned_gpu and the helpers
only it used from tests/unit/ffn_faas/test_gpu_adapter.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
docs(ffn_faas): drop NPU/GPU parallel comments from standalone FFN code
30d9246 
Remove the "NPU parallel:" / "GPU parallel:" and "Reuse:" comment blocks
that compared each standalone FFN class and method with its counterpart on
the other platform, together with the dangling REFACTOR-CANDIDATE(R8) tag.
The cross-references named file paths that no longer match main.

Where a block carried a platform-local explanation that the docstrings did
not already state, move it into the docstring: the adapter's fixed GPU
constraints and handshake-adopted options, the backend lifecycle and
release order, the per-step synchronize that makes execute_step fence-free,
and the base-class versus platform split of both FFN environments.

Comments and docstrings only; no behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
Merge upstream/main into ffn-faas-qwen3-30b
8cdeedb 
The branch's base (codex/ffn-faas-framework) landed on main as the vllm-project#490
squash, so files were resolved against the vllm-project#490 commit this branch had
already merged (47c75d8) rather than the older main merge base. That
keeps the GPU FFN service changes, takes upstream's later NPU work
(including the ffn_model/deepseek_v4/npu regrouping, without resurrecting
the old platforms/npu/v4_* files), and leaves every file outside this
branch's own changes identical to upstream/main.

tests/e2e takes upstream's model-owned case layout (vllm-project#561) as-is; the
standalone FFN service E2E cases are ported to it in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@ronenkat @claude
test(e2e): port the standalone FFN service cases to model-owned cases
67eeb01 
vllm-project#561 moved E2E scenarios into model-owned Case/Deployment declarations,
which left the branch's legacy runner path for the standalone FFN
service unreachable. Port the four afd-deepep-m2n-faas-*-2a2f cases to
tests/e2e/models/qwen3_moe/cases.py.

Deployment.ffn_service declares that the FFN role runs the standalone
FFN service under torchrun instead of `vllm serve`. The deployment
builds its command from the same AFD config as the vLLM FFN role, waits
for every rank's /ready before the Attention API, and finish() ends a
passing case in the service's stop order (/stop on every rank, then
Attention) and requires a clean launcher exit and released resources
per rank. Cleanup sweeps the torchrun workers by run environment, as
they live outside the launcher's process group. Execution evidence is
required from Attention only; the service has no vLLM Worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ronenkat <16743404+ronenkat@users.noreply.github.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes NVIDIA NVIDIA GPU, CUDA, and related changes labels Nov 13, 2026
@ronenkat ronenkat added the ready Used to trigger ready CI in PRs. label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hsliuustc0106 hsliuustc0106 Awaiting requested review from hsliuustc0106 hsliuustc0106 is a code owner
@jiangkuaixue123 jiangkuaixue123 Awaiting requested review from jiangkuaixue123 jiangkuaixue123 is a code owner

Assignees

No one assigned

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation NVIDIA NVIDIA GPU, CUDA, and related changes ready Used to trigger ready CI in PRs.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[RFC]: GPU support for the standalone FFN runner

2 participants

@ronenkat @jiaran-king