Skip to content

[NPU] Add E2E coverage for Async CAM Pull MegaMoE - #579

Merged
jiangkuaixue123 merged 2 commits into
vllm-project:mainfrom
lirx-pd:test/576-async-cam-pull-megamoe
Nov 14, 2026
Merged

jiangkuaixue123 merged 2 commits into
vllm-project:mainfrom
lirx-pd:test/576-async-cam-pull-megamoe

Conversation

@lirx-pd

@lirx-pd lirx-pd commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add a registered DeepSeek-V4 Flash A3 E2E case for the opt-in Async CAM Pull MegaMoE path. The existing Async CAM case defaults to Mega off and does not exercise the Pull/native Mega executor.

Issue

Scope

  • In scope: Register a 16-NPU Mega-on case; check concurrent responses and live Pull/Mega execution on every A/F rank; retain separate cleanup evidence; document the launch and artifacts.
  • Out of scope: A5 CAMAsyncMegamoeAFDConnector, the FFN FULL graph work tracked by [Feature]: Add E2E coverage for AsyncCam FFN Fullgraph #509, GPU coverage, performance benchmarking, and changes to the default Mega-off case.

Implementation Notes

  • Register afd-dsv4-flash-async-cam-pull-mega-dp2tp4-ep8 in the model-owned case list and pytest entry. Set ffn_mega_moe=true and dynamicQuant=1 for both Attention and FFN through the shared connector configuration.
  • Reuse the existing ten concurrent arithmetic requests and exact-answer checks. A test-only worker extension records completed Pull operations on Attention and completed non-idle native Mega transactions on FFN during the request window. Missing ranks, zero completions, or a Push/Layered fallback fail the check.
  • Configure the plugin-owned CANN vendor paths before device initialization. The observer adds device synchronization and is intended for functional coverage, not performance measurement.
  • Attempt process cleanup and retain orphan checks and cleanup errors. For this Mega case, cleanup errors are recorded separately and do not fail pytest after request and execution checks pass. Startup, request, answer, and execution-evidence failures remain failures.
  • The test-only observer wraps plugin-owned methods; it does not modify the vLLM or vLLM-Ascend source checkout. The observer is specific to this case and does not have an explicit version guard or idempotence guard.

Test Plan

  • Run the single-host harness regression, including negative evidence, wrong-answer and cleanup cases:

    python -m pytest -o addopts='' tests/e2e/test_runner.py -k 'not multi_pod' -q --tb=short
  • Build the plugin-owned Ascend operators and run the registered Mega-on case on A3 using the repository's pinned stack and the documented model and topology.

  • Run the existing Mega-off Async CAM case with the same model and topology as an answer-check comparison.

  • Run the applicable repository hooks; retain cleanup evidence separately.

Test Result

  • Single-host harness: 32 passed, 8 deselected (multi-pod cases). Ruff, mypy, git diff --check, and applicable repository hooks passed.
  • Real A3 run: DeepSeek-V4 Flash DSpark W4A8, Attention DP2/TP4 and FFN DP8/TP1/EP8. The native operator package was built from the tested source. Python 3.12.13, vLLM 0.32.0+empty, vLLM-Ascend 0.19.1rc2.dev3100, torch 2.10.0+cpu, torch-npu 2.10.0.post7, and CANN 9.2.0 were used.
  • Mega-on pytest: 1 passed; all 10/10 concurrent arithmetic answers were correct. All 8 Attention ranks recorded Pull completion, and all 8 FFN ranks recorded non-idle native Mega completions (340–374 per rank).
  • Mega-off comparison: pytest 1 passed, with the same 10/10 correct answers. This is a fixed-prompt functional check, not GSM8K accuracy or performance evidence.
  • Cleanup is recorded separately from pytest acceptance; no orphan processes were detected in the Mega-on run.
  • GPU and multi-pod tests were not run.
  • Local evidence and SHA256 manifests: results/issue576-lifecycle-20261113/, including REPORT.md, evidence/mega6.log, and evidence/pytest-mega6/.

Docs Impact

  • Files updated: tests/e2e/README.md and docs/design/module/e2e_testing.md.
  • If none, reason: Not applicable.

Essential PR Checklist
  • Purpose is clear and linked to public context when possible.
  • Scope is bounded.
  • Compatibility with vLLM v0.32.0 and, for NPU changes, the pinned vLLM-Ascend source is considered, as applicable.
  • No changes are made to the vLLM source checkout.
  • Plugin-owned classes or explicit dotted class paths are preferred over monkey patches.
  • Any compat shim or monkey patch is isolated, idempotent, version-guarded, documented, and tested.
  • Imports remain CPU-safe; CUDA-heavy work is delayed or GPU-gated.
  • Validation evidence is included, including skipped GPU tests when applicable.
  • Documentation impact is stated.

Signed-off-by: lirx-pd <616517220@qq.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation Ascend Ascend NPU platform and related changes labels Nov 13, 2026
Signed-off-by: lirx-pd <616517220@qq.com>
@jiangkuaixue123
jiangkuaixue123 merged commit 3463372 into vllm-project:main Nov 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend Ascend NPU platform and related changes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants