feat(simccl): SimCCL v2.30 mock with PAT, protocol-aware, standalone binary#4
Open
tianhao909 wants to merge 4 commits into
Open
feat(simccl): SimCCL v2.30 mock with PAT, protocol-aware, standalone binary#4tianhao909 wants to merge 4 commits into
tianhao909 wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR replaces placeholder SimCCL content with a fuller NCCL v2.30-oriented mock translation layer and adds a standalone, CPU-only CLI that can emit FlowModel CSVs without ns-3, along with extensive scripts and documentation to validate/operate the new capabilities.
Changes:
- Added
simccl-standalone(CMake + build script + CLI) and a minimal workload parser to generatencclFlowModel_detailed_flows.csvoutside the full SimAI/ns-3 pipeline. - Introduced PAT-focused tooling/tests (PAT step simulator, cartesian test framework, and standalone test runner scripts).
- Expanded documentation (EN/CN) describing architecture, environment variables, integration, and validation methodology/results.
Reviewed changes
Copilot reviewed 63 out of 68 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/pat_simulator.cc | PAT host-side step simulator utility |
| standalone/workload_parser.h | Workload parser API for standalone |
| standalone/workload_parser.cc | Workload text parsing implementation |
| standalone/main.cc | Standalone CLI entrypoint / orchestration |
| standalone/CMakeLists.txt | Standalone build definition (CMake) |
| standalone/build.sh | Standalone build helper script |
| standalone/.gitignore | Ignore standalone build artifacts |
| scripts/setup_53_symlinks.sh | Helper for multinode path alignment |
| scripts/run_standalone_all.sh | Standalone positive/negative test runner |
| scripts/run_multinode_2x8.sh | 2-node 8-GPU/node nccl-tests runner |
| scripts/run_multinode_2x1_pat.sh | 2-node 1-GPU/node PAT-condition runner |
| scripts/nsys_multinode.sh | nsys profiling wrapper for multinode |
| scripts/hostfile_52_53.txt | MPI hostfile template (2x8) |
| scripts/hostfile_1gpu.txt | MPI hostfile template (2x1) |
| scripts/cartesian_test/run_sim_cartesian.sh | Cartesian simulation executor |
| scripts/cartesian_test/results/real/summary_real.csv | Real-machine measurement summary data |
| scripts/cartesian_test/results/real/README.md | Real-machine test report narrative |
| scripts/cartesian_test/results/cartesian_test_report.md | Generated cartesian test report |
| scripts/cartesian_test/plot_results.py | Plot/report generator for cartesian results |
| scripts/cartesian_test/gen_cases.py | Test case CSV generator (cartesian) |
| scripts/cartesian_test/configs/sim_cases_observe.csv | Generated observe-mode case matrix |
| scripts/cartesian_test/configs/real_cases.csv | Generated real-machine case matrix |
| README.md | Project overview + standalone quickstart |
| mock/v2.30/MockNcclQps.h | v2.30 QP simulation header |
| mock/v2.30/MockNcclLog.h | v2.30 logging interface/impl header |
| mock/v2.30/MockNcclLog.cc | v2.30 logging static definitions |
| mock/v2.30/MockNcclGroup.h | v2.30 group/flow-model interface header |
| mock/v2.30/MockNcclChannel.h | v2.30 channel/comm interface header |
| mock/v2.30/MockNcclChannel.cc | v2.30 channel/comm implementation |
| mock/v2.30/MockNccl.h | v2.30 constants/tables (algo/proto/lat) |
| mock/v2.20/MockNcclQps.h | v2.20 QP simulation header |
| mock/v2.20/MockNcclLog.h | v2.20 logging interface/impl header |
| mock/v2.20/MockNcclLog.cc | v2.20 logging static definitions |
| mock/v2.20/MockNcclGroup.h | v2.20 group/flow-model interface header |
| mock/v2.20/MockNcclChannel.h | v2.20 channel/comm interface header |
| mock/v2.20/MockNcclChannel.cc | v2.20 channel/comm implementation |
| mock/v2.20/MockNccl.h | v2.20 constants/tables (algo/proto/lat) |
| mock/README.md | Mock version layout + build switch docs |
| docs/README.md | Docs index (EN/CN navigation) |
| docs/integration/integration-with-simai.md | Integration env/flow pipeline guide (EN) |
| docs/getting_started/quickstart.md | Standalone quickstart (EN) |
| docs/getting_started/installation.md | Installation/build guide (EN) |
| docs/development/adding-new-collective.md | Dev guide for adding collectives (EN) |
| docs/design/nccl-comparison.md | NCCL vs SimCCL comparison (EN) |
| docs/design/callstack-analysis.md | Execution callstack analysis (EN) |
| docs/design/architecture.md | Architecture overview (EN) |
| docs/configuration/env-variables.md | Env var reference (EN) |
| docs/configuration/build-options.md | Standalone build options (EN) |
| docs/CN/integration/integration-with-simai.md | Integration guide (CN) |
| docs/CN/getting_started/quickstart.md | Standalone quickstart (CN) |
| docs/CN/getting_started/installation.md | Installation/build guide (CN) |
| docs/CN/development/adding-new-collective.md | Dev guide for adding collectives (CN) |
| docs/CN/design/nccl-comparison.md | NCCL vs SimCCL comparison (CN) |
| docs/CN/design/callstack-analysis.md | Execution callstack analysis (CN) |
| docs/CN/design/architecture.md | Architecture overview (CN) |
| docs/CN/configuration/env-variables.md | Env var reference (CN) |
| docs/CN/configuration/build-options.md | Standalone build options (CN) |
| .gitignore | Repo-level ignore rules update |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+48
to
+52
| // Parse: name comp_time comp_size fwd_type fwd_size fwd_group wg_type wg_size wg_group ig_type ig_size ig_group | ||
| iss >> name >> comp_time >> comp_size | ||
| >> fwd_type_str >> ld.fwd_pass_comm_size >> ld.fwd_pass_group_type | ||
| >> wg_type_str >> ld.weight_grad_comm_size >> ld.weight_grad_group_type | ||
| >> ig_type_str >> ld.input_grad_comm_size >> ld.input_grad_group_type; |
Comment on lines
+15
to
+16
| return AstraSim::ComType::None; | ||
| } |
Comment on lines
+88
to
+92
| if (workload_file.empty() && op_str.empty()) { | ||
| std::cerr << "[ERROR] Must specify either -w <workload> or --op <op> --size <bytes>\n"; | ||
| print_usage(argv[0]); | ||
| return 1; | ||
| } |
Comment on lines
+59
to
+60
| return GPUType::H20; // default | ||
| } |
Comment on lines
+44
to
+45
| logfile.open(LogName, std::ios::app); | ||
| } |
Comment on lines
+41
to
+53
| MockNccl::TreeChannels MockNcclComm::get_treechannels(){ | ||
| TreeChannels nvlschannel ; | ||
| nvlschannel[0][0]=ncclTree(-1,0,8,{}); | ||
| nvlschannel[0][1]=ncclTree(-1,1,8,{}); | ||
| nvlschannel[0][2]=ncclTree(-1,2,8,{}); | ||
| nvlschannel[0][3]=ncclTree(-1,3,8,{}); | ||
| nvlschannel[0][4]=ncclTree(-1,4,8,{}); | ||
| nvlschannel[0][5]=ncclTree(-1,5,8,{}); | ||
| nvlschannel[0][6]=ncclTree(-1,6,8,{}); | ||
| nvlschannel[0][7]=ncclTree(-1,7,8,{}); | ||
| nvlschannel[0][8]=ncclTree(-1,8,-1,{0,1,2,3,4,5,6,7}); | ||
| return nvlschannel; | ||
| } |
Comment on lines
+66
to
+68
| ## License | ||
|
|
||
| MIT — See [LICENSE](../LICENSE) |
Comment on lines
+16
to
+20
| #include <cstdio> | ||
| #include <cstdlib> | ||
| #include <cstdint> | ||
| #include <algorithm> | ||
|
|
tianhao909
force-pushed
the
simai1_7_pr_260707_1249
branch
from
July 7, 2026 10:48
2170580 to
d755f77
Compare
…binary [EN] - Add mock/v2.20 (frozen baseline) and mock/v2.30 (PAT + protocol-aware + NVLS) - Add standalone binary (no ns3 dependency, CPU-only, ~300KB) - Add comprehensive documentation (EN/CN, getting_started/config/design/integration) - Add test scripts (run_standalone_all.sh, cartesian_test, multinode scripts) - Add tools (pat_simulator, draw_channels) - Supported collectives: AllReduce, AllGather, ReduceScatter, AlltoAll, Broadcast - Algorithms: Ring, PAT (multi-node 1GPU/node), NVLS (single-node >=8GPU) - Protocol-aware: LL/LL128/Simple auto-selection by message size [ZH] - 添加 mock/v2.20（冻结基线）和 mock/v2.30（PAT + 协议感知 + NVLS） - 添加独立二进制（无 ns3 依赖，纯 CPU，约 300KB） - 添加完整文档（中英双语，入门/配置/设计/集成） - 添加测试脚本（run_standalone_all.sh、笛卡尔积测试、多节点脚本） - 添加工具（pat_simulator、draw_channels） - 支持集合操作：AllReduce、AllGather、ReduceScatter、AlltoAll、Broadcast - 算法：Ring、PAT（多节点1GPU/节点）、NVLS（单节点>=8GPU） - 协议感知：基于消息大小自动选择 LL/LL128/Simple
tianhao909
force-pushed
the
simai1_7_pr_260707_1249
branch
from
July 7, 2026 12:31
d755f77 to
2c33ec5
Compare
MXtremist
reviewed
Jul 9, 2026
| @@ -0,0 +1,313 @@ | |||
| /** | |||
…tor, add root wrappers [EN] - Untrack scripts/cartesian_test/ and tools/pat_simulator.cc (kept local via .gitignore) - Add root build.sh/run.sh wrappers delegating to standalone/ (set -euo pipefail, arg passthrough, missing-binary guard) - README: add "One-Command Standalone" section pointing to root scripts - Docs: remove cartesian_test step from adding-new-collective (EN/CN) and renumber steps - Docs: note standalone build requires SimAI checkout layout (astra-sim-alibabacloud sibling, Common.hh) - .gitignore: consolidate; ignore cartesian_test/, pat_simulator(.cc), standalone/build_*/ [ZH] - 取消跟踪 scripts/cartesian_test/ 与 tools/pat_simulator.cc（通过 .gitignore 本地保留） - 新增根级 build.sh/run.sh 薄封装，转调 standalone/（set -euo pipefail、参数透传、缺二进制报错） - README：新增“一键独立运行”小节，指向根脚本 - 文档：从 adding-new-collective（中英）移除 cartesian_test 步骤并重新编号 - 文档：注明 standalone 构建需在 SimAI 布局下（astra-sim-alibabacloud 同级、依赖 Common.hh） - .gitignore：整理；忽略 cartesian_test/、pat_simulator(.cc)、standalone/build_*/
Author
Review update (round 2)Changes
Tests
Review 更新（第 2 轮）改动
测试
|
…back [EN] - main.cc: parse_gpu_type() errors on unknown GPU type instead of silently defaulting to H20 - main.cc: reject --size=0 in single-op mode; wrap workload parsing in try/catch for clean errors - workload_parser.cc: throw on unknown comm type and on malformed lines (with line number) - mock/v2.30/MockNcclLog.h: default log name + is_open() check to avoid silently-disabled logging - README.md: fix broken LICENSE link (../LICENSE -> ./LICENSE) [ZH] - main.cc：parse_gpu_type() 对未知 GPU 类型报错退出，不再静默默认 H20 - main.cc：单 op 模式拒绝 --size=0；workload 解析加 try/catch 输出清晰错误 - workload_parser.cc：未知通信类型与坏行（带行号）抛错 - mock/v2.30/MockNcclLog.h：空 LogName 时设默认日志名并校验 is_open，避免静默关闭日志 - README.md：修复 LICENSE 链接（../LICENSE -> ./LICENSE）
Author
Review 修复 + 测试结果 / Review fixes + test resultsCommit: 已修复的 review 点 / Fixed review items
对 reviewer 评论的回复 / Replies
静态测试 / Static testscompiler: g++ 13.3.0, cmake 3.31.6. 0 errors (仅历史 -Wsign-compare/-Wunused warnings)。 动态测试 - 正向 / Dynamic - positive动态测试 - 负向（证明修复生效）/ Dynamic - negative (proving fixes)
|
Author
三仓统一集成验证通过 / Unified 3-repo integration verified本 PR（SimCCL v2.30 mock）与 aliyun/ns-3-alibabacloud#22、aliyun/SimAI#294 一同构成 SimAI 1.7，需三者联合合并。 在整合开发仓上的完整验证（详见 aliyun/SimAI#294 的“三仓统一端到端集成测试”评论）：
|
…ame test script
[EN]
- Move standalone sources into src/ (mock/, main.cc, CMakeLists.txt, build.sh,
workload_parser.*); mock is now src/mock/<version>
- Remove root wrappers build.sh / run.sh and standalone/.gitignore; merge ignore
rules into top-level .gitignore (src/build/, src/build_*/)
- Rename scripts/run_standalone_all.sh -> scripts/run_all.sh; fix internal binary
path to src/build/
- src/CMakeLists.txt: MOCK_DIR now ${CMAKE_SOURCE_DIR}/mock/<version>
- README/docs: update paths standalone -> src; build via
`cd SimCCL/src && bash build.sh`, full test via `scripts/run_all.sh`
- Note: the standalone build still requires SimCCL to sit inside a SimAI tree
(references ../../astra-sim-alibabacloud); validated inside the SimAI monorepo
[ZH]
- 将 standalone 源码整体迁入 src/（mock/、main.cc、CMakeLists.txt、build.sh、
workload_parser.*）；mock 现为 src/mock/<版本>
- 删除根级 wrapper build.sh / run.sh 与 standalone/.gitignore；忽略规则合并进
顶层 .gitignore（src/build/、src/build_*/）
- scripts/run_standalone_all.sh 改名为 scripts/run_all.sh；修正内部二进制路径为
src/build/
- src/CMakeLists.txt：MOCK_DIR 改为 ${CMAKE_SOURCE_DIR}/mock/<版本>
- README/文档：路径 standalone -> src；构建用 `cd SimCCL/src && bash build.sh`，
全量测试用 `scripts/run_all.sh`
- 说明：standalone 构建仍需 SimCCL 位于 SimAI 树内（引用
../../astra-sim-alibabacloud）；已在 SimAI 单体仓内验证通过
Author
Update addressing review (2026-07)Thanks for the review — restructured as requested. Structure (re: threads on removing
Note on keeping a single Static/dynamic test (Ubuntu 24.04, GCC 13.3.0), inside a SimAI checkout (the standalone build references
|
Author
Full-suite test result (
|
MXtremist
approved these changes
Jul 14, 2026
Author
Test Report / 测试报告 (SimCCL v2.30 standalone)Environment: Ubuntu 24.04, g++ 13.3.0, x86_64. Built inside a SimAI tree ( 1. Static build / 静态构建
2. Dynamic run — tutorials (
|
| Case | Command | Result |
|---|---|---|
| Mode 1 single op | ./simccl-standalone --op AllReduce --size 4194304 --nRanks 8 --nNodes 1 --gpus_per_node 8 --gpu_type H20 |
PASS (exit 0, 896 flow rows in CSV) |
| Mode 2 workload file | ./simccl-standalone -w example/microAllReduce.txt --nRanks 8 --nNodes 1 --gpus_per_node 8 --gpu_type H20 |
PASS (exit 0, parsed 2 layers) |
| NVLS enabled | AS_NVLS_ENABLE=1 ./simccl-standalone --op AllReduce --size 4194304 --nRanks 8 --nNodes 1 --gpus_per_node 8 |
PASS (exit 0) |
| Invalid topology | --nRanks 8 --nNodes 1 --gpus_per_node 4 |
PASS — correctly rejected (exit 1, clear error) |
3. Full test suite / 完整测试套件
cd SimCCL && bash scripts/run_all.sh (as documented in quickstart "Testing"):
Results: PASS=56 FAIL=0 TOTAL=56
Breakdown: 52 valid topology combinations (1n8g, 2n8g, 2n1g-PAT, 1n4g, 2n2g) + 4 invalid inputs (expected failures). Summary written to results/standalone_test_summary.csv.
明细：52 个有效拓扑组合 + 4 个非法输入（预期失败），结果写入 results/standalone_test_summary.csv。
All installation.md / quickstart.md steps complete successfully.
installation.md 与 quickstart.md 全部步骤执行通过。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SimCCL v2.30 mock translation layer (Ring / PAT / NVLS, protocol-aware), with the standalone binary. Restructured per review.
Repository layout (updated per review)
Supported Collectives
Quick Verification (inside a SimAI checkout)
Related PRs (land together)
src/mock)SimCCL/mockPart of SimAI 1.7 Release