Nccl translator#50
Open
Amos-Rai-KEYS wants to merge 12 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add NCCL topology translator
Summary
This branch adds a new NCCL translator that parses NCCL-generated topology XML into the infragraph device/infrastructure model, alongside annotations capturing per-GPU and per-NIC metadata. It also generalizes the translator framework to support multiple tools and adds a
--device_nameoption.What's new
NCCL translator (
src/infragraph/translators/nccl_translator.py— 604 lines)NcclParser— parses NCCL XML topology into aDevice/Infrastructuregraph: CPUs and CPU fabric links, PCI bridge hierarchy, GPUs/XPUs, NICs, and NVLink/NVSwitch peer connections.NcclHelper.generate_nccl_topology()— generates a fresh topology XML at runtime by loadinglibnccl.so/libcudart.soand callingncclCommInitAll(used when no--inputfile is supplied).get_annotations()— produces anInfraGraphServiceannotated with per-GPU metadata (rank, busid, device model, CPU affinity) and per-NIC metadata.run_nccl_parser()— writes both the translated device file and anannotated_infragraph.json(infrastructure + annotations).Translator framework changes
translator_handler.py— addsncclto supported translators and threads adevice_nameargument through; requires--device_namefor nccl.__main__.py— new--device_nameCLI option; default output renameddev.yaml→device.yaml.Tests & notebooks
test_nccl_translator.py— validates component counts, CPU/NVSwitch parsing, edges/links.dgx_a100_nccl_topo.xml— DGX A100 mock fixture.nccl_translate.py— example notebook.