We need to implement multi-GPU support for systems with 2 or more GPUs connected via x16 PCI-E lanes.
- we can leverage
device_map="auto" to automatically shard models for HuggingFace models
- Utilize
torch.distributed or torch.nn.parallel.DistributedDataParallel for efficient parallelism
Extend torch_node to track multi-device configurations, and ensure this device metadata is broadcast across the P2P network for smarter job routing and coordination.
We need to implement multi-GPU support for systems with 2 or more GPUs connected via x16 PCI-E lanes.
device_map="auto"to automatically shard models for HuggingFace modelstorch.distributedortorch.nn.parallel.DistributedDataParallelfor efficient parallelismExtend torch_node to track multi-device configurations, and ensure this device metadata is broadcast across the P2P network for smarter job routing and coordination.