[Feature] Support on Multi-Node Nvlink

[zyksir]

I found that NCCL has support blackwell GPU with multi-node Nvlink since nccl 2.25, is this feature supported on mscclpp? this is a related blog: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html

[chhwang]

You can already do that with the current MSCCL++. We are working on adding an example to the repo though.

[zyksir]

do you have any quick hint about how to use it? Should I simply use CudaIpc as Transport? Looking forward to see your demo

[chhwang]

@zyksir Yes, you will just need to create a cross-node connection with CudaIpc (or Nvls) transport.

[zyksir]

in connection.cc, I just found the following code. I think this will cause a problem in cross-node connection with CudaIpc @chhwang

// sanity check: make sure the IPC connection is being made within a node
  if (getImpl(remoteEndpoint)->hostHash_ != getImpl(localEndpoint)->hostHash_) {
    std::stringstream ss;
    ss << "Cuda IPC connection can only be made within a node: " << std::hex << getImpl(remoteEndpoint)->hostHash_
       << " != " << std::hex << getImpl(localEndpoint)->hostHash_;
    throw mscclpp::Error(ss.str(), ErrorCode::InvalidUsage);
  }

[chhwang]

@zyksir You are right. Let us create a patch soon.