Add NCCL backend by msimberg · Pull Request #55 · ghex-org/oomph

msimberg · 2025-12-22T14:21:51Z

This adds a NCCL backend, with some strong constraints compared to the MPI, libfabric, and UCX backends:

Cancellation isn't supported
Tags aren't supported (they are ignored)
Send/recv submission requirements are stronger (communication should mostly be launched within groups)
Multithreading is not allowed with NCCL (because of ordering requirements and lack of tags)

If one sticks to these requirements one should be able to use any backend. If one needs any of the above features, NCCL can't be used.

Adds a few extra features to communicators:

start_group/end_group: These map to ncclGroupStart/ncclGroupEnd for NCCL, and no-ops for other backends.
is_stream_aware: The NCCL backend is the only one that returns true for this. If a backend is_stream_aware it will take into account the optional stream argument that can be passed to send/recv.

msimberg · 2026-01-08T12:55:54Z

This now seems to work in icon fortran. While I still have some open TODOs I'd be grateful for feedback on this already. The general implementation is pretty much what I want it to be, though I still have some profiling to do with NCCL to check if I'm missing some additional low hanging fruit.

Besides any comments you may have on the implementation itself (in particular I'm grateful if you have comments on me misunderstanding oomph requirements for backends) I guess we may need to discuss some sort of CI for the NCCL backend...

I can't request reviews so pinging @boeschf @biddisco @philip-paul-mueller.

philip-paul-mueller

I have some comments/suggestions, but I am not sure what they are worth; probably not much.

Mostly just copy MPI implementation to a new directory, not functional.

…nality to nccl

…f debugging

# Conflicts: # ext/hwmalloc

NCCL natively supports self-send/recv within groups (ncclGroupStart/End). The previous implementation unconditionally rejected self-send/recv, but this is only necessary outside of groups. Changes: - Only throw when self-send/recv is attempted outside an active group - Allow self-send/recv inside groups (NCCL handles it correctly) - Update test to verify both behaviors - Update README to document the new behavior This unblocks structured tests in GHEX which use self-send/recv with periodic boundaries and multiple subdomains per rank.

Copilot

Pull request overview

Copilot reviewed 45 out of 45 changed files in this pull request and generated 5 comments.

Check that start_group isn't called when a group is already active, or end_group called when a group isn't active.

msimberg · 2026-07-01T13:23:01Z

I'm going to go ahead and merge this with the knowledge that the backend may not yet be perfect. I have for example seen a few hangs with the NCCL backend now in CI which have been transient, and I don't know if they're a configuration issue or an actual bug in the backend. However, in the interest of getting this into real usage sooner rather than later, I'll merge this as an "experimental" backend and revisit remaining issues later depending on how serious they are.

Uses ghex-org/oomph#55. Updates `communication_object{,_ipr}` to use the `start_group`/`end_group` functionality from oomph/NCCL, as well as taking `is_stream_aware` into account. Also does a minor refactoring of packer and communication object helper functions so that the different stages are a bit easier to follow: - optional: sync before packing - pack - send - recv - unpack - optional: sync after unpacking