CI: exercise the flexible-rank RMA path on two MPI implementations

[DLWoodruff]

Closes the last release gate for flexible (unequal) rank assignments.

The flexible-rank path is the only place mpi-sppy uses passive-target MPI RMA (Lock/Get/Put), the least portable corner of the spec. doc/designs/flexible_rank_assignments.md therefore gates recommending the feature for production on exercising it against "at least two MPI implementations (e.g. OpenMPI and MPICH) and more than one mpi4py / MPI version".

Until now the one job that runs those tests — test-cylinders — did a bare conda install mpi4py, which resolves to MPICH. So only MPICH was ever covered, and nothing in the logs said which implementation had been used.

The matrix

test-cylinders becomes a three-cell include matrix. Resolved versions, from dry-run solves:

cell	MPI	mpi4py
mpich	MPICH 5.0.1	4.1.2
openmpi	OpenMPI 5.0.10	4.1.2
mpich, mpi4py 3.1	MPICH 4.3.2	3.1.6

Two implementations, two MPICH versions, two mpi4py versions. fail-fast: false, because the point is to learn which builds the RMA path works on — every cell has to report rather than being cancelled by a sibling.

A new step prints mpiexec --version and MPI.Get_library_version(), so the job log is the evidence the gate asks for instead of an assumption about what conda picked.

conda-forge, exclusively

The job now sets channels: conda-forge with conda-remove-defaults. conda-forge is the channel that maintains per-implementation MPI variants: asking for openmpi there also pulls the openmpi-flavored mpi4py through the mpi metapackage, so each cell exercises the MPI it names.

On the defaults channel the mpi4py build is not variant-selected, so an "openmpi" cell could install OpenMPI beside an mpi4py still linked against MPICH — a cell that looks like coverage and is not.

A bug this turned up

-oversubscribe is OpenMPI-only (MPICH rejects it) and the np=6 runs ask for more ranks than a runner has cores, so it has to be applied conditionally. The existing detection in run_coverage.bash greps the mpiexec --version banner for Open MPI/OpenRTE.

That breaks on OpenMPI 5, which launches through PRRTE and no longer reliably prints either string. The openmpi cell would have silently dropped the flag and failed at np=6 with "not enough slots available".

Both the workflow and run_coverage.bash now ask mpi4py which library it is linked against (MPI.Get_library_version()), which is stable across versions. Verified locally on OpenMPI 4.1.6: detection fires, and single-dash -oversubscribe works at np=6.

Notes for review

• The check names change: the matrix reports tests on some cylinders (mpich) / (openmpi) / (mpich, mpi4py 3.1). Branch protection requires the old, un-suffixed tests on some cylinders, which a matrix never reports, so a gate job carries that name and passes only when every cell passed. Listing the three cell names in branch protection instead would block every PR branched before this one, since those branches cannot report contexts their workflow does not have.
• Each cell needs a distinct coverage artifact name (upload-artifact@v4 errors on duplicates); the aggregator globs coverage-*, so they are still collected.
• The design doc's release-gate paragraphs are flipped, but only after the matrix was observed green (see below).

Follow-ups folded in after the first run

The first run failed in the openmpi cell — not on RMA. It died in the pre-existing equal-rank step (test_with_cylinders.py at np=2), which then skipped the flexible-rank step, leaving the question this job exists to ask unanswered.

test_xhatshuffle_coverage compares the value its helper extension records on iteration 0 against iteration 1, so the hub has to run both. It asks for max_iterations=2 but never overrides cfg.rel_gap, which defaults to 0.05, and the hub stops as soon as the inter-cylinder gap drops below it. The log shows Terminating based on inter-cylinder relative gap 4.595% immediately before the IndexError. That is a race between spoke and hub, so it fires on some MPI/host combinations and not others. rel_gap is now pinned to 0 in that test, which is what the assertion already assumed; guarding on len(v) >= 2 would have hidden the check instead of fixing it.

The flexible-rank step also gets if: !cancelled(), so a pre-existing equal-rank failure can no longer conceal the flexible-rank result. Both still fail the job.

Result

All three cells green, and the OpenMPI cell's log carries coherence diagnostic lines from PHHub, LagrangianOuterBound, XhatShuffleInnerBound and CGHub — those only print when multi-source reads actually happen, so the unequal-rank path demonstrably ran there rather than being skipped.

That was the last item on the "finish before recommending it" list, so the design doc now records that no release gates remain, and states what the matrix covers instead of what someone still has to do.

With the feature no longer gated, flexible ranks gets its first user-facing page, doc/src/flexible_ranks.rst (Advanced Topics). Until now the feature was documented only in doc/designs/, which is not part of the Sphinx build. The page covers the eight --<spoke>-rank-ratio options, the largest-remainder apportionment, the fact that all-1.0 ratios take the same code path as before (which is what makes reverting a complete fallback), and the limitations — reduced_costs has no ratio option, and any other unsupported field fails at startup rather than mis-assembling. The worked example was run end to end first: 14 ranks at 1.0/0.5/0.25 does print per-cylinder rank counts [8, 4, 2].

The read-outcome diagnostic from #816 also gets an entry in doc/src/secretmenu.rst: the always-on finalization summary, what each of the five buckets means, how to tell a coherence problem from a slow upstream sender, and the coherence_diagnostics_period knob. Three review caveats are documented rather than fixed, since they affect how the numbers read and not whether they are collected — counters aggregate per Field with no peer attribution, the periodic line's buckets sum to total - 1, and equal-rank runs print nothing.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.42%. Comparing base (21cc24a) to head (6b81cff).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #827   +/-   ##
=======================================
  Coverage   78.42%   78.42%           
=======================================
  Files         185      185           
  Lines       24808    24808           
=======================================
  Hits        19455    19455           
  Misses       5353     5353           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[DLWoodruff]

The mpich, mpi4py 3.1 cell went red after the merge from main, but not on anything in this PR — it never reached a test. Both MPI steps aborted in MPI_Init:

UCX ERROR mana_0: iface ... failed to create UD QP ... Operation not supported
UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
Abort(608299535) on node 0: Fatal error in internal_Init_thread

That runner exposes an Azure MANA RDMA verbs device. conda-forge builds MPICH --with-device=ch4:ucx, UCX probes mana_0 at init, cannot create a UD queue pair on it, and kills every rank.

Which runner a job lands on is luck, so this is not specific to that cell: the mpich (5.0.1) cell has the same ch4:ucx build and passed only because its runner had no MANA device, and all three cells were green ten days ago.

020f5f6 sets UCX_TLS: "self,sm,tcp" in the workflow-level env: — workflow-level rather than in the matrix, since the other mpiexec jobs are one bad runner away from the same abort. Every rank is on one node here, so passive-target RMA goes over shared memory either way and no coverage is lost.

Checked locally against OpenMPI 5.0.8 (also UCX-linked) at the CI's np=6: test_spwindow_multisource.py and test_flexible_rank_cylinders.py both pass with the variable set.