Skip to content

use MPI file function to output KS_eigenvector - #3

Open
userchba wants to merge 1 commit into
AroundPeking:master_ghjfrom
userchba:master_hbchen
Open

userchba wants to merge 1 commit into
AroundPeking:master_ghjfrom
userchba:master_hbchen

Conversation

@userchba

Copy link
Copy Markdown

Summary

Replace the rank-0 gather-and-write implementation of the LibRPA v1 KS eigenvector output (KS_eigenvector_0.dat, active when out_librpa_reader_version == 1 under __MPI) with a collective MPI-IO writer.

Motivation

The old implementation had several scalability and robustness problems:

  • It issued one MPI_Allreduce (over MPI_COMM_WORLD) per (k-point, spin, band), i.e. nks × nspin × nbands collectives of size nlocal each.
  • It materialized the full nbands × nlocal wavefunction matrix on every rank, and rank 0 held the entire file payload in memory (nks × nspin × nbands × nlocal × 16 B) before writing it serially — prohibitive for production RPA/GW system sizes.
  • It wrote directly to the final file name, so a mid-write failure could leave a truncated/corrupt KS_eigenvector_0.dat.
  • MPI return codes were unchecked; a rank-0 file-open failure threw while other ranks proceeded into the next collective (hang/abort hazard), and the Allreduce was hardcoded to MPI_COMM_WORLD instead of the wavefunction communicator.

What this PR does

RPA_LRI::out_eigen_vector (v1 branch, MPI build) now uses a collective MPI-IO writer:

  • Each rank's block-cyclic ownership is described by a run-length-encoded MPI_Type_create_hindexed file datatype; payload is written with MPI_File_write_at_all in chunks bounded to 4M complex elements (64 MB pack buffer per rank, independent of system size).
  • Header + k-point directory are written by rank 0 within the same collective calls; no payload Allreduces remain.
  • Atomic publication: the file is written to KS_eigenvector_0.dat.tmp, synced, closed, then renamed by rank 0; the temp file is removed on any error.
  • Collective error propagation (Allreduce/Bcast-based checks) so all ranks fail together with a clear message; MPI error codes are converted to readable exceptions.
  • Collective validation before writing: psi vs Parallel_Orbitals dimensions, spin/SOC consistency (even nlocal, even/odd spinor component split), and an exact global ownership invariant (summed per-rank element counts must equal nbands × spatial_basis).
  • All offset arithmetic is overflow-guarded (MPI_Aint / MPI_Offset / int32 range checks).

Compatibility

  • The binary v1 format, the file name, and the payload layout ([component][band][orbital], SOC split into even/odd spinor components) are unchanged — downstream LibRPA readers are unaffected.
  • The serial (non-MPI) path and the legacy text output (out_librpa_reader_version != 1) are untouched.

Verification

  • A standalone MPI test harness including the new writer verbatim from RPA_LRI.hpp, compared against a re-implementation of the old writer: outputs are byte-for-byte identical over ~150 runs spanning 1–8 MPI ranks, nspin = 1/2/4 (incl. SOC), ScaLAPACK block sizes 1–33, uneven grids with ranks owning zero bands, and payloads crossing the 4M-element chunk boundary (multi-chunk collective writes).
  • Every produced file was additionally validated against an independent analytic ground truth (header fields, directory offsets, and every payload element).
    ``

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant