checkpointing: --hpc flag to relax §4.7.1 cache-flush gap on HPC shared filesystems

[zhenghh04]

Problem

Rules.md §4.7.1 models a client-node failover: when the storage system needs an
external callout between the write and read phases (e.g. to clear a client-side
cache when checkpoint-per-client < 3× node memory), the submitter must run the
write and read phases as two invocations, and the validator requires the
read invocation to reach main.py within 30 seconds of the write invocation
ending.

On an HPC shared parallel filesystem this 30-second budget is not
achievable:

1. The write and read phases are separate batch-scheduler jobs; the node set
   cannot be guaranteed identical across the two allocations.
2. The read invocation's unavoidable per-invocation MPI spawn + DLIO
   re-initialization over a multi-TB checkpoint tree alone takes minutes to
   reach main.py — we measured ~226–268 s on ALCF systems, far over the
   30 s cap.

So a valid CLOSED checkpoint submission is currently impossible on these
systems, even though cold reads are already guaranteed by the storage layer /
benchmark on HPC: a remote parallel filesystem (Lustre/GPFS/DAOS), --o-direct,
and/or the benchmark's own posix_fadvise(POSIX_FADV_DONTNEED) on file close.
The 30 s wall-clock window is an enterprise-recovery model that does not map to
the HPC deployment.

Change

Add an opt-in --hpc flag to mlpstorage ... checkpointing run (and
configview) that declares this environment.

• argparse records it as args.hpc; Benchmark.write_metadata already
  serializes vars(self.args) into each run's metadata.json, so the flag
  reaches the validator with no extra plumbing.
• When set, the validator relaxes the two §4.7.1 checks —
  cache_flush_validation (the 30 s gap) and checkpoint_invocation_structure
  (the 1-or-2 invocation shape) — from errors to warnings. The exemption is
  therefore visible in the report, not silently applied, and is recorded in
  metadata.json for auditability.
• Rules.md §4.7.1, ManPage.md, and the CLI help document the exemption and
  make clear the submitter remains responsible for cold reads.

This is a deliberate, transparent relaxation for the HPC deployment class; it is
structured so it can later be tightened to require an explicit attested eviction
mechanism (fadvise / O_DIRECT) without reworking the plumbing, if the WG prefers.

Testing

mlpstorage_py/tests/test_checkpoint_hpc_exemption.py (new):

• regression guards — a two-invocation split with a >30 s gap still fails
  cache_flush_validation, and 3 invocations still fail
  checkpoint_invocation_structure, when --hpc is absent;
• with --hpc both pass and emit a warning instead;
• argparse registers --hpc on run/configview (not datasize) and defaults
  it False in the namespace everywhere.

All checkpointing + gap validator tests pass (88 passed locally).

Context

Sibling of the ALCF PALS launcher fixes #818 (CAP-02b results-dir probe) and
#819 (kvcache --mca), which unblocked multi-node checkpoint/kvcache runs on
ALCF HPE-Cray PALS. This PR removes the last blocker to a valid CLOSED
checkpoint submission from an HPC shared parallel filesystem.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[FileSystemGuy]

@zhenghh04 You can submit this as OPEN and we can massage it later into CLOSED. OPEN does not check the inter-phase-gap at all.

PR#834 is available for you. This PR and PR#834 will not be merged into main, just clone PR#834 into a local workspace and use that code for your run. We'll come back to how to support systems like yours after this submission round.