Checkpoint/resume: design + phase 1a (serial hub, dill-reload)

[DLWoodruff]

Checkpoint / resume — design + phase 1a

This PR carries both the design document
(doc/designs/checkpointing_design.md) and its phase 1a implementation.
Phase 1 is implemented in the design's own branch so that corrections the
implementation turns up ride along with the code that motivated them — and it
turned up a great many.

What it does

A serial PH hub writes a resumable checkpoint at the end of every completed
iteration, and a later run picks up from it. Flags: --checkpoint-dir (enables
it), --resume-from, --checkpoint-backend, and
--checkpoint-every-iterations K.

That last one is the cost control. Serializing every scenario every iteration
is negligible against a MIP whose solves take minutes, but it was measured at
262% overhead on 50-scenario farmer, where serialization dominates. K writes
at every K-th completed iteration instead — checkpoints land on iteration
numbers that are multiples of K, counted from the start of the study, so a
resume continues the cadence rather than restarting it. A stop that is not a
checkpoint point loses the iterations since the last one, up to K-1 of them,
and that is the whole trade. Writes still happen only at iteration boundaries,
so the invariant below is untouched; K only changes which boundaries are
checkpoint points. The final iteration of an exhausted --max-iterations is
written whatever K is, since raising the limit and resuming is an ordinary way
to extend a study and that iterate is coherent and already in memory.

There is deliberately no flag to disable the checkpoint at termination.
--checkpoint-at-termination existed for the trigger this design replaced,
and was removed once writes moved to the iteration boundary -- it had been
left registered and documented while nothing read it, which is the same
silent-no-op failure this PR removes everywhere else.

A checkpoint always describes a completed PH iteration. That is the whole
invariant, and everything else follows from it.

It is worth saying why, because the obvious alternative — snapshot whatever
state exists when the run ends — does not work and cannot be patched into
working. iterk_loop computes xbar, updates the dual weights, runs the
miditer extension hook, and may break (user converger, convergence
threshold, --time-limit) before it solves. A run ending that way leaves the
models describing half an iteration: W at iteration k, nonants still at k-1.
--time-limit — the planned-stop recipe this feature exists for — exits that
way every time. Reconstructing a coherent iterate means undoing everything the
first half of the iteration did, and miditer lets any extension change rho,
fix variables, relax domains or add cuts, so that list is open by construction.
An earlier revision of this PR tried it and had to unwind W, then rho, then
fixedness, with domains and cuts next.

Writing at enditer — after the solve — sidesteps all of it, with no knowledge
of any extension. The cost is a model serialization per iteration rather than
one per run, paid only when --checkpoint-dir is given: without it the
extension is never constructed and none of its hooks exist. Retention is a
single generation, so disk does not grow with the iteration count, and each
write is bracketed by global_toc so the per-iteration cost is visible rather
than guessed at.

Consequence, deliberately: a run that ends before completing iteration 1
publishes nothing. No iteration completed, so there is no iterate to resume.

Resuming

The resume branch lives in Iter0, replacing the iteration-0 solve, with the
model swap ahead of _create_solvers so a persistent solver's set_instance
is paid once. It disarms the deferred objective attach while restoring the
opt-object state that attach would have set, refreshes saved_objectives,
restores the bound, the incumbent objective and the initially-fixed-nonant
baseline, and skips the rho_setter and the smoothing rescale that the
reloaded model already reflects. _PHIter is the global iteration, so
--max-iterations bounds the run as a whole.

What is refused, rather than half-supported

At setup, before any solve: non-PH hubs, multi-rank hubs, an unimplemented
--checkpoint-backend, an unwritable --checkpoint-dir, scenario names that
would collide once made safe for a file name (they would otherwise overwrite
each other's models and restore one scenario for both), and — the one that
used to lose a day silently — a configuration where --checkpoint-dir was
given but the extension is not actually attached to the hub.

The hub-type refusal applies to --resume-from as well as --checkpoint-dir.
Only the write side had it at first, so a resume-only run never constructed the
extension that carried the check, and --APH --resume-from would splice a PH
checkpoint into a hub that had already annotated its own models and iterates a
range ignoring the resume offset — with no error at startup.

A mid-run write failure is the deliberate exception to all of this: a full disk
or an NFS hiccup warns and the run continues on its still-published previous
generation, retrying at the next checkpoint point. Refusing at startup protects
a run that has done no work; aborting at iteration 400 destroys the very
progress the checkpoint exists to preserve.

On resume: a different rank count or scenario distribution, or a configuration
that differs anywhere outside a named list of settings a resume may change
(budget, solver choice and mipgaps, display/tracking/output, which cylinders
run). Checking by default is what stops a farmer checkpoint being resumed with
--farmer-with-integers and quietly answering the linear program.

The gate

mpisppy/tests/test_checkpoint.py (64 tests) includes a 20-cell acceptance
matrix: every way a run can end — iteration limit, convergence, --time-limit
inside iteration 1, --time-limit later — crossed with plain PH, smoothing,
linearized prox, a rho updater, and an extension that changes rho and fixes
variables in miditer. Every cell asserts the checkpoint round-trips exactly
and names the last completed iteration; cells without a stateful extension
additionally assert that a resumed run is indistinguishable from one that was
never interrupted. It caught a real bug within minutes of existing.

Known limitations, stated rather than implied

• Serial PH hub only. Non-PH hubs and multi-rank are refused; bundles,
  cylinders and stoch-ADMM currently run without refusal and are not validated.
• Extension and converger state is not checkpointed. Rho updaters, fixer
  and slammer start fresh on a resume, so a resumed run using one will not
  retrace an uninterrupted one. The model state is correct and the run
  continues correctly; the matrix encodes this distinction explicitly and the
  user docs say so. A converger is the same story and now says so at runtime —
  the resume warns, since one that accumulates history restarts empty at
  iteration N+1 and can terminate at a different iteration.
  The rho-setting extensions are the exception: they no longer recompute rho
  in post_iter0 on a resume, which would have clobbered the checkpointed rho
  that the resume path skips the rho_setter specifically to preserve.
• Extension order matters at enditer. The Checkpointer is attached before
  configure_extensions appends anything else, and MultiExtension dispatches
  in attach order, so an extension whose enditer mutates models acts after
  that iteration's checkpoint was written and a resume never re-applies it.
  Nothing shipped is affected — every enditer in the tree is a no-op or
  read-only, and the xhat evaluators work in post_everything — so this is
  documented rather than reordered. The durable fix — a dedicated write call in
  iterk_loop — is implemented on the phase 4 branch below, not in this PR.
• A checkpoint is tied to the mpi-sppy and model version that wrote it.

User documentation

doc/src/checkpointing.rst, in Advanced Topics beside pickling.rst. The
design document under doc/designs/ is a design record; Sphinx does not build
it and users do not read it.

What comes next

The rest of the design is built, as a stack of branches on this one. Each is
based on the branch above it, so each carries one phase's diff. None can open
upstream until the one below it merges — a cross-repo PR has to target main,
which would make every stacked PR carry all the phases beneath it. Each has a
CI-green draft preview PR on the fork instead (not for merging there), so
a phase can be read, or tried, on its own:

branch	phase	preview
checkpoint-cylinders	4 — the wheel: hub plus spokes, each spoke keeping its own best xhat	#19
checkpoint-multirank	2 — a cylinder spanning ranks; bundles and stoch-ADMM	#20
checkpoint-extension-state	3 — extension and converger state across a resume	#21
checkpoint-spoke-cursor	5 — the xhat spoke resumes its own exploration	#22

Phase 4 went first, rather than 1b or 2, because mpi-sppy is run hub-and-spoke:
a serial-hub-only feature has few real users. It depends on neither phase 2 nor
phase 3, and §11 of the design records why — n_proc is the cylinder's comm
size rather than COMM_WORLD, so one rank per cylinder already passes the
multi-rank guard here (phase 2 is multi-rank within a cylinder; phase 4 is
multiple cylinders).

One thing on that stack is worth naming here, because it retires a limitation
listed above: checkpoint-cylinders gives the checkpoint write a hook of its
own, Extension.maybe_checkpoint, which iterk_loop calls directly after every
extension's enditer. A model change a user extension makes in its enditer is
then part of that iteration's checkpoint, rather than something a resume drops
for good. (enditer_after_sync is not a substitute — it is skipped on the
spcomm.is_converged() break, so a run ending on cylinder convergence would
write nothing.)

One option beyond the phases, on checkpoint-spoke-cursor:
--checkpoint-before-seconds S writes at the end of the last iteration expected
to finish within S seconds of the start of the run. §8 had dropped it,
reasoning that writing at every completed iteration leaves a recent checkpoint
on disk anyway — which assumes K = 1, and K = 1 is what
--checkpoint-every-iterations exists to avoid on the models this feature
targets. At K > 1 a run stopped by a wall clock stops between multiples of
K, or before the first one, in which case there is no checkpoint at all while
the directory still holds spokes/, so it looks like it worked. The test is
elapsed + the last iteration's duration >= S; it goes through allreduce_or,
because elapsed wall clock is rank-local while the hub write is a collective
bracketed by barriers, and it fires at most once.

Phase 1b is retired. Phase 6 (the leaf-rebuild backend) stays deliberately
unplanned: the use case driving this is served without it.

---

Review history

This PR has been through five review passes. The first found that the
--time-limit planned-stop recipe silently produced a wrong resume; the
second found that the fix for it was incomplete in the same way; the third
motivated the rewrite above. Individual findings and their resolutions are in
the commit messages, which record the measured evidence for each.

The fourth pass concentrated on the resume path, where most of what it found
was silent rather than loud: --APH --resume-from bypassing every guard;
post_iter0 rho extensions clobbering the checkpointed rho; pre_iter0
running on models the splice then discarded, so --init-W-fname was quietly
ignored; the initially-fixed-nonant baseline keyed by a Pyomo name that is not
scenario-qualified, smearing one scenario's fixing across all of them; the
issue-#762 solver-capability checks gated on iteration 1 and therefore never
firing on a resume, which is exactly when a solver switch is permitted; and a
publish sequence that fsynced every model file but no directory, so the
documented commit point did not survive power loss.

The fifth pass found that the fourth's own warn-and-continue fix was
incomplete — only the model dump is wrapped as a RuntimeError, so the leaf
write, the renames and the manifest raise a bare OSError that still took the
run down — along with the enditer ordering note above and two stale
documentation claims. That finding is the reason the write-failure test is now
parameterized over both exception types.

Two corrections to earlier revisions of this description are preserved below
for anyone who read them: stoch_distr is no longer undillable (#830 merged),
and the claim that a Pyomo ConfigDict survives stdlib pickle but not dill
was false — it fails the first serialization under both, for reasons traced to
Pyomo's UninitializedMixin.

---

Original design PR description

This PR adds a design document (doc/designs/checkpointing_design.md) for
checkpointing an mpi-sppy run so it can be stopped and resumed later. It is
doc-only and intended for design review before any implementation.

Use case driving the design

A long (multi-day) run that is intentionally stopped and resumed on a schedule
— e.g. a three-day study that ends each day and picks up the next morning on the
same cluster — with large MIP scenarios. Checkpoints are infrequent (a
handful over the whole run, roughly twice as many writes as resumes). This regime
is what makes the primary design choice below the right one; a different regime
(frequent checkpoints purely for hard-kill safety) is served by the low-cost
backend, also in the doc.

TL;DR

• Do not dill the opt/hub object graph — it holds live MPI communicators, RMA
  windows, and persistent solver handles. Those are always reconstructed via
  normal startup (a fresh MPI job each morning).
• Do dill the mid-run scenario models — this is the recommended backend for
  the use case above. It captures W/rho/nonants/fixedness, the second-stage
  MIP warm start, the linearized-prox cuts, and model-attached extension state
  in one consistent shot, and avoids re-running an expensive scenario_creator.
  At a few checkpoints the write cost is negligible, and same-environment resume
  makes dill's version fragility a non-issue.
• A low-cost leaf-rebuild backend remains available (--checkpoint-backend leaf): rebuild via scenario_creator and overlay tiny numeric arrays — small,
  fast, version-robust — for small/cheap-creator runs or frequent kill-safety
  writes.
• Keep the best xhat, not just the best bound: in a cylinders run the best
  xhat solution values live on the xhat spoke, while the hub holds only the
  incumbent objective. The design checkpoints both.

Validated by proofs of concept

• Framework + leaf-rebuild backend (farmer LP, gurobi_persistent): serial and
  multi-rank (-np 3, uneven -np 2) resume reproduces a full run
  bit-identically for W/nonants/rho; cylinders (PH hub + lagrangian +
  xhatshuffle) hub primal resumes bit-identical inside WheelSpinner; geometry
  mismatch is refused with a clear error.
• dill-reload backend on a MIP (sizes SIZES3, gurobi_persistent,
  single-thread for a deterministic solve): a mid-run scenario model — including
  the linearized-prox case with 845 cuts + 65 ProxApproxManagers — dills and
  reloads in-process and cross-process, re-solving to identical objective and
  decision variables; a serial stop → reload → continue is bit-identical to
  an uninterrupted run for both quadratic and linearized prox.
• Still to prove in later phases: the dill-reload backend under multi-rank and
  cylinders, incumbent carry across a dill-reload stop, a measured warm-start
  speedup, and the disk footprint at true model scale.

Determinism contract

• MIPs (the target): resume continues correctly and warm-started, with the
  best xhat preserved — but is not bit-reproducible (multi-threaded MIP solves
  are nondeterministic and admit multiple optima). The PoC's bit-identity results
  used a single-thread deterministic solve as a validation crutch.
• Leaf-rebuild on a deterministic LP/QP solve: primal trajectory can be
  bit-identical (a bonus, not the target).
• Bounds and incumbent: valid, best-so-far, not bit-reproducible (async).

What the doc specifies

State inventory (reconstructed vs restored; what rides in the dilled model vs
non-model leaf data), the required core changes (global iteration counter, a
reload-model resume branch that skips re-attach and refreshes objective handles, a
one-line xhatter write hook so a single Checkpointer serves hub + spoke, an
extension checkpoint_state/restore_state contract, async per-spoke incumbent
checkpoints with no hub↔spoke coordination, atomic single-generation writes via a
manifest), a proposed file layout, and a 6-phase rollout (each phase a
review-sized PR).

Ready for review. Implementation would land as the phased PRs described in the doc.

---

Update log

Per review (@bknueven): leaned into mpi-sppy's asynchrony — dropped the
hub-triggered snapshot barrier (each spoke checkpoints its own incumbent
asynchronously on improvement) and simplified retention to a single
manifest-published generation.

Design pivot (this revision): re-centered on the concrete use case above
(multi-day planned stop/resume, huge MIP scenarios, same environment). The
recommended backend is now dill the mid-run scenario models rather than
leaf-data-only — at this checkpoint cadence the overhead is negligible and it
captures the MIP warm start + prox cuts for free — with leaf-rebuild kept as the
low-cost alternative. Softened the determinism contract for MIPs, shifted the
primary trigger to end-of-run/on-signal, and validated the load-bearing
assumption (a mid-run MIP model round-trips through dill) with the MIP PoC above.

Design review pass (this revision): added §8.2 stochastic ADMM (wrapped-name
file discovery — no scenario_names_creator/extract_num; the creator-cost saving
does not apply and resume must release the wrapper-held fresh models; the
variable_probability identity-keying invariant; the wrapper-mutated model dill
round-trip flagged for early validation; AdmmBundler; spoke-set differences).
Restore is now an in-core resume branch in Iter0 replacing the iter-0 solve
(precedent: iter0_from_pickle) — no throwaway W = 0 solve on resume, and the
special end-of-iteration-0 checkpoint is dropped. Added §11.1, an A/B
(uninterrupted vs stop+resume) CI harness with explicit similarity criteria over
small instances: farmer, farmer+--cvar, stoch-distr (--stoch-admm), and
sizes. §12 recast as resolved decisions + deferrals.

Anticipated checkpoint trigger (this revision): added
--checkpoint-before-seconds S (§8) — a one-shot, anticipatory trigger
that writes one checkpoint at the last iteration boundary preceding S
wall-clock seconds and lets the run continue (it terminates nothing). The existing
--checkpoint-every-seconds is reactive: it fires at the first boundary at or
after its interval, so the write lands up to a full iteration late — no good
when the deadline is a scheduler walltime and the iterations are hour-scale MIP
solves. The new trigger tests
allreduce_or(elapsed + last_iteration_seconds >= S) at enditer, the same
collective pattern as the time_limit check in phbase.py, then latches. It
OR's with the other triggers.

S is used exactly as given: no fudge factor on the iteration estimate, and
no attempt to predict the checkpoint write cost. Instead the library measures
and the user does the arithmetic — every checkpoint write is now bracketed by
global_toc (§9 item 10), on every trigger, hub and spokes, so the pair of
timestamps is the measured write duration; the doc tells the user to set S to
their walltime minus that duration minus their own margin. The doc is also
explicit about what the clock measures — elapsed runs from self.start_time
in SPBase.__init__, so queue wait, interpreter startup, imports, and MPI init
fall outside S (scenario construction and the resume-time dill reload fall
inside), and that startup gap has to be subtracted too.

Supporting core change: §9 item 9 hoists the most-recent iteration duration onto
self (today it is a local in iterk_loop used only by the display_progress
print), seeded with iteration 0's duration so no special case is needed the first
time the trigger is tested. Phase 1 in §11 picks all of this up, and the §1
signal-handling non-goal now points at this trigger as the walltime answer instead
of a signal handler.

Pre-implementation verification pass (this revision): re-checked every §9
touch-point against current main before starting Phase 1, and corrected three
things the implementation would have hit immediately.

• The model swap goes before _create_solvers(), not after (§5.1, §9 item
  2). The old placement makes every scenario pay set_instance twice per resume
  — once on the fresh model that is about to be discarded — which is exactly the
  cost the dill-reload backend exists to avoid at this design's target scale.
• The two objective attaches are not symmetric, and the doc named the wrong
  one. attach_Ws_and_prox runs in PH_Prep, upstream of Iter0, so a
  branch inside Iter0 cannot skip it — and does not need to, since it decorates
  models the swap discards. What must be handled is the deferred attach at the
  end of Iter0, which would double the W terms and duplicate the prox
  components on a reloaded model. The branch clears _deferred_ph_attach rather
  than relying on control flow to miss it.
• New §9 item 11: restore the initially-fixed-nonant baseline, by name.
  _initial_fixed_varibles is opt-object state keyed by variable identity, so
  the model swap breaks it in both directions. Left alone it holds vardata from
  the discarded models, so _can_update_best_bound sees every fixed nonant as
  unrecognized and a resumed run silently stops updating its best bound —
  including a run whose only fixed nonant came from the user's own
  scenario_creator, no extension involved. Naively rebuilt after the swap it
  absorbs whatever fixer/slammer pinned before the stop, so those pass as
  original and the resumed run accepts a bound the uninterrupted run would have
  refused. The checkpoint records the baseline by variable name. Same
  identity-keying hazard §8.2 item 3 already catches for ADMM's varprob_dict,
  here in the core PH path — which is why it lands in the first phase rather than
  with the fixing extensions.

Phase 1 split (this revision): §11 Phase 1 was one PR carrying the framework,
four triggers, CLI flags, and a three-instance A/B harness. It is now 1a (the
whole serial stop-and-resume story with the terminal trigger the primary use case
actually needs; farmer A/B) and 1b (the optional periodic and anticipated
triggers, plus the farmer+CVaR and sizes MIP instances). Each is green on its
own, and 1a is independently useful: a run that stops at --time-limit and
resumes the next morning needs nothing from 1b.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 89.51407% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.01%. Comparing base (29671ec) to head (f3d9423).

Files with missing lines	Patch %	Lines
mpisppy/utils/checkpointing.py	86.00%	28 Missing ⚠️
mpisppy/extensions/sensi_rho.py	0.00%	3 Missing ⚠️
mpisppy/extensions/checkpointer.py	96.66%	2 Missing ⚠️
mpisppy/extensions/grad_rho.py	33.33%	2 Missing ⚠️
mpisppy/extensions/sep_rho.py	33.33%	2 Missing ⚠️
mpisppy/phbase.py	97.18%	2 Missing ⚠️
mpisppy/extensions/integer_relax_then_enforce.py	80.00%	1 Missing ⚠️
mpisppy/utils/cfg_vanilla.py	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #777      +/-   ##
==========================================
+ Coverage   77.77%   78.01%   +0.23%     
==========================================
  Files         177      179       +2     
  Lines       23764    24137     +373     
==========================================
+ Hits        18482    18830     +348     
- Misses       5282     5307      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bknueven]

I'm confused on this point -- doesn't serializing the best solution as already stored by the xhat / incumbent spokes already do this?

[DLWoodruff]

Yes — and that's the direction I took the revision. §5.3 already identified the xhat spoke's best_solution_cache as the thing to serialize; I've now made it explicit (revised §5.3 and §9 item 6) that the spoke checkpoints that cache itself, asynchronously on each improvement (reusing _maybe_write_incumbent_on_improvement), with no separate hub-driven coordination. So there's no extra machinery beyond serializing what the spoke already holds.

[bknueven]

Isn't it sufficient to just keep a single snapshot? After we complete the operations to write the new snapshot, can't we just immediately delete the prior one?

[DLWoodruff]

Agreed — revised §9 item 7 to a single published generation. The new generation is written to fresh files, then manifest.json is atomically renamed to point at it (the single commit point), then the prior generation is deleted. A kill before the flip keeps the previous complete checkpoint; a kill after keeps the new one. So one committed generation is enough for kill-safety; keeping older generations is now just optional history, not a requirement.

[DLWoodruff]

Follow-up: the doc now goes one step further and pins this down — retention of more than one checkpoint is explicitly not supported (d11832a). Exactly one committed generation exists at any time; the prior one is deleted right after the manifest flip, and the only time two exist is transiently during a publish (the documented peak disk footprint). The earlier "older generations are optional history" wording is gone.

[Copilot]

Pull request overview

Adds a design document describing a checkpoint/resume architecture for mpi-sppy runs (including cylinders), focusing on planned stop/resume of long-running, large-scenario MIP studies by reconstructing MPI/solver scaffolding and restoring algorithm/model state (primarily via dilling mid-run scenario models).

Changes:

• Introduces a doc-only design specifying what state is reconstructed vs restored, and proposes two backends (dill-model recommended; leaf alternative).
• Defines checkpoint semantics (triggers, atomic publication via a manifest, geometry/cfg fingerprinting) and details state inventory including incumbents and extension state.
• Proposes a phased implementation plan with testing expectations and rollout milestones.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DLWoodruff]

Good catch — this was inaccurate, and it flagged a real implementation gotcha. Corrected in 3bcc029: a plain PH has no local_subproblems, but the generic file-based path the dill-reload backend builds on sets sp.local_subproblems = sp.local_scenarios (scenario_io.py) and CGBase.solve_loop iterates it, so the reload branch must refresh that alias where it exists or it dangles to the discarded model. (Re-opened — this had been resolved by mistake.)

[DLWoodruff]

Agreed — replaced the file.py:line references with file + symbol (e.g. warmstart_subproblems in spopt.py) in 3bcc029 so they stay stable as the code moves.

[DLWoodruff]

Fixed in 3bcc029 — the CLI flag is now --checkpoint-backend dill-reload (was dill-model), so the flag and the narrative use one name, with "reload" as the restore step.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Suppressed comments (6)

doc/designs/checkpointing_design.md:969

• The rollout record is stale: this heading still calls phase 1a “terminal trigger only,” although the implementation has no terminal trigger and this PR includes --checkpoint-every-iterations; the phase 1b bullet below also still claims that option. Update the phase split so the design agrees with the implemented scope described in §8 and the PR description.

- **Phase 1a — Serial hub checkpoint/resume, terminal trigger only.** The

mpisppy/phbase.py:1230

• Setting solution_available alone does not enable a warm start. SPOpt._solve_one only passes warmstart to the solver when options["warmstart_subproblems"] is true, and that option defaults to false (config.py:305-308). Thus the documented default resume path is cold-started. Either arrange a one-shot warm start for the first resumed solve or document/require --warmstart-subproblems.

            model._mpisppy_data.solution_available = True

mpisppy/utils/checkpointing.py:278

• This probes only the first local scenario, so a later scenario with scenario-specific unserializable state is not discovered until the first real checkpoint. enditer then catches that failure and the run can finish after hours without ever publishing a checkpoint, contrary to the setup-time refusal guarantee. Probe every local scenario here.

    sname, s = next(iter(opt.local_scenarios.items()))

mpisppy/tests/test_checkpoint.py:254

• The A/B resume occurs in the same interpreter, despite the design's acceptance gate requiring a fresh process (checkpointing_design.md:907-909) and the real use case being a new job. This can miss failures caused by module re-import, process-local registrations, or dill behavior that only appears when no writer-process state remains. Run the resumed leg in a subprocess and compare serialized snapshots across the process boundary.

        resumed = _make_ph(_options(self.N, resume_from=self.ckpt_dir))
        resumed.ph_main()

mpisppy/tests/test_checkpoint.py:79

• This fixed 0.25-second threshold is not guaranteed to allow iteration 1 to complete: the time limit includes scenario construction, solver startup, and Iter0, and several matrix cells add smoothing or linearized-prox setup. On a slower CI host the “later” case becomes the “iteration 1” case and incorrectly expects a checkpoint. Control perf_counter in the test (or synchronize the limit to an observed completed iteration) instead of relying on wall-clock speed.

_LATE_TIME_LIMIT = 0.25

doc/designs/checkpointing_design.md:940

• This leftover fragment says the stoch-ADMM model still needs to be fixed immediately after the preceding lines state that #830 fixed and unblocked it. Remove the stale sentence so the validation status is unambiguous.

This issue also appears on line 969 of the same file.

  the model is fixed or another stoch-ADMM model is chosen.

[DLWoodruff]

Thanks — all six of the suppressed findings from the latest review were real. Each was checked against the code before being acted on; fixes are in 511decf.

1. phbase.py — the warm-start claim was false by default. Confirmed: _solve_one (spopt.py) gates on options["warmstart_subproblems"], which defaults to False (config.py), and solution_available is only consulted inside that gate. So the reloaded recourse values did nothing unless the user asked for warm starts. Fixed as documentation rather than behavior — mpi-sppy does not turn a solver option on for you, so the code comment, checkpointing.rst and §9 item 2 now say the flag is what makes --warmstart-subproblems work across a resume and is inert without it. The unqualified "warm-started" promise is gone.

2. checkpointing.py — the dillability probe checked one scenario. Confirmed, and fixed to probe every local scenario. What makes a model undillable is usually something the scenario_creator closed over, and a creator that does it for one scenario is exactly the case a single-scenario probe waves through — after which the run fails at every write, survives each failure by design, and finishes having published nothing, which is what the setup-time refusal exists to rule out. Nothing covered this before; there are now tests for the first-scenario and later-scenario cases, and the later-scenario one fails against the old probe.

3. §11 phase 1a still said "terminal trigger only". Confirmed stale. The heading now describes what shipped, and phase 1b no longer claims --checkpoint-every-iterations; it also records that the two seconds-triggers are not implemented and not planned, since §8 concluded that writing at every completed iteration subsumes them.

4. The dangling stoch-ADMM clause at §11.1. Confirmed — a leftover from the rewrite when #830 landed. Removed.

5. The A/B harness never left the writing interpreter. Confirmed, and it matters for the reason given: the real use case is a new job. The resumed leg now runs in a subprocess and the snapshots are compared across the process boundary. One addition worth noting — the test also asserts the subprocess actually resumed, because farmer is deterministic: a leg that ignored the checkpoint and ran all N iterations from scratch would produce identical state and pass.

6. _LATE_TIME_LIMIT = 0.25 raced the clock. Confirmed. The limit also covers model construction, solver startup and Iter0, so on a slow host the cell became the "no checkpoint published" case and inverted its own assertion. An extension now rewinds start_time at a chosen iteration, so the real check, allreduce and break still run with the clock under control. It asserts the run stopped at that iteration, since otherwise the cell would pass by hitting the iteration limit instead and quietly stop covering the --time-limit exit.

On the followup branch (checkpoint-cylinders, phase 4), the same fresh-process discipline turned up a real bug that had been invisible: a spoke given --resume-from without --checkpoint-dir attempted a write with no directory to write to, on every improvement. Nothing failed, because checkpoint write failures warn and continue by design — the only trace was two lines in a log.

🤖 Generated with Claude Code