Repository navigation
Description
Objective
Decide whether to remove the snapshot quiesce timeout from OpenVMM and drop microsoft/openvmm#4625, fork position 021 (03291e1ea), and carry out the removal if it is accepted.
The microVM capture would then quiesce with upstream's unbounded StateUnits::stop, and roll back a failure that follows a complete quiesce with StateUnits::start and the start rollback of microsoft/openvmm#4872. The process that drives OpenVMM, nvx-rs or a host agent, would own the capture's deadline.
Background
microsoft/openvmm#4625 adds StateUnits::quiesce_for_save, which stops every state unit within --snapshot-quiesce-timeout-ms, 5 s by default, and classifies a failure as rollback-safe or uncertain, and StateUnits::resume_after_failed_save. Fork positions 076 and 080 use them in the microVM capture, through LoadedVm::quiesce_for_snapshot and the controller's handling of SnapshotQuiesceError, and the time ABI commits at positions 143, 151, and 206 extend that path.
The timeout adds no safety and little liveness:
- A timed-out quiesce is always terminal. It leaves a Stop in flight, so it is uncertain, and the source is terminated. Rollback only follows a complete quiesce, when the time contract check, the save, the flush, or the publication fails, and it needs no timeout.
- It does not let a stuck VMM exit. With the head of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625, teardown's
stopfirst awaits the retained Stop without a deadline, and the controller joins the VM worker without one, so a VMM whose unit never stops runs until something kills it. The pin exits at about 5 s only because teardown aborts on the uncertain unit, as openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148 reports; release builds abort on a panic. - NVX's captures already have a deadline outside OpenVMM.
Store::create_templateandStore::refresh_templaterequire one, andrun_supervisedends the VMM once it passes, withSIGKILLthrough a pidfd orTerminateProcess, then waits up toKILL_TIMEOUT, 10 s. The runtime adapter allows 120 s for a whole creation, and the Python harness'scapture_snapshothelpers require a timeout. - Normal captures stay far below the bound. The p50 of
openvmm_snapshot_generation, from input gating through publication, is about 1.3 ms on KVM, 3 ms on MSHV, and 760 ms on WHP. - The timeout causes most of the complexity of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625. Uncertain and terminal quiesce states, retained Stops, and rollback validation exist because a timed-out Stop leaves an operation in flight. The pull request adds 1,946 lines, is a draft, has had 14 Copilot review passes that kept finding high-severity issues, has open threads, and has no maintainer review.
Pros and cons
Pros of dropping it:
- The largest and least settled part of PR09 leaves the upstream queue.
- The uncertain-quiesce state machine goes away, and so do the uncertain-quiesce cases of openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148.
- No safety is lost. An unfinished quiesce never returns, so the source never resumes, and nothing is published, because the rename is the commit point.
- A slow quiesce succeeds instead of failing at 5 s.
- State-transition rules stay in OpenVMM and time budgets with the caller, as upstream's
state_unitalready assumes.
Cons of dropping it:
- OpenVMM no longer names the unit that did not stop, unless it logs the units that are still stopping. The driver learns of the failure no later, since with the head of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625 a VMM whose unit never stops cannot exit before the driver kills it either.
- It forecloses a self-bounded OpenVMM. Finishing state_unit: add bounded quiesce transactions with rollback classification openvmm#4625 so that the VMM exits after an uncertain quiesce, without awaiting the retained Stop, would bound a capture without any driver.
- Upstream's own save paths, which also stop units without a bound, lose a general fix, although state_unit: add bounded quiesce transactions with rollback classification openvmm#4625 does not wire them in.
SnapshotQuiesceError, the quiesce timeout's CLI and ttrpc settings, four design sections, and the work list of openvmm: refactor device restoration onto upstream async save/restore #1232 change.
Either way:
- Every driver must bound a capture with a deadline that ends the VMM, because a unit that never stops keeps the VMM alive in both designs. The runtime adapter's 120 s also covers boot and lock waits.
- A driver cannot tell when a guest starts a capture. A deadline on the whole VM fits template captures, which boot, capture, and exit, but a capture during a workload would need OpenVMM to report its start.
- The deadline holds only while the driver runs, because OpenVMM outlives its caller, and a Linux host without pidfds cannot kill the VMM, which ends the launch as
VmmIndeterminate. - A unit whose task ends aborts OpenVMM:
run_oppanics on a lost unit both upstream and at the head of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625.
Work
- Decide, and record the decision in openvmm: refactor device restoration onto upstream async save/restore #1232, whose work list and acceptance criteria keep bounded quiesce and failed-save deadlines.
- Replace
quiesce_for_saveandresume_after_failed_saveinLoadedVm::quiesce_for_snapshotandLoadedVm::resume_after_failed_snapshotwithStateUnits::stopandStateUnits::start, and removeSnapshotQuiesceError::Uncertainand the controller's handling of it. - Remove the bounds of input gating, from vmcore: add device input quiesce and resume openvmm#4626, and of the console flush before a capture too, since their expiry recovers nothing: the gate's timeout abandons the wait but not the work, so its rollback queues behind the unit that it stopped waiting for and releases the guest no sooner, and the flush's expires only when the endpoint's reader stops reading, and then saves the unsent bytes in the snapshot, which every restored VM writes again. Input gating and its rollback wait for every unit; a unit that reports an error still rolls the gate back and lets the guest continue, and only a failed rollback ends the VM. Remove
--snapshot-quiesce-timeout-ms, and reserve ttrpc's field 3 with both of its names,quiesce_timeout_msandboundary_timeout_ms.--restore-gate-timeout-ms, which bounds the guest's acknowledgement of a restore, stays. - Log the state units that are still stopping after a few seconds, so that the log of a killed capture names them.
- Rebase vmcore: add device input quiesce and resume openvmm#4626 onto state_unit: roll back a partial start instead of panicking openvmm#4872 and drop its input gate's bound, and close state_unit: add bounded quiesce transactions with rollback classification openvmm#4625.
- Add a microVM scenario in which a capture fails after a complete quiesce and the source resumes. No test covers that path today: the
SNAPSHOT-CONTINUEDtests cover a request without a destination. - Re-triage the children of OpenVMM/microVM umbrella: snapshot quiesce, rollback, and device restoration in the VM worker #1181. The uncertain-quiesce cases of openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148 go away; check the rest of openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148, and openvmm: failed snapshot rollback leaves restarted devices running #1149, against state_unit: roll back a partial start instead of panicking openvmm#4872. openvmm: snapshot quiesce and boundary release allow ordinary Resume to bypass rollback #1170 stays, and openvmm: snapshot input gating closes device mutexes while other vCPUs still access them #1138 is unaffected.
- Update the design: 5.3.1 Capture Configuration, for the quiesce timeout; 5.3.3 Capture Transaction, for "one bounded transaction" and the console flush; 5.3.5 Commit Point and 5.3.6 Capture Outcomes, for an input gate that fails rather than one that is late; 3.4.5 Snapshot Contract and Quiesce, for the drain's bound; and 1.4.1 Defense in Depth, for "fallible quiesce".
- Promote the OpenVMM revision with the gitlink-only workflow, and update the roadmap: PR09 loses position 021, and position 022 rests on state_unit: roll back a partial start instead of panicking openvmm#4872.
- Optionally, and whatever the decision, make
StateUnits::stopfail instead of panicking when a unit's task has ended, as state_unit: roll back a partial start instead of panicking openvmm#4872 does forstart.
Acceptance criteria
- A capture whose state unit never stops publishes no snapshot and never resumes its source; the driver's deadline ends it, and the VMM's log names the unit.
- A capture that fails after a complete quiesce resumes its source, whose guest gets
Captured::Continued. - A capture whose input gate fails lets the guest continue, and a slow input gate is awaited rather than abandoned.
- Every driver bounds its captures: the nvx crate's template store, the runtime adapter, the Python harness, and the OpenVMM VMM tests, whose microVM ttrpc test no longer sets
quiesce_timeout_msand bounds its capture with its 15 s drain andTestChild's kill on drop. - Package-scoped OpenVMM checks and the affected microVM scenarios pass on KVM, MSHV, and WHP, through available hosts or CI, recording unavailable coverage explicitly.
Coordination
| Related work | Integration |
|---|---|
| microsoft/openvmm#4625 | Closed if this is accepted; fork position 021 is dropped. |
| microsoft/openvmm#4626 | Rebased onto microsoft/openvmm#4872; keeps only its input gate, without a deadline. |
| microsoft/openvmm#4872 | Becomes the base of the rest of PR09's lifecycle work; has no timeouts. |
| #1232 | Its work list and acceptance criteria keep bounded quiesce; update them with the decision. |
| #1181 | Umbrella of #1138, #1148, #1149, and #1170; re-triage as above. |
Activity
test-microvm: roll back a capture that fails after its quiesce
openvmm: pin the snapshot quiesce without a deadline
doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap
doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap
state_unit: log the units that a state change waits for
microVM: quiesce snapshots without a deadline
test-microvm: roll back a capture that fails after its quiesce
openvmm: pin the snapshot quiesce without a deadline
doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap
test-microvm: roll back a capture that fails after its quiesce
openvmm: pin the snapshot quiesce without a deadline
doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap
microVM: quiesce snapshots without a deadline
test-microvm: roll back a capture that fails after its quiesce
openvmm: pin the snapshot quiesce without a deadline
doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap
state_unit: log the units that a state change waits for