Skip to content

openvmm: drop the snapshot quiesce timeout and microsoft/openvmm#4625 #1260

Closed

Description

Objective

Decide whether to remove the snapshot quiesce timeout from OpenVMM and drop microsoft/openvmm#4625, fork position 021 (03291e1ea), and carry out the removal if it is accepted.

The microVM capture would then quiesce with upstream's unbounded StateUnits::stop, and roll back a failure that follows a complete quiesce with StateUnits::start and the start rollback of microsoft/openvmm#4872. The process that drives OpenVMM, nvx-rs or a host agent, would own the capture's deadline.

Background

microsoft/openvmm#4625 adds StateUnits::quiesce_for_save, which stops every state unit within --snapshot-quiesce-timeout-ms, 5 s by default, and classifies a failure as rollback-safe or uncertain, and StateUnits::resume_after_failed_save. Fork positions 076 and 080 use them in the microVM capture, through LoadedVm::quiesce_for_snapshot and the controller's handling of SnapshotQuiesceError, and the time ABI commits at positions 143, 151, and 206 extend that path.

The timeout adds no safety and little liveness:

  • A timed-out quiesce is always terminal. It leaves a Stop in flight, so it is uncertain, and the source is terminated. Rollback only follows a complete quiesce, when the time contract check, the save, the flush, or the publication fails, and it needs no timeout.
  • It does not let a stuck VMM exit. With the head of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625, teardown's stop first awaits the retained Stop without a deadline, and the controller joins the VM worker without one, so a VMM whose unit never stops runs until something kills it. The pin exits at about 5 s only because teardown aborts on the uncertain unit, as openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148 reports; release builds abort on a panic.
  • NVX's captures already have a deadline outside OpenVMM. Store::create_template and Store::refresh_template require one, and run_supervised ends the VMM once it passes, with SIGKILL through a pidfd or TerminateProcess, then waits up to KILL_TIMEOUT, 10 s. The runtime adapter allows 120 s for a whole creation, and the Python harness's capture_snapshot helpers require a timeout.
  • Normal captures stay far below the bound. The p50 of openvmm_snapshot_generation, from input gating through publication, is about 1.3 ms on KVM, 3 ms on MSHV, and 760 ms on WHP.
  • The timeout causes most of the complexity of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625. Uncertain and terminal quiesce states, retained Stops, and rollback validation exist because a timed-out Stop leaves an operation in flight. The pull request adds 1,946 lines, is a draft, has had 14 Copilot review passes that kept finding high-severity issues, has open threads, and has no maintainer review.

Pros and cons

Pros of dropping it:

  • The largest and least settled part of PR09 leaves the upstream queue.
  • The uncertain-quiesce state machine goes away, and so do the uncertain-quiesce cases of openvmm: an uncertain snapshot quiesce makes teardown or a later lifecycle request panic #1148.
  • No safety is lost. An unfinished quiesce never returns, so the source never resumes, and nothing is published, because the rename is the commit point.
  • A slow quiesce succeeds instead of failing at 5 s.
  • State-transition rules stay in OpenVMM and time budgets with the caller, as upstream's state_unit already assumes.

Cons of dropping it:

Either way:

  • Every driver must bound a capture with a deadline that ends the VMM, because a unit that never stops keeps the VMM alive in both designs. The runtime adapter's 120 s also covers boot and lock waits.
  • A driver cannot tell when a guest starts a capture. A deadline on the whole VM fits template captures, which boot, capture, and exit, but a capture during a workload would need OpenVMM to report its start.
  • The deadline holds only while the driver runs, because OpenVMM outlives its caller, and a Linux host without pidfds cannot kill the VMM, which ends the launch as VmmIndeterminate.
  • A unit whose task ends aborts OpenVMM: run_op panics on a lost unit both upstream and at the head of state_unit: add bounded quiesce transactions with rollback classification openvmm#4625.

Work

Acceptance criteria

  • A capture whose state unit never stops publishes no snapshot and never resumes its source; the driver's deadline ends it, and the VMM's log names the unit.
  • A capture that fails after a complete quiesce resumes its source, whose guest gets Captured::Continued.
  • A capture whose input gate fails lets the guest continue, and a slow input gate is awaited rather than abandoned.
  • Every driver bounds its captures: the nvx crate's template store, the runtime adapter, the Python harness, and the OpenVMM VMM tests, whose microVM ttrpc test no longer sets quiesce_timeout_ms and bounds its capture with its 15 s drain and TestChild's kill on drop.
  • Package-scoped OpenVMM checks and the affected microVM scenarios pass on KVM, MSHV, and WHP, through available hosts or CI, recording unavailable coverage explicitly.

Coordination

Related work Integration
microsoft/openvmm#4625 Closed if this is accepted; fork position 021 is dropped.
microsoft/openvmm#4626 Rebased onto microsoft/openvmm#4872; keeps only its input gate, without a deadline.
microsoft/openvmm#4872 Becomes the base of the rest of PR09's lifecycle work; has no timeouts.
#1232 Its work list and acceptance criteria keep bounded quiesce; update them with the decision.
#1181 Umbrella of #1138, #1148, #1149, and #1170; re-triage as above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions