Repository navigation
Description
Objective
Refactor NVX's OpenVMM device restoration onto the asynchronous save/restore interfaces that landed in microsoft/openvmm#4799, instead of using device start to complete restoration and report restore errors.
The decision is to retire the three upstream fallible-start/resume proposals listed below and pursue their replacement here. This does not establish that activation is already infallible: remaining host-backend, queue-start, and worker failures must be accounted for explicitly before removing the existing implementation from NVX's fork.
Background
The current fork validates and stages virtio device-private state during restore, then applies it during activation. For an active restored device, failures such as reopening virtio-fs host handles therefore surface through start/resume.
Upstream now supports awaited, fallible restore through the chipset and remote device-worker boundaries. Use that capability to complete device restoration while the VM remains stopped, separating preparation from activation. Coordinate with microsoft/openvmm#4713, which already proposes parked device-task access, restore-time queue reconstruction, and queue retention across ordinary pause/resume. Its host backend queue creation still occurs during asynchronous activation; it is not proof that every startup failure has disappeared.
Superseded upstream proposals
The following proposals are closed in favor of this tracking issue. Their branches and the current fork implementation are retained:
| Pull request | Scope | Reviewed head |
|---|---|---|
| microsoft/openvmm#4619 | Fallible state-unit start and partial-start rollback | 960826ee06b6d9fb5c2dee0c292c9f039d295e83 |
| microsoft/openvmm#4622 | Fallible chipset-device start and worker acknowledgments | 24d13d56fb9c5df355d62a13c90ad369f5be8f39 |
| microsoft/openvmm#4623 | Fallible VM resume and caller propagation | d776c0eb30ba06d42d15189f4b09721577e9892d |
These are superseded proposals, not merged fixes. Closing them must not silently discard their error-handling and cleanup requirements.
Work
- Agree the restore/prepare/activate contract with the authors of virtio_net: preserve queues across pause and restore from saved cursors openvmm#4713 and virtio_net: separate vhost-user ring enable from start and stop openvmm#4729. Reuse their queue-ownership and parked-task design where applicable rather than introducing a competing lifecycle.
- Move device-private restoration and appropriate host-resource preparation into awaited restore. Cover both PCI and MMIO transports, worker-backed devices, and the virtio-fs, console, and network consumers. Report restoration failures even when the caller requests a restored-but-paused VM.
- Preserve the stopped-state boundary: restoration must not start live guest queue processing or admit external input. Preserve inactive/not-DRIVER_OK behavior and defer restore-generated notifications until the correct interrupt/start boundary.
- Audit every remaining fallible activation path, including cold boot, ordinary pause/resume, fresh-snapshot activation, hot-add, worker restart/recovery, and rollback after a failed save. Move fallible preparation earlier or retain/propose the narrowly scoped error handling still required. Do not turn failures into successful fire-and-forget starts.
- Define cleanup after partial restore, cancellation, reset, and removal before first start. Release prepared resources without abandoning issued I/O or publishing premature removal success.
- Extract/rebase state_unit: add bounded quiesce transactions with rollback classification openvmm#4625 and vmcore: add device input quiesce and resume openvmm#4626 away from the superseded state_unit: make unit start fallible and roll back partial starts openvmm#4619 implementation as needed. Preserve bounded quiesce, dependency ordering, uncertain-outcome classification, partial-rollback cleanup, and pre-vCPU-stop input gating. Async restore does not replace these guarantees.
- Preserve the independent exact-inventory validation of state_unit: record the exact unit inventory in saved state openvmm#4624 before device restoration begins. Audit error propagation through launch, restore-ready signaling, REPL, ttrpc, and test callers.
- Update affected tests and documentation, including the lifecycle and generic virtio persistence sections of the OpenVMM upstream roadmap, to distinguish restore-time failures from residual activation failures and record the superseded PRs.
- After validation, promote the replacement OpenVMM revision into NVX using the repository promotion workflow: gitlink-only integration commit and a separate roadmap update. Only then remove obsolete fork plumbing.
Coordination
| Related work | Required integration |
|---|---|
| microsoft/openvmm#4713, microsoft/openvmm#4729 | Retained internal pause versus ownership-releasing queue stop; restore-time cursor reconstruction; backend activation failures. |
| microsoft/openvmm#4662, microsoft/openvmm#4863 | Awaited resource release, stopped-only teardown, and cleanup ownership independent of the requesting future. |
| microsoft/openvmm#4855 | Establish restored interrupt outputs while forwarding is stopped; start targets before replay and producers. Preserve virtio notification handling as well. |
| microsoft/openvmm#4553, microsoft/openvmm#4676 | RestoreVM with and without automatic resume, consistent error reporting, and cleanup of failed restored workers. Reconcile newly added resume callers with whichever activation contract remains. |
| microsoft/openvmm#4624 | Exact state-unit inventory remains independent and must not be dropped. |
| microsoft/openvmm#4625, microsoft/openvmm#4626 | Keep these proposals; adapt their dependencies and rollback implementation rather than closing them. |
The source work belongs in the nested OpenVMM repository. The roadmap's PR13 deferred-persistence work and its microVM device consumers are part of the migration, even where they have no standalone upstream PR yet.
Related NVX work:
- OpenVMM/microVM umbrella: snapshot quiesce, rollback, and device restoration in the VM worker #1181 tracks existing snapshot quiesce/rollback correctness issues. Preserve its requirements; this refactor does not close those issues by itself.
- snapshot: align NVX manifest with upstream SaveVM/RestoreVM #1210 tracks snapshot-manifest alignment with upstream SaveVM/RestoreVM. Do not duplicate its format/tag migration here or assume a lifecycle refactor authorizes another format change.
Acceptance criteria
- A restored-but-paused VM has completed the required device restoration. Invalid device-private state, missing host resources, and failed worker restoration are reported before successful restore/readiness, without requiring a subsequent resume.
- Restoring does not run live queues or deliver notifications/input prematurely. Tests cover active and inactive devices, restore-generated notifications, and re-save/reset/removal before first start.
- Repeated pause/resume works without invoking snapshot restore, losing queue progress, duplicating TX, or leaking resources. Retained-resource and ownership-releasing stop paths remain distinct.
- Fault-injection tests cover partial restoration, backend/queue activation failures that remain, worker/channel failure, and interrupted cleanup. A partially prepared or activated VM never reports success; resources and already-started work are cleaned up according to the documented contract.
- Failed-save recovery retains deadlines and dependency ordering. Uncertain transitions cannot be bypassed by ordinary resume, and later lifecycle requests do not panic on invalid unit state.
- Package-scoped OpenVMM checks and affected NVX tests pass. Validate the affected lifecycle scenarios on KVM, MSHV, and WHP through available hosts/CI, explicitly recording unavailable coverage rather than assuming it passed.
- The replacement is integrated and the roadmap reflects the final architecture before obsolete fork implementations are removed.
Activity
openvmm: pin microVM v4.0, reorganized onto upstream main
doc: map the OpenVMM roadmap onto the microVM v4.0 reorganization
openvmm: pin microVM v4.0, reorganized onto upstream main
doc: map the OpenVMM roadmap onto the microVM v4.0 reorganization
ppenna commentedon Nov 5, 2026
Decision recorded from #1260, implemented by #1288 and microsoft/openvmm#4921: the snapshot quiesce has no deadline of its own any more, so this issue's bounded-quiesce requirements change.
StateUnits::stop, which keeps the reverse dependency order but has no deadline and no uncertain outcome. Partial-rollback cleanup comes from state_unit: roll back a partial start instead of panicking openvmm#4872's start rollback, which the capture's rollback now uses, and pre-vCPU-stop input gating stays bounded in vmcore: add device input quiesce and resume openvmm#4626, now rebased onto state_unit: roll back a partial start instead of panicking openvmm#4872.openvmm: pin virtio-net queue starts that fail without endpoint queues
doc: map the virtio queue starts after restore in the OpenVMM roadmap
vmcore: confirm device starts before the VM runs
openvmm: pin device starts confirmed before the VM runs
doc: map the device start confirmation in the OpenVMM roadmap
vmcore: confirm device starts before the VM runs
openvmm: pin device starts confirmed before the VM runs
doc: map the device start confirmation in the OpenVMM roadmap
ppenna commentedon Nov 7, 2026
Status at
dev8394746fd, with OpenVMM pinned at3cf58a523. #1350 is open for the specification change below.Met
Open, each waiting on work outside this repository or on another issue
release_snapshot_boundarydoes not require a running VM, andquiesce_for_snapshotdoes not require an active boundary.doc: specify that a failed device start fails the VM's start
ppenna commentedon Nov 10, 2026
Status at
devcbc20e07a, with OpenVMM pinned atcd5cd3814. Since the status at8394746fd, #1170, #1148, and #1363 have closed the lifecycle overlap that #1181 notes, and #1451 has completed the restore tests on both virtio transports.Met
8394746fdrecords.Open, each waiting on work outside this repository
Recommendation: adopt each of those pull requests at the chain rebase after it lands upstream, rather than port a draft into the chain, since review may still change its contract.
Outside the microVM's reach: upstream's remote chipset device proxy still panics when its device worker ends during a stop or a reset. Only the TPM uses a remote device worker.
openvmm: pin the PCI transport's restore tests
virtio: keep the progress of queues that a failed start stops
vmcore: confirm device starts before the VM runs
virtio: test PCI re-save before start and missing queue progress
openvmm: pin the device restore and start contract
doc: map the device restore and start contract in the OpenVMM roadmap