Skip to content

openvmm: refactor device restoration onto upstream async save/restore #1232

Description

Objective

Refactor NVX's OpenVMM device restoration onto the asynchronous save/restore interfaces that landed in microsoft/openvmm#4799, instead of using device start to complete restoration and report restore errors.

The decision is to retire the three upstream fallible-start/resume proposals listed below and pursue their replacement here. This does not establish that activation is already infallible: remaining host-backend, queue-start, and worker failures must be accounted for explicitly before removing the existing implementation from NVX's fork.

Background

The current fork validates and stages virtio device-private state during restore, then applies it during activation. For an active restored device, failures such as reopening virtio-fs host handles therefore surface through start/resume.

Upstream now supports awaited, fallible restore through the chipset and remote device-worker boundaries. Use that capability to complete device restoration while the VM remains stopped, separating preparation from activation. Coordinate with microsoft/openvmm#4713, which already proposes parked device-task access, restore-time queue reconstruction, and queue retention across ordinary pause/resume. Its host backend queue creation still occurs during asynchronous activation; it is not proof that every startup failure has disappeared.

Superseded upstream proposals

The following proposals are closed in favor of this tracking issue. Their branches and the current fork implementation are retained:

Pull request Scope Reviewed head
microsoft/openvmm#4619 Fallible state-unit start and partial-start rollback 960826ee06b6d9fb5c2dee0c292c9f039d295e83
microsoft/openvmm#4622 Fallible chipset-device start and worker acknowledgments 24d13d56fb9c5df355d62a13c90ad369f5be8f39
microsoft/openvmm#4623 Fallible VM resume and caller propagation d776c0eb30ba06d42d15189f4b09721577e9892d

These are superseded proposals, not merged fixes. Closing them must not silently discard their error-handling and cleanup requirements.

Work

  • Agree the restore/prepare/activate contract with the authors of virtio_net: preserve queues across pause and restore from saved cursors openvmm#4713 and virtio_net: separate vhost-user ring enable from start and stop openvmm#4729. Reuse their queue-ownership and parked-task design where applicable rather than introducing a competing lifecycle.
  • Move device-private restoration and appropriate host-resource preparation into awaited restore. Cover both PCI and MMIO transports, worker-backed devices, and the virtio-fs, console, and network consumers. Report restoration failures even when the caller requests a restored-but-paused VM.
  • Preserve the stopped-state boundary: restoration must not start live guest queue processing or admit external input. Preserve inactive/not-DRIVER_OK behavior and defer restore-generated notifications until the correct interrupt/start boundary.
  • Audit every remaining fallible activation path, including cold boot, ordinary pause/resume, fresh-snapshot activation, hot-add, worker restart/recovery, and rollback after a failed save. Move fallible preparation earlier or retain/propose the narrowly scoped error handling still required. Do not turn failures into successful fire-and-forget starts.
  • Define cleanup after partial restore, cancellation, reset, and removal before first start. Release prepared resources without abandoning issued I/O or publishing premature removal success.
  • Extract/rebase state_unit: add bounded quiesce transactions with rollback classification openvmm#4625 and vmcore: add device input quiesce and resume openvmm#4626 away from the superseded state_unit: make unit start fallible and roll back partial starts openvmm#4619 implementation as needed. Preserve bounded quiesce, dependency ordering, uncertain-outcome classification, partial-rollback cleanup, and pre-vCPU-stop input gating. Async restore does not replace these guarantees.
  • Preserve the independent exact-inventory validation of state_unit: record the exact unit inventory in saved state openvmm#4624 before device restoration begins. Audit error propagation through launch, restore-ready signaling, REPL, ttrpc, and test callers.
  • Update affected tests and documentation, including the lifecycle and generic virtio persistence sections of the OpenVMM upstream roadmap, to distinguish restore-time failures from residual activation failures and record the superseded PRs.
  • After validation, promote the replacement OpenVMM revision into NVX using the repository promotion workflow: gitlink-only integration commit and a separate roadmap update. Only then remove obsolete fork plumbing.

Coordination

Related work Required integration
microsoft/openvmm#4713, microsoft/openvmm#4729 Retained internal pause versus ownership-releasing queue stop; restore-time cursor reconstruction; backend activation failures.
microsoft/openvmm#4662, microsoft/openvmm#4863 Awaited resource release, stopped-only teardown, and cleanup ownership independent of the requesting future.
microsoft/openvmm#4855 Establish restored interrupt outputs while forwarding is stopped; start targets before replay and producers. Preserve virtio notification handling as well.
microsoft/openvmm#4553, microsoft/openvmm#4676 RestoreVM with and without automatic resume, consistent error reporting, and cleanup of failed restored workers. Reconcile newly added resume callers with whichever activation contract remains.
microsoft/openvmm#4624 Exact state-unit inventory remains independent and must not be dropped.
microsoft/openvmm#4625, microsoft/openvmm#4626 Keep these proposals; adapt their dependencies and rollback implementation rather than closing them.

The source work belongs in the nested OpenVMM repository. The roadmap's PR13 deferred-persistence work and its microVM device consumers are part of the migration, even where they have no standalone upstream PR yet.

Related NVX work:

Acceptance criteria

  • A restored-but-paused VM has completed the required device restoration. Invalid device-private state, missing host resources, and failed worker restoration are reported before successful restore/readiness, without requiring a subsequent resume.
  • Restoring does not run live queues or deliver notifications/input prematurely. Tests cover active and inactive devices, restore-generated notifications, and re-save/reset/removal before first start.
  • Repeated pause/resume works without invoking snapshot restore, losing queue progress, duplicating TX, or leaking resources. Retained-resource and ownership-releasing stop paths remain distinct.
  • Fault-injection tests cover partial restoration, backend/queue activation failures that remain, worker/channel failure, and interrupted cleanup. A partially prepared or activated VM never reports success; resources and already-started work are cleaned up according to the documented contract.
  • Failed-save recovery retains deadlines and dependency ordering. Uncertain transitions cannot be bypassed by ordinary resume, and later lifecycle requests do not panic on invalid unit state.
  • Package-scoped OpenVMM checks and affected NVX tests pass. Validate the affected lifecycle scenarios on KVM, MSHV, and WHP through available hosts/CI, explicitly recording unavailable coverage rather than assuming it passed.
  • The replacement is integrated and the roadmap reflects the final architecture before obsolete fork implementations are removed.

Activity

  1. ppenna commented on Nov 5, 2026

    @ppenna
    ContributorAuthor

    Decision recorded from #1260, implemented by #1288 and microsoft/openvmm#4921: the snapshot quiesce has no deadline of its own any more, so this issue's bounded-quiesce requirements change.

  2. ppenna commented on Nov 7, 2026

    @ppenna
    ContributorAuthor

    Status at dev 8394746fd, with OpenVMM pinned at 3cf58a523. #1350 is open for the specification change below.

    Met

    Open, each waiting on work outside this repository or on another issue

  3. ppenna commented on Nov 10, 2026

    @ppenna
    ContributorAuthor

    Status at dev cbc20e07a, with OpenVMM pinned at cd5cd3814. Since the status at 8394746fd, #1170, #1148, and #1363 have closed the lifecycle overlap that #1181 notes, and #1451 has completed the restore tests on both virtio transports.

    Met

    Open, each waiting on work outside this repository

    Recommendation: adopt each of those pull requests at the chain rebase after it lands upstream, rather than port a draft into the chain, since review may still change its contract.

    Outside the microVM's reach: upstream's remote chipset device proxy still panics when its device worker ends during a stop or a reset. Only the TPM uses a remote device worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: openvmmInitial title-keyword triage suggestion

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions