Skip to content

openvmm: pin the device restore and start contract - #1738

Open
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
devfrom
agents/issue-1232-awaited-device-restore
Open

Pedro Henrique Penna (ppenna) wants to merge 3 commits into
devfrom
agents/issue-1232-awaited-device-restore

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1232

What does this pull request change?

#1232 asks that a restore complete every device's restoration while the VM is stopped, report a failure without needing a resume, run no queue and deliver no notification before the VM starts, and leave no step that can fail at the start to succeed without the device. Upstream's awaited chipset and device-worker restore (microsoft/openvmm#4799, in the fork base) and the chain's confirmed start already met most of it, as the issue's last status records. Checking the rest against the pin found three gaps, which microsoft/openvmm#5136 closes, and a contract that its documentation stated too strongly, which it corrects. This pull request pins it, records the restore contract that #1232 asked to decide in the OpenVMM roadmap, and specifies a device's restore in the restore protocol. Start with the first commit's message and microsoft/openvmm#5136's description.

  • openvmm: pin the device restore and start contract moves the openvmm gitlink, and nothing else, from c9755a905, the head of feature/microvm/dev, to 8bbe042b1, the head of [DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device openvmm#5136, six commits on it:
    • 39f3c9f91: a virtio-net NIC built for save and restore, as each microVM NIC is, keeps the queue-pair lifecycle that it saves through a failed start. It cleared the count as each pair started and left a half-open pair when its second queue failed to build, so a save after the failed start wrote a snapshot whose restore failed with "virtio-net queue lifecycle does not match transport state". No microVM flow saves there today (a microVM refuses a host save), so this fixes the contract that a device whose start fails stays as it was.
    • ea2169dc7: a restored PCI device's MSI-X vectors deliver again. The restore programmed the vector table but not the MSI interrupt behind each vector, so the device delivered nothing until the guest rewrote the table; a save also lost an interrupt that a masked vector held, and now records it without taking it. A microVM has no PCI device.
    • f8c63a6c9: the virtio transports drive no interrupt line during a restore, when the interrupt controller that the line reaches can still be restoring. A legacy virtio-mmio line or a PCI INTx line goes to the restored status as the device starts; the shared-status line of a microVM slot, which only pulses, is driven by neither.
    • 3a6a29318 and 5dac44c33: tests that the awaited restore of a virtio-fs transport fails when a file that the guest held open is gone from the host, and that a remote chipset device whose worker has ended fails its start instead of succeeding without the device.
    • 8bbe042b1: ChangeDeviceState::confirm_start, VirtioDevice::start_queue, and the Guide said that a device whose start fails keeps the state that it had before the start. A queue that started before another failed can complete the requests that were pending in it, and the stop keeps its progress, so they now say that a failed start keeps the state that its stop records, which a later start or a save carries on from. No code changes.
  • doc: map the device restore and start contract in the OpenVMM roadmap maps positions 307-312 and adds The device restore contract, the final architecture that openvmm: refactor device restoration onto upstream async save/restore #1232 asked to record.
  • doc: specify what a device's restore does before the VM starts updates step 14 and the management RPC's paragraph of 5.5 Restore Protocol.

No version or format changed: no snapshot manifest version or magic, time ABI, restore packet, machine contract, template compatibility key, saved-state schema, or element or field of any saved state.

The contract

#1232 asked to agree the restore, prepare, and activate contract with the authors of microsoft/openvmm#4713 and microsoft/openvmm#4729, both drafts. I decided it, reusing their designs where they apply, and the roadmap records it:

  • Restore is awaited while the VM is stopped, after the exact unit inventory check (state_unit: record the exact unit inventory in saved state openvmm#4624). It validates and applies each device's state and reacquires the host resources that the state refers to, as virtio-fs reopens its handles, so the contract has no separate prepare step. It starts no queue, admits no host input, and no virtio device drives its interrupt line.
  • Start is confirmed before any vCPU runs. Its steps that can still fail, which need a live backend or a running queue (virtio-net's endpoint queues, the construction of each queue, virtio-console's check of its restored partial transmission, and a remote device worker's answer), fail the resume, which stops the devices that started again. The devices keep the state that this stop records, including the requests that a queue that started completed before another failed, since the confirmation keeps the vCPUs from running but does not undo a device's I/O, and a later resume or a save carries on from that state.
  • Stop releases each queue with its exact progress.

Left to upstream, adopted at the rebase after each lands: the retained pause of #4713 for NICs without a static identity (microVM NICs keep their queue progress through the stop), the independent vhost-user ring enable of #4729, and the replay of every line set's level of microsoft/openvmm#4855. No restore-at-start plumbing remained to remove: the v4.0 reorganization had retired it. The roadmap also accounts for each activation path: cold boot, an ordinary resume, the resume after a capture, the restart after a failed save, hot-add, and worker restart, which a microVM refuses.

Left out

  • Upstream's remote device proxy still panics when its worker ends during a stop, a reset, a poll, or a teardown. Only the TPM uses a remote device worker, and no microVM has one, so the chain leaves it to upstream.
  • ea2169dc7's pci_core change and the PCI part of f8c63a6c9 apply to upstream main and pass their packages' tests there, so they can go upstream on their own; that is not part of this change.

Why is this change needed?

A restore that leaves part of a device's restoration to the start reports a failure only when the VM resumes, and a restored but paused VM looks healthy until then; a restore that drives interrupts before the VM starts can lose an interrupt or deliver one twice. The three gaps above were what remained of that after #1238, #1308, #1316, and #1451.

Alternatives that I rejected:

  • #4713's reconstruction of virtio-net's queues at restore in a parked device task. It still creates the host backend's queues at activation, so it would not meet the issue alone, and it would add a retained-pause path that microVM NICs do not need. The chain validates queue progress at restore and builds the queues at the confirmed start instead.
  • Moving a masked vector's interrupt into its pending bit at save, as a guest's read of the pending bits does. It changes the saved VM: an interrupt that reaches the vector's route before the guest unmasks it is then delivered twice. The save now records the bit without taking the interrupt.
  • Driving every level at the start through a replay of the line sets, as #4855 proposes. It is a draft that changes every interrupt producer; holding the virtio transports' lines until their device starts closes the gap that this change found.

How was this change validated?

Checks and results:

OpenVMM, at 8bbe042b1 (details in microsoft/openvmm#5136); its last commit changes only documentation, so the runs at 5dac44c33 below hold for it, and cargo doc --no-deps -p vmcore -p virtio with RUSTDOCFLAGS=-D warnings and cargo xtask fmt pass at 8bbe042b1:

  • git rebase --exec "cargo nextest run --profile agent --no-fail-fast -p virtio_net -p pci_core -p virtio -p virtiofs -p chipset_device_worker" c9755a905 on Windows: every commit passes (507, 510, 514, 515, and 516 passed, 2 skipped). Each new test fails without its change.
  • cargo clippy --all-targets and cargo doc --no-deps with RUSTDOCFLAGS=-D warnings for those packages, without warnings from them; cargo xtask fmt: clean.
  • The same nextest run on the KVM and MSHV hosts: 521 passed, 2 skipped, on each.
  • With upstream main, c4419be15: cargo nextest run -p pci_core with ea2169dc7's change, 113 passed; cargo check -p virtio --all-targets and cargo nextest run -p virtio with the PCI part of f8c63a6c9, 118 passed.

NVX, at this branch's head:

  • python scripts/nvx.py verify: consistent.
  • cargo test --locked --manifest-path nvx/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored: 4 passed; --test openvmm_cpu_profiles -- --ignored: 1 passed. No vector, excerpt, version, or CPU profile that nvx copies changed.
  • python scripts/openvmm_pin.py check-branch --topic, check-roadmap, and source 8bbe042b1 --pr 5136 (good signature): passed. python scripts/check_markdown_links.py, python scripts/check_test_layout.py, and python scripts/check_test_changes.py --base origin/dev (no change to an existing test): passed.
  • python scripts/local_gates.py --base origin/dev --run at this change's previous base: nvx-cli (2302 and 84 tests) and the gates above passed; openvmm-ci waits for [DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device openvmm#5136's CI.
  • At the previous head, which pinned 5dac44c33, python3 scripts/nvx.py test-microvm --backend BACKEND --processors 1 2 with --scenario for snapshot-core, snapshot-rollback, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, endpoint-policy-snapshot, scratch-snapshot, smp-snapshot, restore-memory, restore-processors, restore-downtime, lifecycle, vm-lifecycle, managed-lifecycle, virtio-net, interrupt-map, guest-boot, sandbox-lifecycle, and platform-restore-processors, with this branch's head and the OpenVMM built from its pin: all 20 passed on KVM, MSHV, and WHP.

Backend, host OS, host type, and guest:

  • KVM: Azure Linux 3.0 with Linux 6.18.57, virtual machine, Intel Xeon Platinum 8573C, Alpine.
  • MSHV: Azure Linux 3.0 with Linux 6.18.34 for MSHV, virtual machine, Intel Xeon Platinum 8573C, Alpine.
  • WHP: Windows Server 2025 Datacenter Azure Edition, virtual machine, Intel Xeon Platinum 8370C, Alpine.

Left to CI: the rest of the microVM suite, the Ubuntu and Azure Linux guests, OpenVMM's VMM tests (in microsoft/openvmm#5136's CI), and the benchmarks.

Checklist

  • Tests cover the new or fixed behavior, in the layout that
    python3 scripts/check_test_layout.py checks.
  • python3 scripts/check_test_changes.py --base origin/dev lists no change
    to an existing test, or a Changes to existing tests section below
    explains each one.
  • The documentation that the change affects is updated, including
    .github/copilot-instructions.md and the skills when a convention or
    workflow changes.
  • A change to the public API of nvx updates nvx-rs/public-api/ with
    python3 scripts/public_api.py --update.
  • Snapshots, protocols, and on-disk formats stay compatible, or the
    description says how the change versions or migrates them.

OpenVMM pin

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The roadmap incorrectly promises that a failed start preserves the original restored state, although queues may complete work before the failure and retain that progress after rollback.

1 open issue
What changed in this PR

This PR advances NVX’s OpenVMM pin to complete the device restore/start contract tracked by #1232 and documents the boundary between stopped-state restoration and confirmed activation.

Changes:

  • Pins five OpenVMM commits covering failed-start queue state, restored interrupt delivery, and restore/start failure tests.
  • Maps positions 307–311 and records the restore, start, and stop contract in the upstream roadmap.
  • Clarifies that device restoration failures surface before resume, without starting queues or driving interrupt lines.
File Description
doc/​openvmm-upstream-roadmap.md Maps the new pin and commits, documents lifecycle guarantees, and records remaining upstream work.
doc/​design/​5.5-restore-protocol.md Clarifies stopped-device restoration and failure reporting for paused VMs.

Review effort: Balanced

Comment thread doc/openvmm-upstream-roadmap.md Resolved
A restore completes the restoration of every device while the VM is
stopped, and the VM's start confirms each device's start before any
vCPU runs (#1232). Three gaps remained at c9755a905:

- A virtio-net NIC built for save and restore, as each microVM NIC is,
  changed its saved queue-pair lifecycle when its start failed, so that
  a save after the failed start would write a snapshot whose restore
  fails. No microVM flow saves there today: a microVM refuses a host
  save, and its guest cannot request a capture while the VM is stopped.
- A restored PCI device that uses MSI-X delivered none of its interrupts
  until the guest wrote its vector table again, and a save lost the
  interrupt that a masked vector held. A microVM has no PCI device, so
  only OpenVMM's other VMs met this.
- The virtio transports drove their interrupt lines while they
  restored, so that a restore could reach an interrupt controller that
  was still restoring, or raise a restored controller's line from the
  low that the transport had just driven.

Pin 8bbe042b1322023e95cce03ef6d4e1ae8e18a6be, the head of
microsoft/openvmm#5136, six commits on c9755a905, the head of
feature/microvm/dev:

- 39f3c9f91 virtio_net: keep the saved queue lifecycle through a
  failed start
- ea2169dc7 pci_core: program the delivery of restored MSI-X vectors
- f8c63a6c9 virtio: hold restored interrupt lines until the device
  starts
- 3a6a29318 virtiofs: test a transport restore of a file gone from the
  host
- 5dac44c33 chipset_device_worker: test requests to an ended device
  worker
- 8bbe042b1 vmcore, virtio: say that a failed start keeps its queues'
  progress

A NIC whose start fails now keeps the lifecycle that the restore or the
last stop recorded, so that its save matches the queue progress that
the transport keeps. A restore programs each MSI-X vector's delivery
without delivering, and a save records the interrupt that a masked
vector holds in the vector's saved pending bit, without taking it. No
virtio transport drives its interrupt line during a restore: the start
drives a legacy virtio-mmio or PCI INTx line to the restored status,
and the shared-status line of a microVM slot, which only pulses, is not
driven at all. The two test commits send a virtio-fs file that is gone
from the host through the transport's awaited restore, and check that a
remote device whose worker has ended fails its start rather than
succeeding. The last commit corrects what the documentation of the
start confirmation said that a failed start keeps: the state that its
stop records, with the progress of a queue that started and completed
requests before another failed, not the state from before the start.
No saved state, saved-state format, or version changes, and none is
created or bumped.

NVX needs no coordinated change: nvx's tests of the control-session
vectors, the source excerpts, the contract versions, and the activation
bound that it copies from OpenVMM pass against the revision, and so do
its tests of the CPU profiles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 8bbe042b1, position 312, 186 commits past the
mapped tip. Positions 307 to 312, the six commits of
microsoft/openvmm#5136 on c9755a905, the head of feature/microvm/dev,
are not yet assigned to PRs.

Position 307 keeps the queue-pair lifecycle that a virtio-net NIC built
for save and restore saves through a failed start, which the start's
contract requires although no microVM flow saves there today. Position
308 has pci_core's MSI-X capability program each restored vector's
delivery, and a save record a masked vector's interrupt without taking
it; a microVM has no PCI device. Position 309 holds the virtio
transports' interrupt lines from a restore until the device starts.
Positions 310 and 311 test a virtio-fs transport restore of a file gone
from the host and the start of a remote device whose worker has ended.
Position 312 corrects what positions 213 and 307 say that a failed
start keeps: the state that its stop records, with the progress of a
queue that started and completed requests before another failed.
Positions 307 and 309 change lines that PR13's position 046 and PR16's
position 063 add, but fix no defect of either: position 307 completes
the failed start of positions 212 and 213, which no PR maps yet, and
position 309 changes the drive of the line at restore that upstream
main's transports also have. Replaying them on upstream main,
c4419be15, conflicts in the chain's files, apart from position 308's
pci_core change and position 309's PCI change, which apply, type-check,
and pass their packages' tests there, so they can go upstream on their
own.

A new section, The device restore contract, records the contract that
#1232 asked for: an awaited restore of the stopped VM
validates and applies each device's state and reacquires its host
resources, without starting a virtio queue or driving a virtio
interrupt line; the confirmed start runs the steps that still need a
live backend or a running queue, and a failed start keeps the state
that its stop records, from which a later start or a save carries on;
and a stop releases each queue with its exact progress. It accounts for
each activation path, says where the chain departs from the drafts
microsoft/openvmm#4713, microsoft/openvmm#4729, and
microsoft/openvmm#4855, which it adopts once they land, and how
RestoreVM follows the contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The restore protocol said that every state unit restores while the VM
is stopped, but not what a device's restore does. Since the OpenVMM pin
of the device restore and start contract, a device's restore validates
its saved state and reacquires the host resources that the state refers
to, such as the files that a virtio-fs device holds open, so that a
device that cannot restore fails the restore itself. A virtio device's
restore starts no queue and drives no interrupt line: the line of a
microVM's virtio devices pulses only as a device raises its interrupt
status, and the pulse for a restored status was sent before the
capture.

Say so in step 14, and say in the management RPC's paragraph that a
device that cannot restore fails the request itself, although the
restored VM stays paused.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-1232-awaited-device-restore branch from 1c712fe to 9584b73 Compare November 17, 2026 21:38
Copilot AI balanced review requested due to automatic review settings November 17, 2026 21:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the documentation updates clarifying OpenVMM’s device restore and start contract.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: refactor device restoration onto upstream async save/restore

2 participants