Skip to content

[DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device - #5136

Open
Pedro Henrique Penna (ppenna) wants to merge 6 commits into
feature/microvm/devfrom
copilot/nvx-issue-1232-awaited-device-restore
Open

Pedro Henrique Penna (ppenna) wants to merge 6 commits into
feature/microvm/devfrom
copilot/nvx-issue-1232-awaited-device-restore

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Refs microsoft/nvx#1232.

NVX pins the head of this branch in microsoft/nvx#1738, so this is a prerequisite of that pull request, and it lands by fast-forwarding feature/microvm/dev to that exact commit, not with the merge button, which squashes.

What and why

microsoft/nvx#1232 asks that a restore complete every device's restoration while the VM is stopped, that it run no queue and deliver no notification before the VM starts, and that each step that can still fail at the start fail it rather than succeed without the device. Upstream's awaited chipset and device-worker restore (#4799) and the chain's confirmed start already carry most of it; three gaps remained, and two tests were missing.

  • virtio-net keeps its saved lifecycle through a failed start (39f3c9f91). A NIC built for save and restore, as each microVM NIC is, saves how many queues its last stop or restore stopped, and its restore checks that count against the queue progress that the transport saves. prepare_queue_start cleared the count as each pair started, and a pair whose second queue failed to build stayed half open, so the stop that undid a failed start recorded one queue: a save after a failed start, as when the endpoint cannot create its queues, wrote a snapshot whose restore failed with "virtio-net queue lifecycle does not match transport state". prepare_queue_start now only checks the saved state, and the start drops a pair whose second queue does not build. No microVM flow saves between a failed start and the next start today (a microVM refuses a host save), so this fixes the contract of confirm_start: a device whose start fails keeps a state that its stop records consistently, which a save keeps.
  • MSI-X vectors are programmed at restore (ea2169dc7). MsixCapability::restore restored the vector table but never enabled the MSI interrupt behind each vector, so a restored PCI device delivered no MSI-X interrupt until the guest wrote its vector table again. The restore now enables each enabled, unmasked vector at its restored message and disables the others, delivering nothing itself. A save also lost an interrupt that a masked vector held in its MSI interrupt or on its route's event; it now records the pending bit that a read of the pending bits would return, without taking the interrupt, so that the save changes nothing that the guest sees (taking it, as that read does, would deliver it twice if another interrupt reached the route before the guest unmasked the vector). A microVM has no PCI device.
  • Interrupt lines are held until the device starts (f8c63a6c9). The virtio transports drove their interrupt lines while they restored, although the interrupt controller that a line reaches may still be restoring: a line set drops a level that reaches a controller while it restores, and the IOAPIC restores the line levels that it saved. A restore now restores the interrupt status without driving the line; the start drives a legacy virtio-mmio line or a PCI INTx line to the restored status before the queues start, and a reset ends the hold. A shared-status line, which every microVM slot uses, only pulses as the device raises the status, so neither the restore nor the start drives it, and a restore sends no MSI-X message.
  • What a failed start keeps (8bbe042b1). ChangeDeviceState::confirm_start, VirtioDevice::start_queue, and the Guide's snapshots page said that a device whose start fails keeps the state that it had before the start. A virtio device whose second queue fails to start can have started its first one, which completes the requests that were pending in it before the VM stops the device again, and the stop keeps that queue's progress, as failed_confirmation_keeps_queue_progress_{mmio,pci} require; the confirmation keeps the vCPUs from running, but does not undo a device's I/O. The three now say that a failed start keeps the state that its stop records, which a later start or a save carries on from. Documentation only.
  • Tests (3a6a29318, 5dac44c33). The awaited restore of a virtio-fs transport fails when a file that the guest held open is gone from the host, before any queue starts; a remote chipset device whose worker has ended fails its start confirmation and its input-gate changes instead of succeeding without the device.

No saved-state element, field, format, or version changes.

Validation

At this head, 8bbe042b1, six commits on c9755a905, the head of feature/microvm/dev. The last commit changes only documentation: cargo doc --no-deps -p vmcore -p virtio with RUSTDOCFLAGS=-D warnings and cargo xtask fmt pass at it, and the results below, from 5dac44c33, hold for it:

  • Each commit passes cargo nextest run --profile agent -p virtio_net -p pci_core -p virtio -p virtiofs -p chipset_device_worker on Windows (507, 510, 514, 515, and 516 tests passed, 2 skipped), checked with git rebase --exec. Each new test fails without its change.
  • Lints. cargo clippy --all-targets and cargo doc --no-deps with RUSTDOCFLAGS=-D warnings for the five packages on Windows, without warnings from them; cargo xtask fmt, clean.
  • Linux. The same nextest run on an Intel KVM host and an Intel MSHV host, both Azure virtual machines with Linux 6.18: 521 passed, 2 skipped, on each.
  • Upstream main. With c4419be15, pci_core's change of ea2169dc7 applies, and cargo nextest run -p pci_core passes (113 tests); the PCI transport's change of f8c63a6c9 applies, and cargo check -p virtio --all-targets and cargo nextest run -p virtio pass (118 tests).
  • End to end. NVX's restore and lifecycle microVM scenarios with this head built for each backend (snapshot-core, snapshot-rollback, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, endpoint-policy-snapshot, scratch-snapshot, smp-snapshot, restore-memory, restore-processors, restore-downtime, lifecycle, vm-lifecycle, managed-lifecycle, virtio-net, interrupt-map, guest-boot, sandbox-lifecycle, and platform-restore-processors, with 1 and 2 processors and the Alpine guest): all 20 pass on KVM, MSHV, and WHP.

Not tested here

  • OpenVMM's VMM tests, which exercise the PCI devices' save and restore of standard VMs, run in this pull request's CI.

A device whose start fails must keep the state that it had before the
start, so that the stop that follows leaves it as it was
(ChangeDeviceState::confirm_start). A virtio-net NIC built for save and
restore did not: when the VM's start of a restored or paused NIC failed,
the NIC saved a queue-pair lifecycle that disagreed with the queues
whose progress the transport kept, and the next restore of that
snapshot failed with "virtio-net queue lifecycle does not match
transport state" (microsoft/nvx#1232).

Two steps of the start changed the lifecycle before the pair started.
prepare_queue_start reset the stopped queue count of the pair to zero
when its first queue started. And when the pair's second queue could
not be created, the first one stayed half open, so the rollback's stop
recorded a half-open pair of one queue and returned that queue's
progress, although the transport had saved two. Only the failure of the
endpoint's queue creation, which drops the pair, avoided the second.

Keep the lifecycle that the last stop or restore recorded until a stop
records it again: prepare_queue_start only checks the pair's state, and
a stop of a started pair recomputes the count, as it did. And drop the
half-open first queue, which has not run, when the second cannot be
created, as the endpoint failure does, so that the rollback's stop
reports no progress and the transport keeps the progress that it saved
for both queues.

Tests: a restored NIC whose pair fails to start, once because its
endpoint cannot create its queues and once because its TX queue cannot
be created, reports no progress when the transport stops its queues,
saves a lifecycle that its restore accepts with the restored progress,
and then starts and transmits the guest's next packet only. Without the
first change, the save fails the validation; without the second, the
stop of the RX queue reports the progress of a half-open pair.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restored PCI device that uses MSI-X delivers none of its interrupts
until the guest writes the vector table again, which a guest resumed
mid-flight never does: its queues complete their work, and the guest
never learns of it. MsixCapability::restore restored the vector table,
but not the MSI interrupt behind each vector, which stays disabled, as
a new device's are, so each delivery only latches a pending bit. The
MSI capability's restore programs its interrupt; the MSI-X one did not.
A save also lost the interrupt that a masked vector had received: the
vector keeps it in its MSI interrupt, or on its route's event, until
the guest reads the pending bits, and the save recorded only the
pending bit.

Program each restored vector's delivery as the guest's writes of the
table did before the save: enable an unmasked vector of an enabled
capability with its restored message, and disable the others. Deliver
nothing during the restore, so that its interrupts reach the guest once
the VM runs (microsoft/nvx#1232): a vector restored masked keeps its
pending bit, and delivers it when the guest unmasks it, and an
interrupt that the destination latched before the restore is dropped,
as it is not part of the restored state. And have the save record the
pending bit that the guest's read of the pending bits would return,
counting the interrupt that a masked vector holds, but leave that
interrupt where it is, so that the save changes nothing that the guest
sees: moving it into the pending bit, as the read does, would have the
vector deliver it twice if another interrupt reached its route before
the guest unmasked it.

Tests: a restored capability delivers nothing during the restore, then
delivers an unmasked vector's interrupt to its restored message, and
the interrupt that a masked vector received before the save, which the
guest never read from the pending bits, when the guest unmasks it; a
save records the interrupts that masked vectors hold on a route and in
the MSI interrupt, and the saved capability then delivers the route's
once, on the route, when the guest unmasks the vector, although another
interrupt arrived after the save; a restored virtio-pci device delivers
its queue's completion to the vector that the guest programmed before
the save. Without the restore change no interrupt arrives, without the
save change the masked vector's pending bit is lost, and a save that
moves the interrupt into the pending bit delivers it twice.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restore must not deliver notifications before the VM starts
(microsoft/nvx#1232). The virtio transports drove their interrupt lines
while they restored: the MMIO transport lowered its line and raised it
again to the restored status, and the PCI transport set its INTx line to
the restored status. The interrupt controller that a line reaches is a
chipset device that restores in the same pass, before or after the
transport, so the restore either reached a controller whose state it
was still restoring, where the restore of the controller overwrote it,
or it raised the line of a restored controller from the low that the
transport had just driven, an edge that can deliver the interrupt again
to a processor whose state is not yet restored. In the shared-status
mode of the microVM, the restore drove the line low twice.

Hold the lines instead. A restore keeps the interrupt status that it
restores, and drives no line. The start that follows drives a legacy
MMIO line, or a PCI INTx line, to the restored status. A shared-status
line, which only pulses as the device raises the status, is not driven
at all, since the pulse of the status that the saved word reports was
sent before the save, and a restore sends no MSI-X message: the queues'
messages begin with the queues, which start with the device. A stop
before the start keeps the hold, a reset ends it with the line low, and
a later start, as from a pause, drives nothing. The saved state is
unchanged.

Tests: a restored MMIO device and a restored PCI device with INTx drive
no line during their restore, although the restored status asserts it,
drive it high when they start, and drive nothing at a second start; a
save before the start keeps the status; a reset or a removal before the
start leaves the line low. The MMIO interrupt state drives no legacy
line while held, and its release drives the line once; a shared-status
restore and its release drive the line not at all. The transport tests
fail without the change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restore of a microVM virtio-fs reopens the host files of the handles
and the inodes that the guest held, so that a file that is gone from the
host fails the restore itself, before any vCPU runs, even for a VM
restored paused (microsoft/nvx#1232). The filesystem's tests checked
that its restore fails then, and the device's tests that the transport
reports an invalid device-private state, but no test sent a missing
host file through the transport's awaited restore.

Add that test: a device whose guest holds a file open is saved through
the MMIO transport, the file is removed from the host, and the restore
of a new transport fails with the reopen error, before any queue
starts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The proxy of a remote chipset device sends a start without waiting for
it, as the device-state contract expects, and reports the outcome from
confirm_start, which the state units await before the VM runs. A start
must not turn the failure of a worker that has ended into a success
(microsoft/nvx#1232). The tests checked a confirmation that the remote
device fails, and a save or restore after the worker ends, but not a
start or an input-gate change after it.

Add that test: once the worker has ended, the confirmation of a start
and both input-gate changes fail.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code labels Nov 17, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

ChangeDeviceState::confirm_start said that a device whose start fails
keeps the state that it had before the start, so that stopping it
leaves it as it was, and VirtioDevice::start_queue and the Guide's
snapshots page said that such a device keeps the private state that it
saves as it was before the start, and its restored state. A virtio
device whose second queue fails to start can have started its first
one, which completes the requests that were pending in it before the VM
stops the device again, and the stop keeps that queue's progress, as
the virtio test failed_confirmation_keeps_queue_progress requires. So a
failed start leaves the state that its stop records, with that
progress, and a save of the stopped VM saves that state; the confirmed
start keeps the vCPUs from running, but does not undo the device's I/O.

Say so in the three places. No code changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 17, 2026
A restore completes the restoration of every device while the VM is
stopped, and the VM's start confirms each device's start before any
vCPU runs (#1232). Three gaps remained at c9755a905:

- A virtio-net NIC built for save and restore, as each microVM NIC is,
  changed its saved queue-pair lifecycle when its start failed, so that
  a save after the failed start would write a snapshot whose restore
  fails. No microVM flow saves there today: a microVM refuses a host
  save, and its guest cannot request a capture while the VM is stopped.
- A restored PCI device that uses MSI-X delivered none of its interrupts
  until the guest wrote its vector table again, and a save lost the
  interrupt that a masked vector held. A microVM has no PCI device, so
  only OpenVMM's other VMs met this.
- The virtio transports drove their interrupt lines while they
  restored, so that a restore could reach an interrupt controller that
  was still restoring, or raise a restored controller's line from the
  low that the transport had just driven.

Pin 8bbe042b1322023e95cce03ef6d4e1ae8e18a6be, the head of
microsoft/openvmm#5136, six commits on c9755a905, the head of
feature/microvm/dev:

- 39f3c9f91 virtio_net: keep the saved queue lifecycle through a
  failed start
- ea2169dc7 pci_core: program the delivery of restored MSI-X vectors
- f8c63a6c9 virtio: hold restored interrupt lines until the device
  starts
- 3a6a29318 virtiofs: test a transport restore of a file gone from the
  host
- 5dac44c33 chipset_device_worker: test requests to an ended device
  worker
- 8bbe042b1 vmcore, virtio: say that a failed start keeps its queues'
  progress

A NIC whose start fails now keeps the lifecycle that the restore or the
last stop recorded, so that its save matches the queue progress that
the transport keeps. A restore programs each MSI-X vector's delivery
without delivering, and a save records the interrupt that a masked
vector holds in the vector's saved pending bit, without taking it. No
virtio transport drives its interrupt line during a restore: the start
drives a legacy virtio-mmio or PCI INTx line to the restored status,
and the shared-status line of a microVM slot, which only pulses, is not
driven at all. The two test commits send a virtio-fs file that is gone
from the host through the transport's awaited restore, and check that a
remote device whose worker has ended fails its start rather than
succeeding. The last commit corrects what the documentation of the
start confirmation said that a failed start keeps: the state that its
stop records, with the progress of a queue that started and completed
requests before another failed, not the state from before the start.
No saved state, saved-state format, or version changes, and none is
created or bumped.

NVX needs no coordinated change: nvx's tests of the control-session
vectors, the source excerpts, the contract versions, and the activation
bound that it copies from OpenVMM pass against the revision, and so do
its tests of the CPU profiles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 17, 2026
The NVX pin moves to 8bbe042b1, position 312, 186 commits past the
mapped tip. Positions 307 to 312, the six commits of
microsoft/openvmm#5136 on c9755a905, the head of feature/microvm/dev,
are not yet assigned to PRs.

Position 307 keeps the queue-pair lifecycle that a virtio-net NIC built
for save and restore saves through a failed start, which the start's
contract requires although no microVM flow saves there today. Position
308 has pci_core's MSI-X capability program each restored vector's
delivery, and a save record a masked vector's interrupt without taking
it; a microVM has no PCI device. Position 309 holds the virtio
transports' interrupt lines from a restore until the device starts.
Positions 310 and 311 test a virtio-fs transport restore of a file gone
from the host and the start of a remote device whose worker has ended.
Position 312 corrects what positions 213 and 307 say that a failed
start keeps: the state that its stop records, with the progress of a
queue that started and completed requests before another failed.
Positions 307 and 309 change lines that PR13's position 046 and PR16's
position 063 add, but fix no defect of either: position 307 completes
the failed start of positions 212 and 213, which no PR maps yet, and
position 309 changes the drive of the line at restore that upstream
main's transports also have. Replaying them on upstream main,
c4419be15, conflicts in the chain's files, apart from position 308's
pci_core change and position 309's PCI change, which apply, type-check,
and pass their packages' tests there, so they can go upstream on their
own.

A new section, The device restore contract, records the contract that
#1232 asked for: an awaited restore of the stopped VM
validates and applies each device's state and reacquires its host
resources, without starting a virtio queue or driving a virtio
interrupt line; the confirmed start runs the steps that still need a
live backend or a running queue, and a failed start keeps the state
that its stop records, from which a later start or a save carries on;
and a stop releases each queue with its exact progress. It accounts for
each activation path, says where the chain departs from the drafts
microsoft/openvmm#4713, microsoft/openvmm#4729, and
microsoft/openvmm#4855, which it adopts once they land, and how
RestoreVM follows the contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant