Repository navigation
[DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device - #5136
Open
Pedro Henrique Penna (ppenna) wants to merge 6 commits into
Conversation
A device whose start fails must keep the state that it had before the start, so that the stop that follows leaves it as it was (ChangeDeviceState::confirm_start). A virtio-net NIC built for save and restore did not: when the VM's start of a restored or paused NIC failed, the NIC saved a queue-pair lifecycle that disagreed with the queues whose progress the transport kept, and the next restore of that snapshot failed with "virtio-net queue lifecycle does not match transport state" (microsoft/nvx#1232). Two steps of the start changed the lifecycle before the pair started. prepare_queue_start reset the stopped queue count of the pair to zero when its first queue started. And when the pair's second queue could not be created, the first one stayed half open, so the rollback's stop recorded a half-open pair of one queue and returned that queue's progress, although the transport had saved two. Only the failure of the endpoint's queue creation, which drops the pair, avoided the second. Keep the lifecycle that the last stop or restore recorded until a stop records it again: prepare_queue_start only checks the pair's state, and a stop of a started pair recomputes the count, as it did. And drop the half-open first queue, which has not run, when the second cannot be created, as the endpoint failure does, so that the rollback's stop reports no progress and the transport keeps the progress that it saved for both queues. Tests: a restored NIC whose pair fails to start, once because its endpoint cannot create its queues and once because its TX queue cannot be created, reports no progress when the transport stops its queues, saves a lifecycle that its restore accepts with the restored progress, and then starts and transmits the guest's next packet only. Without the first change, the save fails the validation; without the second, the stop of the RX queue reports the progress of a half-open pair. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restored PCI device that uses MSI-X delivers none of its interrupts until the guest writes the vector table again, which a guest resumed mid-flight never does: its queues complete their work, and the guest never learns of it. MsixCapability::restore restored the vector table, but not the MSI interrupt behind each vector, which stays disabled, as a new device's are, so each delivery only latches a pending bit. The MSI capability's restore programs its interrupt; the MSI-X one did not. A save also lost the interrupt that a masked vector had received: the vector keeps it in its MSI interrupt, or on its route's event, until the guest reads the pending bits, and the save recorded only the pending bit. Program each restored vector's delivery as the guest's writes of the table did before the save: enable an unmasked vector of an enabled capability with its restored message, and disable the others. Deliver nothing during the restore, so that its interrupts reach the guest once the VM runs (microsoft/nvx#1232): a vector restored masked keeps its pending bit, and delivers it when the guest unmasks it, and an interrupt that the destination latched before the restore is dropped, as it is not part of the restored state. And have the save record the pending bit that the guest's read of the pending bits would return, counting the interrupt that a masked vector holds, but leave that interrupt where it is, so that the save changes nothing that the guest sees: moving it into the pending bit, as the read does, would have the vector deliver it twice if another interrupt reached its route before the guest unmasked it. Tests: a restored capability delivers nothing during the restore, then delivers an unmasked vector's interrupt to its restored message, and the interrupt that a masked vector received before the save, which the guest never read from the pending bits, when the guest unmasks it; a save records the interrupts that masked vectors hold on a route and in the MSI interrupt, and the saved capability then delivers the route's once, on the route, when the guest unmasks the vector, although another interrupt arrived after the save; a restored virtio-pci device delivers its queue's completion to the vector that the guest programmed before the save. Without the restore change no interrupt arrives, without the save change the masked vector's pending bit is lost, and a save that moves the interrupt into the pending bit delivers it twice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restore must not deliver notifications before the VM starts (microsoft/nvx#1232). The virtio transports drove their interrupt lines while they restored: the MMIO transport lowered its line and raised it again to the restored status, and the PCI transport set its INTx line to the restored status. The interrupt controller that a line reaches is a chipset device that restores in the same pass, before or after the transport, so the restore either reached a controller whose state it was still restoring, where the restore of the controller overwrote it, or it raised the line of a restored controller from the low that the transport had just driven, an edge that can deliver the interrupt again to a processor whose state is not yet restored. In the shared-status mode of the microVM, the restore drove the line low twice. Hold the lines instead. A restore keeps the interrupt status that it restores, and drives no line. The start that follows drives a legacy MMIO line, or a PCI INTx line, to the restored status. A shared-status line, which only pulses as the device raises the status, is not driven at all, since the pulse of the status that the saved word reports was sent before the save, and a restore sends no MSI-X message: the queues' messages begin with the queues, which start with the device. A stop before the start keeps the hold, a reset ends it with the line low, and a later start, as from a pause, drives nothing. The saved state is unchanged. Tests: a restored MMIO device and a restored PCI device with INTx drive no line during their restore, although the restored status asserts it, drive it high when they start, and drive nothing at a second start; a save before the start keeps the status; a reset or a removal before the start leaves the line low. The MMIO interrupt state drives no legacy line while held, and its release drives the line once; a shared-status restore and its release drive the line not at all. The transport tests fail without the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A restore of a microVM virtio-fs reopens the host files of the handles and the inodes that the guest held, so that a file that is gone from the host fails the restore itself, before any vCPU runs, even for a VM restored paused (microsoft/nvx#1232). The filesystem's tests checked that its restore fails then, and the device's tests that the transport reports an invalid device-private state, but no test sent a missing host file through the transport's awaited restore. Add that test: a device whose guest holds a file open is saved through the MMIO transport, the file is removed from the host, and the restore of a new transport fails with the reopen error, before any queue starts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The proxy of a remote chipset device sends a start without waiting for it, as the device-state contract expects, and reports the outcome from confirm_start, which the state units await before the VM runs. A start must not turn the failure of a worker that has ended into a success (microsoft/nvx#1232). The tests checked a confirmation that the remote device fails, and a save or restore after the worker ends, but not a start or an input-gate change after it. Add that test: once the worker has ended, the confirmation of a start and both input-gate changes fail. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
requested review from
a team and
Anatol Belski (weltling)
as code owners
November 17, 2026 21:26
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
5 tasks done
ChangeDeviceState::confirm_start said that a device whose start fails keeps the state that it had before the start, so that stopping it leaves it as it was, and VirtioDevice::start_queue and the Guide's snapshots page said that such a device keeps the private state that it saves as it was before the start, and its restored state. A virtio device whose second queue fails to start can have started its first one, which completes the requests that were pending in it before the VM stops the device again, and the stop keeps that queue's progress, as the virtio test failed_confirmation_keeps_queue_progress requires. So a failed start leaves the state that its stop records, with that progress, and a save of the stopped VM saves that state; the confirmed start keeps the vCPUs from running, but does not undo the device's I/O. Say so in the three places. No code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 17, 2026
A restore completes the restoration of every device while the VM is stopped, and the VM's start confirms each device's start before any vCPU runs (#1232). Three gaps remained at c9755a905: - A virtio-net NIC built for save and restore, as each microVM NIC is, changed its saved queue-pair lifecycle when its start failed, so that a save after the failed start would write a snapshot whose restore fails. No microVM flow saves there today: a microVM refuses a host save, and its guest cannot request a capture while the VM is stopped. - A restored PCI device that uses MSI-X delivered none of its interrupts until the guest wrote its vector table again, and a save lost the interrupt that a masked vector held. A microVM has no PCI device, so only OpenVMM's other VMs met this. - The virtio transports drove their interrupt lines while they restored, so that a restore could reach an interrupt controller that was still restoring, or raise a restored controller's line from the low that the transport had just driven. Pin 8bbe042b1322023e95cce03ef6d4e1ae8e18a6be, the head of microsoft/openvmm#5136, six commits on c9755a905, the head of feature/microvm/dev: - 39f3c9f91 virtio_net: keep the saved queue lifecycle through a failed start - ea2169dc7 pci_core: program the delivery of restored MSI-X vectors - f8c63a6c9 virtio: hold restored interrupt lines until the device starts - 3a6a29318 virtiofs: test a transport restore of a file gone from the host - 5dac44c33 chipset_device_worker: test requests to an ended device worker - 8bbe042b1 vmcore, virtio: say that a failed start keeps its queues' progress A NIC whose start fails now keeps the lifecycle that the restore or the last stop recorded, so that its save matches the queue progress that the transport keeps. A restore programs each MSI-X vector's delivery without delivering, and a save records the interrupt that a masked vector holds in the vector's saved pending bit, without taking it. No virtio transport drives its interrupt line during a restore: the start drives a legacy virtio-mmio or PCI INTx line to the restored status, and the shared-status line of a microVM slot, which only pulses, is not driven at all. The two test commits send a virtio-fs file that is gone from the host through the transport's awaited restore, and check that a remote device whose worker has ended fails its start rather than succeeding. The last commit corrects what the documentation of the start confirmation said that a failed start keeps: the state that its stop records, with the progress of a queue that started and completed requests before another failed, not the state from before the start. No saved state, saved-state format, or version changes, and none is created or bumped. NVX needs no coordinated change: nvx's tests of the control-session vectors, the source excerpts, the contract versions, and the activation bound that it copies from OpenVMM pass against the revision, and so do its tests of the CPU profiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 17, 2026
The NVX pin moves to 8bbe042b1, position 312, 186 commits past the mapped tip. Positions 307 to 312, the six commits of microsoft/openvmm#5136 on c9755a905, the head of feature/microvm/dev, are not yet assigned to PRs. Position 307 keeps the queue-pair lifecycle that a virtio-net NIC built for save and restore saves through a failed start, which the start's contract requires although no microVM flow saves there today. Position 308 has pci_core's MSI-X capability program each restored vector's delivery, and a save record a masked vector's interrupt without taking it; a microVM has no PCI device. Position 309 holds the virtio transports' interrupt lines from a restore until the device starts. Positions 310 and 311 test a virtio-fs transport restore of a file gone from the host and the start of a remote device whose worker has ended. Position 312 corrects what positions 213 and 307 say that a failed start keeps: the state that its stop records, with the progress of a queue that started and completed requests before another failed. Positions 307 and 309 change lines that PR13's position 046 and PR16's position 063 add, but fix no defect of either: position 307 completes the failed start of positions 212 and 213, which no PR maps yet, and position 309 changes the drive of the line at restore that upstream main's transports also have. Replaying them on upstream main, c4419be15, conflicts in the chain's files, apart from position 308's pci_core change and position 309's PCI change, which apply, type-check, and pass their packages' tests there, so they can go upstream on their own. A new section, The device restore contract, records the contract that #1232 asked for: an awaited restore of the stopped VM validates and applies each device's state and reacquires its host resources, without starting a virtio queue or driving a virtio interrupt line; the confirmed start runs the steps that still need a live backend or a running queue, and a failed start keeps the state that its stop records, from which a later start or a save carries on; and a stop releases each queue with its exact progress. It accounts for each activation path, says where the chain departs from the drafts microsoft/openvmm#4713, microsoft/openvmm#4729, and microsoft/openvmm#4855, which it adopts once they land, and how RestoreVM follows the contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs microsoft/nvx#1232.
NVX pins the head of this branch in microsoft/nvx#1738, so this is a prerequisite of that pull request, and it lands by fast-forwarding
feature/microvm/devto that exact commit, not with the merge button, which squashes.What and why
microsoft/nvx#1232 asks that a restore complete every device's restoration while the VM is stopped, that it run no queue and deliver no notification before the VM starts, and that each step that can still fail at the start fail it rather than succeed without the device. Upstream's awaited chipset and device-worker restore (#4799) and the chain's confirmed start already carry most of it; three gaps remained, and two tests were missing.
39f3c9f91). A NIC built for save and restore, as each microVM NIC is, saves how many queues its last stop or restore stopped, and its restore checks that count against the queue progress that the transport saves.prepare_queue_startcleared the count as each pair started, and a pair whose second queue failed to build stayed half open, so the stop that undid a failed start recorded one queue: a save after a failed start, as when the endpoint cannot create its queues, wrote a snapshot whose restore failed with "virtio-net queue lifecycle does not match transport state".prepare_queue_startnow only checks the saved state, and the start drops a pair whose second queue does not build. No microVM flow saves between a failed start and the next start today (a microVM refuses a host save), so this fixes the contract ofconfirm_start: a device whose start fails keeps a state that its stop records consistently, which a save keeps.ea2169dc7).MsixCapability::restorerestored the vector table but never enabled the MSI interrupt behind each vector, so a restored PCI device delivered no MSI-X interrupt until the guest wrote its vector table again. The restore now enables each enabled, unmasked vector at its restored message and disables the others, delivering nothing itself. A save also lost an interrupt that a masked vector held in its MSI interrupt or on its route's event; it now records the pending bit that a read of the pending bits would return, without taking the interrupt, so that the save changes nothing that the guest sees (taking it, as that read does, would deliver it twice if another interrupt reached the route before the guest unmasked the vector). A microVM has no PCI device.f8c63a6c9). The virtio transports drove their interrupt lines while they restored, although the interrupt controller that a line reaches may still be restoring: a line set drops a level that reaches a controller while it restores, and the IOAPIC restores the line levels that it saved. A restore now restores the interrupt status without driving the line; the start drives a legacy virtio-mmio line or a PCI INTx line to the restored status before the queues start, and a reset ends the hold. A shared-status line, which every microVM slot uses, only pulses as the device raises the status, so neither the restore nor the start drives it, and a restore sends no MSI-X message.8bbe042b1).ChangeDeviceState::confirm_start,VirtioDevice::start_queue, and the Guide's snapshots page said that a device whose start fails keeps the state that it had before the start. A virtio device whose second queue fails to start can have started its first one, which completes the requests that were pending in it before the VM stops the device again, and the stop keeps that queue's progress, asfailed_confirmation_keeps_queue_progress_{mmio,pci}require; the confirmation keeps the vCPUs from running, but does not undo a device's I/O. The three now say that a failed start keeps the state that its stop records, which a later start or a save carries on from. Documentation only.3a6a29318,5dac44c33). The awaited restore of a virtio-fs transport fails when a file that the guest held open is gone from the host, before any queue starts; a remote chipset device whose worker has ended fails its start confirmation and its input-gate changes instead of succeeding without the device.No saved-state element, field, format, or version changes.
Validation
At this head,
8bbe042b1, six commits onc9755a905, the head offeature/microvm/dev. The last commit changes only documentation:cargo doc --no-deps -p vmcore -p virtiowithRUSTDOCFLAGS=-D warningsandcargo xtask fmtpass at it, and the results below, from5dac44c33, hold for it:cargo nextest run --profile agent -p virtio_net -p pci_core -p virtio -p virtiofs -p chipset_device_workeron Windows (507, 510, 514, 515, and 516 tests passed, 2 skipped), checked withgit rebase --exec. Each new test fails without its change.cargo clippy --all-targetsandcargo doc --no-depswithRUSTDOCFLAGS=-D warningsfor the five packages on Windows, without warnings from them;cargo xtask fmt, clean.c4419be15,pci_core's change ofea2169dc7applies, andcargo nextest run -p pci_corepasses (113 tests); the PCI transport's change off8c63a6c9applies, andcargo check -p virtio --all-targetsandcargo nextest run -p virtiopass (118 tests).snapshot-core,snapshot-rollback,snapshot-tiers,console-snapshot,filesystem-snapshot,network-snapshot,endpoint-policy-snapshot,scratch-snapshot,smp-snapshot,restore-memory,restore-processors,restore-downtime,lifecycle,vm-lifecycle,managed-lifecycle,virtio-net,interrupt-map,guest-boot,sandbox-lifecycle, andplatform-restore-processors, with 1 and 2 processors and the Alpine guest): all 20 pass on KVM, MSHV, and WHP.Not tested here