Repository navigation
openvmm: pin the device restore and start contract - #1738
Open
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Open
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The roadmap incorrectly promises that a failed start preserves the original restored state, although queues may complete work before the failure and retain that progress after rollback.
1 open issue
What changed in this PR
This PR advances NVX’s OpenVMM pin to complete the device restore/start contract tracked by #1232 and documents the boundary between stopped-state restoration and confirmed activation.
Changes:
- Pins five OpenVMM commits covering failed-start queue state, restored interrupt delivery, and restore/start failure tests.
- Maps positions 307–311 and records the restore, start, and stop contract in the upstream roadmap.
- Clarifies that device restoration failures surface before resume, without starting queues or driving interrupt lines.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Maps the new pin and commits, documents lifecycle guarantees, and records remaining upstream work. |
doc/design/5.5-restore-protocol.md |
Clarifies stopped-device restoration and failure reporting for paused VMs. |
Review effort: Balanced
A restore completes the restoration of every device while the VM is stopped, and the VM's start confirms each device's start before any vCPU runs (#1232). Three gaps remained at c9755a905: - A virtio-net NIC built for save and restore, as each microVM NIC is, changed its saved queue-pair lifecycle when its start failed, so that a save after the failed start would write a snapshot whose restore fails. No microVM flow saves there today: a microVM refuses a host save, and its guest cannot request a capture while the VM is stopped. - A restored PCI device that uses MSI-X delivered none of its interrupts until the guest wrote its vector table again, and a save lost the interrupt that a masked vector held. A microVM has no PCI device, so only OpenVMM's other VMs met this. - The virtio transports drove their interrupt lines while they restored, so that a restore could reach an interrupt controller that was still restoring, or raise a restored controller's line from the low that the transport had just driven. Pin 8bbe042b1322023e95cce03ef6d4e1ae8e18a6be, the head of microsoft/openvmm#5136, six commits on c9755a905, the head of feature/microvm/dev: - 39f3c9f91 virtio_net: keep the saved queue lifecycle through a failed start - ea2169dc7 pci_core: program the delivery of restored MSI-X vectors - f8c63a6c9 virtio: hold restored interrupt lines until the device starts - 3a6a29318 virtiofs: test a transport restore of a file gone from the host - 5dac44c33 chipset_device_worker: test requests to an ended device worker - 8bbe042b1 vmcore, virtio: say that a failed start keeps its queues' progress A NIC whose start fails now keeps the lifecycle that the restore or the last stop recorded, so that its save matches the queue progress that the transport keeps. A restore programs each MSI-X vector's delivery without delivering, and a save records the interrupt that a masked vector holds in the vector's saved pending bit, without taking it. No virtio transport drives its interrupt line during a restore: the start drives a legacy virtio-mmio or PCI INTx line to the restored status, and the shared-status line of a microVM slot, which only pulses, is not driven at all. The two test commits send a virtio-fs file that is gone from the host through the transport's awaited restore, and check that a remote device whose worker has ended fails its start rather than succeeding. The last commit corrects what the documentation of the start confirmation said that a failed start keeps: the state that its stop records, with the progress of a queue that started and completed requests before another failed, not the state from before the start. No saved state, saved-state format, or version changes, and none is created or bumped. NVX needs no coordinated change: nvx's tests of the control-session vectors, the source excerpts, the contract versions, and the activation bound that it copies from OpenVMM pass against the revision, and so do its tests of the CPU profiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 8bbe042b1, position 312, 186 commits past the mapped tip. Positions 307 to 312, the six commits of microsoft/openvmm#5136 on c9755a905, the head of feature/microvm/dev, are not yet assigned to PRs. Position 307 keeps the queue-pair lifecycle that a virtio-net NIC built for save and restore saves through a failed start, which the start's contract requires although no microVM flow saves there today. Position 308 has pci_core's MSI-X capability program each restored vector's delivery, and a save record a masked vector's interrupt without taking it; a microVM has no PCI device. Position 309 holds the virtio transports' interrupt lines from a restore until the device starts. Positions 310 and 311 test a virtio-fs transport restore of a file gone from the host and the start of a remote device whose worker has ended. Position 312 corrects what positions 213 and 307 say that a failed start keeps: the state that its stop records, with the progress of a queue that started and completed requests before another failed. Positions 307 and 309 change lines that PR13's position 046 and PR16's position 063 add, but fix no defect of either: position 307 completes the failed start of positions 212 and 213, which no PR maps yet, and position 309 changes the drive of the line at restore that upstream main's transports also have. Replaying them on upstream main, c4419be15, conflicts in the chain's files, apart from position 308's pci_core change and position 309's PCI change, which apply, type-check, and pass their packages' tests there, so they can go upstream on their own. A new section, The device restore contract, records the contract that #1232 asked for: an awaited restore of the stopped VM validates and applies each device's state and reacquires its host resources, without starting a virtio queue or driving a virtio interrupt line; the confirmed start runs the steps that still need a live backend or a running queue, and a failed start keeps the state that its stop records, from which a later start or a save carries on; and a stop releases each queue with its exact progress. It accounts for each activation path, says where the chain departs from the drafts microsoft/openvmm#4713, microsoft/openvmm#4729, and microsoft/openvmm#4855, which it adopts once they land, and how RestoreVM follows the contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The restore protocol said that every state unit restores while the VM is stopped, but not what a device's restore does. Since the OpenVMM pin of the device restore and start contract, a device's restore validates its saved state and reacquires the host resources that the state refers to, such as the files that a virtio-fs device holds open, so that a device that cannot restore fails the restore itself. A virtio device's restore starts no queue and drives no interrupt line: the line of a microVM's virtio devices pulses only as a device raises its interrupt status, and the pulse for a restored status was sent before the capture. Say so in step 14, and say in the management RPC's paragraph that a device that cannot restore fails the request itself, although the restored VM stays paused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/issue-1232-awaited-device-restore
branch
from
November 17, 2026 21:38
1c712fe to
9584b73
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #1232
What does this pull request change?
#1232 asks that a restore complete every device's restoration while the VM is stopped, report a failure without needing a resume, run no queue and deliver no notification before the VM starts, and leave no step that can fail at the start to succeed without the device. Upstream's awaited chipset and device-worker restore (microsoft/openvmm#4799, in the fork base) and the chain's confirmed start already met most of it, as the issue's last status records. Checking the rest against the pin found three gaps, which microsoft/openvmm#5136 closes, and a contract that its documentation stated too strongly, which it corrects. This pull request pins it, records the restore contract that #1232 asked to decide in the OpenVMM roadmap, and specifies a device's restore in the restore protocol. Start with the first commit's message and microsoft/openvmm#5136's description.
openvmm: pin the device restore and start contractmoves theopenvmmgitlink, and nothing else, fromc9755a905, the head offeature/microvm/dev, to8bbe042b1, the head of [DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device openvmm#5136, six commits on it:39f3c9f91: a virtio-net NIC built for save and restore, as each microVM NIC is, keeps the queue-pair lifecycle that it saves through a failed start. It cleared the count as each pair started and left a half-open pair when its second queue failed to build, so a save after the failed start wrote a snapshot whose restore failed with "virtio-net queue lifecycle does not match transport state". No microVM flow saves there today (a microVM refuses a host save), so this fixes the contract that a device whose start fails stays as it was.ea2169dc7: a restored PCI device's MSI-X vectors deliver again. The restore programmed the vector table but not the MSI interrupt behind each vector, so the device delivered nothing until the guest rewrote the table; a save also lost an interrupt that a masked vector held, and now records it without taking it. A microVM has no PCI device.f8c63a6c9: the virtio transports drive no interrupt line during a restore, when the interrupt controller that the line reaches can still be restoring. A legacy virtio-mmio line or a PCI INTx line goes to the restored status as the device starts; the shared-status line of a microVM slot, which only pulses, is driven by neither.3a6a29318and5dac44c33: tests that the awaited restore of a virtio-fs transport fails when a file that the guest held open is gone from the host, and that a remote chipset device whose worker has ended fails its start instead of succeeding without the device.8bbe042b1:ChangeDeviceState::confirm_start,VirtioDevice::start_queue, and the Guide said that a device whose start fails keeps the state that it had before the start. A queue that started before another failed can complete the requests that were pending in it, and the stop keeps its progress, so they now say that a failed start keeps the state that its stop records, which a later start or a save carries on from. No code changes.doc: map the device restore and start contract in the OpenVMM roadmapmaps positions 307-312 and adds The device restore contract, the final architecture that openvmm: refactor device restoration onto upstream async save/restore #1232 asked to record.doc: specify what a device's restore does before the VM startsupdates step 14 and the management RPC's paragraph of 5.5 Restore Protocol.No version or format changed: no snapshot manifest version or magic, time ABI, restore packet, machine contract, template compatibility key, saved-state schema, or element or field of any saved state.
The contract
#1232 asked to agree the restore, prepare, and activate contract with the authors of microsoft/openvmm#4713 and microsoft/openvmm#4729, both drafts. I decided it, reusing their designs where they apply, and the roadmap records it:
Left to upstream, adopted at the rebase after each lands: the retained pause of #4713 for NICs without a static identity (microVM NICs keep their queue progress through the stop), the independent vhost-user ring enable of #4729, and the replay of every line set's level of microsoft/openvmm#4855. No restore-at-start plumbing remained to remove: the v4.0 reorganization had retired it. The roadmap also accounts for each activation path: cold boot, an ordinary resume, the resume after a capture, the restart after a failed save, hot-add, and worker restart, which a microVM refuses.
Left out
ea2169dc7'spci_corechange and the PCI part off8c63a6c9apply to upstream main and pass their packages' tests there, so they can go upstream on their own; that is not part of this change.Why is this change needed?
A restore that leaves part of a device's restoration to the start reports a failure only when the VM resumes, and a restored but paused VM looks healthy until then; a restore that drives interrupts before the VM starts can lose an interrupt or deliver one twice. The three gaps above were what remained of that after #1238, #1308, #1316, and #1451.
Alternatives that I rejected:
How was this change validated?
Checks and results:
OpenVMM, at
8bbe042b1(details in microsoft/openvmm#5136); its last commit changes only documentation, so the runs at5dac44c33below hold for it, andcargo doc --no-deps -p vmcore -p virtiowithRUSTDOCFLAGS=-D warningsandcargo xtask fmtpass at8bbe042b1:git rebase --exec "cargo nextest run --profile agent --no-fail-fast -p virtio_net -p pci_core -p virtio -p virtiofs -p chipset_device_worker" c9755a905on Windows: every commit passes (507, 510, 514, 515, and 516 passed, 2 skipped). Each new test fails without its change.cargo clippy --all-targetsandcargo doc --no-depswithRUSTDOCFLAGS=-D warningsfor those packages, without warnings from them;cargo xtask fmt: clean.c4419be15:cargo nextest run -p pci_corewithea2169dc7's change, 113 passed;cargo check -p virtio --all-targetsandcargo nextest run -p virtiowith the PCI part off8c63a6c9, 118 passed.NVX, at this branch's head:
python scripts/nvx.py verify: consistent.cargo test --locked --manifest-path nvx/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored: 4 passed;--test openvmm_cpu_profiles -- --ignored: 1 passed. No vector, excerpt, version, or CPU profile thatnvxcopies changed.python scripts/openvmm_pin.py check-branch --topic,check-roadmap, andsource 8bbe042b1 --pr 5136(good signature): passed.python scripts/check_markdown_links.py,python scripts/check_test_layout.py, andpython scripts/check_test_changes.py --base origin/dev(no change to an existing test): passed.python scripts/local_gates.py --base origin/dev --runat this change's previous base:nvx-cli(2302 and 84 tests) and the gates above passed;openvmm-ciwaits for [DO NOT REVIEW] virtio, pci_core: complete device restoration without starting or interrupting the device openvmm#5136's CI.5dac44c33,python3 scripts/nvx.py test-microvm --backend BACKEND --processors 1 2with--scenarioforsnapshot-core,snapshot-rollback,snapshot-tiers,console-snapshot,filesystem-snapshot,network-snapshot,endpoint-policy-snapshot,scratch-snapshot,smp-snapshot,restore-memory,restore-processors,restore-downtime,lifecycle,vm-lifecycle,managed-lifecycle,virtio-net,interrupt-map,guest-boot,sandbox-lifecycle, andplatform-restore-processors, with this branch's head and the OpenVMM built from its pin: all 20 passed on KVM, MSHV, and WHP.Backend, host OS, host type, and guest:
Left to CI: the rest of the microVM suite, the Ubuntu and Azure Linux guests, OpenVMM's VMM tests (in microsoft/openvmm#5136's CI), and the benchmarks.
Checklist
python3 scripts/check_test_layout.pychecks.python3 scripts/check_test_changes.py --base origin/devlists no changeto an existing test, or a Changes to existing tests section below
explains each one.
.github/copilot-instructions.mdand the skills when a convention orworkflow changes.
nvxupdatesnvx-rs/public-api/withpython3 scripts/public_api.py --update.description says how the change versions or migrates them.
OpenVMM pin
8bbe042b1322023e95cce03ef6d4e1ae8e18a6be, with a good signature (python scripts/openvmm_pin.py source).c9755a905, the head offeature/microvm/dev, where [DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots openvmm#5134 (OpenVMM/microVM umbrella: processor state across snapshots #1182) merged; no other prerequisite.feature/microvm/devfromc9755a905to8bbe042b1, not with the merge button, which squashes. This pull request merges next.