From 347ef7261fedd5d602ec664b89cf1dbaef0d0c5d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Mon, 9 Nov 2026 04:09:18 -0800 Subject: [PATCH 1/3] openvmm: pin input gating that keeps devices open to the vCPUs A guest's snapshot request could crash a multi-vCPU microVM, or lose the device accesses of its other vCPUs (#1138). The snapshot boundary gates host input before it stops the vCPUs, so they run while each device's state unit changes the device's gate, and the unit held the device's mutex closed for the change, until the end of its wait. A port or MMIO access that another vCPU made then was dropped, a read returning all ones; an interrupt from another device was lost; and an EOI or a PCI configuration access panicked the VM worker with "lock should not be closed". Pin 46bfe02ed65697e858f66ce55eb60831b3ac3461, the head of microsoft/openvmm#5005, one commit on cba21d109, the head of microsoft/openvmm#5004, on which it stacks, which follows ac3dc27a7, the head of microsoft/openvmm#5003, and b4028c1aa, the head of feature/microvm/dev: - 46bfe02ed vmotherboard: gate a device's input without closing it to the vCPUs The revision splits a device's gate change into a synchronous part, which the unit makes under the lock that the vCPUs' accesses take, and a wait that does not borrow the device, which the unit awaits without it. The virtio transports and the remote device proxy send their gate's request in the synchronous part, in the order of the guest's accesses. Evidence: a microVM guest with 2 and 4 vCPUs requested 2000 snapshots, each of which gated host input and then failed its preflight, since its destination existed, while its other vCPUs read the magic value of the virtio-fs device's MMIO window, and with 4 vCPUs the console's too, in a loop. With OpenVMM at d6addf8c1, 145 to 2358 of a run's reads returned all ones, and OpenVMM logged that the device did not respond, in each of 3 runs per count on nvx-kvm-test (KVM), nvx-mshv-test (MSHV), and nvx-whp-test (WHP), all virtual machines; with a91d4c7a4, the same patch on d6addf8c1, no read did. With a91d4c7a4, the default microVM suite passed on all three, with 1, 2, and 4 vCPUs. At the pinned revision, the OpenVMM tests of the changed packages and the virtio devices pass, 545 on Windows and 550 on Linux, as they did at a91d4c7a4. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index cba21d109..46bfe02ed 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit cba21d109eeb99fbc573c79b2ed9f4754f734b25 +Subproject commit 46bfe02ed65697e858f66ce55eb60831b3ac3461 From da8157aecf4f19e751c07e08678af4e79b1d20c7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Mon, 9 Nov 2026 13:56:38 -0800 Subject: [PATCH 2/3] doc: map the input gating that keeps devices open in the OpenVMM roadmap Maps NVX's new pin, 46bfe02ed, the head of microsoft/openvmm#5005, at position 238, after position 237, cba21d109, the head of microsoft/openvmm#5004, on which it stacks, which follows positions 236, ac3dc27a7, the head of microsoft/openvmm#5003, and 235, b4028c1aa, the head of feature/microvm/dev. The pin is 107 commits past the mapped tip, and positions 132-238 stay unassigned. Position 238 changes the input gate that PR09's position 018 adds: the chipset device unit, which changed a device's gate with the device's mutex closed, now makes the gate's synchronous part under the device's ordinary lock and awaits the rest without the device, since the vCPUs run through a gate change. It changes the gate's methods of ChangeDeviceState, the erased chipset devices, and the remote device proxy, which position 018 adds, and the virtio transports' forwarding of the gate, which PR13's position 040 adds, and adds tests of the unit, both virtio transports, and the proxy. Replaying it on upstream main conflicts in all eight of the files that it changes, since upstream main lacks the input gate of position 018 and its forwarding by position 040, so it follows PR09 and PR13 upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 42 +++++++++++++++++++++++---------- 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 59e3086e2..c9aa6b2a3 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -8,12 +8,13 @@ reorganization kept it there; see [Upstream status](#upstream-status). - Mapped fork tip: `7677ad8d56a59864475012b63b6f01a22575c30d`, position 131. The graph, PR summary, and commit mapping below cover positions 001-131. -- NVX's pinned OpenVMM revision: `cba21d109eeb99fbc573c79b2ed9f4754f734b25`, position 237, the - head of microsoft/openvmm#5004, whose commit follows position 236, `ac3dc27a7`, the head of - microsoft/openvmm#5003, on which it stacks, whose commit follows position 235, `b4028c1aa`, the - head of `feature/microvm/dev`, which microsoft/openvmm#5001 merged after position 234, - `f41a1ecfc`, which microsoft/openvmm#4997 merged after position 233, `d501e5066`, which - microsoft/openvmm#4996 merged after position 232, `6cb6ba87c`, which +- NVX's pinned OpenVMM revision: `46bfe02ed65697e858f66ce55eb60831b3ac3461`, position 238, the + head of microsoft/openvmm#5005, whose commit follows position 237, `cba21d109`, the head of + microsoft/openvmm#5004, on which it stacks, whose commit follows position 236, `ac3dc27a7`, the + head of microsoft/openvmm#5003, on which that one stacks, whose commit follows position 235, + `b4028c1aa`, the head of `feature/microvm/dev`, which microsoft/openvmm#5001 merged after + position 234, `f41a1ecfc`, which microsoft/openvmm#4997 merged after position 233, `d501e5066`, + which microsoft/openvmm#4996 merged after position 232, `6cb6ba87c`, which microsoft/openvmm#4995 merged after position 231, `98e5b3ca5`, which microsoft/openvmm#4994 merged with position 230 after position 229, `ed8fc0713`, which microsoft/openvmm#4993 merged after position 228, `d6addf8c1`, which @@ -30,8 +31,8 @@ position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1 positions 149-220 are positions 148-219, and the tree of position 220 is the tree of - `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 106 commits past - the mapped tip on the same first-parent chain. Positions 132-237 are not yet assigned to PRs; see + `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 107 commits past + the mapped tip on the same first-parent chain. Positions 132-238 are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `ba105dd7f00499acc1862456c179c0930cb2c8f7` (2026-11-06), the fork base itself; see [Upstream main past the fork base](#upstream-main-past-the-fork-base). @@ -996,7 +997,8 @@ ba105dd7f -> kernel tuning admitted on platform command lines [235] -> memory-expansion ranges placed without overflow [236] -> FIFO artifacts refused without waiting for a writer [237] - -> cba21d109 (NVX pin) + -> device input gated without closing devices to the vCPUs [238] + -> 46bfe02ed (NVX pin) ``` For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main, @@ -1012,13 +1014,13 @@ reorganization folded the PR20 and PR12 fixups into the commits that they fix; s ## Commits after the mapped tip -The NVX pin extends the chain past position 131 without merges. None of the 106 later commits is a +The NVX pin extends the chain past position 131 without merges. None of the 107 later commits is a fixup of a mapped PR, and none is yet classified or assigned to PRs. Each of positions 132-186 combines the v2.4 commits that the table lists, as v3.0 positions 145-160 and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0 -reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-237 followed the +reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-238 followed the v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions 222 and 223. Each commit has the position that it had in v5.1 through position 147, and one less from v5.1 position 149 on; see @@ -1133,6 +1135,7 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4 | 235 | `b4028c1aa` | - | - | - | - | openvmm_helpers: admit kernel tuning on platform command lines (microsoft/openvmm#5001) | | 236 | `ac3dc27a7` | - | - | - | - | openvmm_helpers: compute memory-expansion ranges without overflow (microsoft/openvmm#5003) | | 237 | `cba21d109` | - | - | - | - | openvmm_helpers: refuse a FIFO artifact without waiting for a writer (microsoft/openvmm#5004) | +| 238 | `46bfe02ed` | - | - | - | - | vmotherboard: gate a device's input without closing it to the vCPUs (microsoft/openvmm#5005) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -1266,7 +1269,13 @@ restores: on Unix both now open with `O_NONBLOCK`, so that position 032's type c FIFO without waiting for a writer, and then make a regular file's descriptor blocking again. It also adds tests beside position 032's tests of the module and position 035's tests of a restore's artifacts, and extends the restore paragraph that PR12's position 036 adds to the snapshot format -guide. +guide. Position 238 changes the input gate that PR09's position 018 adds: the chipset device unit, +which changed a device's gate with the device's mutex closed, as it changes the device's other +states, now makes the gate's synchronous part under the device's ordinary lock and awaits the rest +without the device, since the vCPUs run through a gate change. It changes the gate's methods of +`ChangeDeviceState`, the erased chipset devices, and the remote device proxy, which position 018 +adds, and the virtio transports' forwarding of the gate, which PR13's position 040 adds, and adds +tests of the unit, both virtio transports, and the proxy. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 177, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -1522,6 +1531,15 @@ adds to the snapshot format guide. Replaying it on upstream main conflicts in al files: upstream main lacks the file system module of position 032, the directory-relative restore of position 035, and that paragraph, so it follows PR11, PR12, and PR19 upstream. +Position 238 changes only the input gate that PR09's position 018 adds to `vmcore`'s device state, +the chipset device unit, the erased chipset devices, and the remote device proxy, whose files +PR10's position 019 and position 218 also change, and the gate's forwarding through the virtio +transports, which PR13's positions 039 and 040 add and PR16's positions 066 and 068 and positions +183 and 218 also change. It adds tests of the unit, the transports, and the proxy, those of the +transports in a new file. Replaying it on upstream main conflicts in all eight of the files that it +changes: upstream main lacks the input gate of position 018 and its forwarding by position 040, so +it follows PR09 and PR13 upstream. + ## PR summary Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0 From 3beaacf6f0713bc2b4eb1e5ec2fa0758e24eeac5 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Mon, 9 Nov 2026 19:12:52 -0800 Subject: [PATCH 3/3] doc: say which files of the input gating conflict upstream Position 238 changes nine files, eight that it modifies and the new file that holds the transports' tests, but the roadmap said that its replay on upstream main conflicts in all eight of the files that it changes. The replay conflicts in the eight that it modifies, and the new file applies cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index c9aa6b2a3..b5c1116cd 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -1536,9 +1536,9 @@ the chipset device unit, the erased chipset devices, and the remote device proxy PR10's position 019 and position 218 also change, and the gate's forwarding through the virtio transports, which PR13's positions 039 and 040 add and PR16's positions 066 and 068 and positions 183 and 218 also change. It adds tests of the unit, the transports, and the proxy, those of the -transports in a new file. Replaying it on upstream main conflicts in all eight of the files that it -changes: upstream main lacks the input gate of position 018 and its forwarding by position 040, so -it follows PR09 and PR13 upstream. +transports in a new file. Replaying it on upstream main conflicts in eight of its nine files, all +but that new one, which applies cleanly: upstream main lacks the input gate of position 018 and its +forwarding by position 040, so it follows PR09 and PR13 upstream. ## PR summary