diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index 2c43f957b..f31ac2009 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -234,7 +234,9 @@ in step 3 waits for its endpoint to read it. A device that never finishes, or an endpoint that stops reading, stalls the capture, with nothing published and the source never resumed, until the process that launched the VMM ends it; the VMM logs each state unit that has not answered after five seconds, so that the -log of such a capture names it. Every launcher therefore bounds a capture with +log of such a capture names it. A device whose worker has ended fails the +quiesce instead, and the source terminates with nothing published, since that +device can never start again. Every launcher therefore bounds a capture with a deadline that ends the VMM, as the nvx crate's template store does with the deadline that creating or refreshing a template requires. diff --git a/doc/design/5.6-shutdown-and-failure-semantics.md b/doc/design/5.6-shutdown-and-failure-semantics.md index 29a4838ca..1a29eca63 100644 --- a/doc/design/5.6-shutdown-and-failure-semantics.md +++ b/doc/design/5.6-shutdown-and-failure-semantics.md @@ -66,6 +66,9 @@ Failures are classified by whether execution can safely continue: - A device that fails to start fails the VM's start, including the first one after a restore, before any virtual processor runs, and a restore then publishes no readiness. +- A device whose worker has ended fails the state change that it does not + answer, such as a stop, save, or restore, and never starts again, so the VM + does not run again either. This fail-closed rule prevents success-shaped recovery from uncertain state. The architecture prefers terminating one process generation over resuming a diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index f31700682..52a30bb30 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -7,11 +7,12 @@ `61994293a9e5e3bcc1d685d708cf462920eed030`, and the `feature/microvm/release/v6.0` reorganization kept it there; see [Upstream status](#upstream-status). - Mapped fork tip: `7677ad8d56a59864475012b63b6f01a22575c30d`, position 131. The graph, PR - summary, and commit mapping below cover positions 001-131 and the PR09 fixups at positions 241 - and 242. -- NVX's pinned OpenVMM revision: `6ddd6ca505a7212d92e2a8e561a19635f7185d4a`, position 242, the - head of microsoft/openvmm#5012, whose two commits, positions 241 and 242, follow position 240, - `4c2697aef`, the head of `feature/microvm/dev`, which microsoft/openvmm#5010 merged with + summary, and commit mapping below cover positions 001-131 and the PR09 fixups at positions + 241-243. +- NVX's pinned OpenVMM revision: `a8a0e35178f61c0e82cbe465b7ebf70e0f0bb543`, position 245, the + head of microsoft/openvmm#5013, whose three commits, positions 243-245, follow position 242, + `6ddd6ca50`, the head of `feature/microvm/dev`, which microsoft/openvmm#5012 merged with + position 241 after position 240, `4c2697aef`, which microsoft/openvmm#5010 merged with position 239 after position 238, `46bfe02ed`, which microsoft/openvmm#5005 merged after position 237, `cba21d109`, which microsoft/openvmm#5004 merged after position 236, `ac3dc27a7`, which microsoft/openvmm#5003 merged after position 235, `b4028c1aa`, which @@ -33,9 +34,9 @@ position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1 positions 149-220 are positions 148-219, and the tree of position 220 is the tree of - `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 111 commits past - the mapped tip on the same first-parent chain. Positions 132-240 are not yet assigned to PRs, and - positions 241 and 242 are PR09 fixups; see + `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 114 commits past + the mapped tip on the same first-parent chain. Positions 132-240, 244, and 245 are not yet + assigned to PRs, and positions 241-243 are PR09 fixups; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `ba105dd7f00499acc1862456c179c0930cb2c8f7` (2026-11-06), the fork base itself; see [Upstream main past the fork base](#upstream-main-past-the-fork-base). @@ -245,13 +246,13 @@ transactions, was closed, and the v5.0 reorganization retired the commit and fol without a deadline into the commits that it changes; see [The v5.0 reorganization](#the-v50-reorganization). microsoft/openvmm#4872, opened on 2026-11-03, proposes position 015's rollback of a partial start, with review changes that the chain lacks, -apart from its handling of a unit in a terminal state and its teardown after a failed restart, -which the PR09 fixups at positions 241 and 242 carry. microsoft/openvmm#4626, which proposes -position 018's input gate without a deadline, as the chain now has it, builds on it. So does -microsoft/openvmm#4937, which proposes no chain commit: it makes -`StateUnits::stop` fail instead of panicking when a unit's task has ended, so a rebase onto it -adapts the chain's own callers of `StateUnits::stop`, in the microVM worker's -`dispatch/microvm.rs` and `dispatch/restore.rs`. +apart from its handling of a unit in a terminal state, its teardown after a failed restart, and +its rollback's stop in reverse dependency order, which the PR09 fixups at positions 241-243 carry. +microsoft/openvmm#4626, which proposes position 018's input gate without a deadline, as the chain +now has it, builds on it. So does microsoft/openvmm#4937, which makes `StateUnits::stop` fail +instead of panicking when a unit's task has ended, as position 244 does in the chain. Position 244 +also adapts the chain's own callers of `StateUnits::stop`, such as the microVM worker's snapshot +quiesce, and makes `VmRpc::Pause` fail where #4937's VM worker panics on a failed stop. On 2026-11-05, microsoft/openvmm#4605 merged as `48e0681e9`, narrowed in review to the read path: a read error is traced, drops the socket, and is reported as EOF, while writes keep upstream's @@ -868,7 +869,7 @@ flowchart TB subgraph Generic["Generic snapshot and persistence chain"] direction LR - P09["PR09
Fallible VM lifecycle
U - 4 commits + 2 fixups, 3 in review"] + P09["PR09
Fallible VM lifecycle
U - 4 commits + 3 fixups, 3 in review"] P10["PR10
CPU/clock state
U - 11 commits"] P11["PR11
Snapshot publication
U - 4 commits"] P12["PR12
Private-CoW restore
U - 5 commits"] @@ -1004,8 +1005,9 @@ ba105dd7f -> FIFO artifacts refused without waiting for a writer [237] -> device input gated without closing devices to the vCPUs [238] -> quiesced microVM run again only through its rollback [239-240] - -> PR09 fixups [241-242] - -> 6ddd6ca50 (NVX pin) + -> PR09 fixups [241-243] + -> state changes failed rather than panicking on a lost unit [244-245] + -> a8a0e3517 (NVX pin) ``` For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main, @@ -1017,20 +1019,22 @@ prepared after PR13; PR23's protocol and broker commits can be prepared after PR The fork tail, positions 127-129, remains outside the upstream queue. PR27 can be prepared once PR22 lands; its generic confinement commit can be submitted independently if preferred. The v6.0 reorganization folded the PR20 and PR12 fixups into the commits that they fix; see -[The v6.0 reorganization](#the-v60-reorganization). Fold positions 241 and 242 into PR09's position -015 before submission, which adds the terminal state that position 241 handles and the restart's -handling of a failed start that position 242 changes; position 241's check of a time advance, with -its test, folds into PR10's position 019, which adds the time advance. +[The v6.0 reorganization](#the-v60-reorganization). Fold positions 241-243 into PR09's position 015 +before submission, which adds the terminal state that position 241 handles, the restart's handling +of a failed start that position 242 changes, and the rollback whose stop position 243 orders; +position 241's check of a time advance, with its test, folds into PR10's position 019, which adds +the time advance, and position 243's description of the snapshot rollback's start into PR17's +position 072, which adds that rollback. ## Commits after the mapped tip -The NVX pin extends the chain past position 131 without merges. Of the 111 later commits, -positions 241 and 242 are PR09 fixups, and the others are not yet classified or assigned to PRs. +The NVX pin extends the chain past position 131 without merges. Of the 114 later commits, +positions 241-243 are PR09 fixups, and the others are not yet classified or assigned to PRs. Each of positions 132-186 combines the v2.4 commits that the table lists, as v3.0 positions 145-160 and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0 -reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-242 followed the +reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-245 followed the v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions 222 and 223. Each commit has the position that it had in v5.1 through position 147, and one less from v5.1 position 149 on; see @@ -1150,6 +1154,9 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4 | 240 | `4c2697aef` | - | - | - | - | openvmm_core: run a quiesced microVM again only through its rollback (microsoft/openvmm#5010) | | 241 | `78e752de1` | - | - | - | - | state_unit: handle a unit in a terminal state instead of panicking (microsoft/openvmm#5012); fold into 015, its time advance check into 019 | | 242 | `6ddd6ca50` | - | - | - | - | openvmm_core: tear the VM down when a restart cannot resume it (microsoft/openvmm#5012); fold into 015 | +| 243 | `2fcd1573d` | - | - | - | - | state_unit: roll back a partial start without panicking on a lost unit (microsoft/openvmm#5013); fold into 015, its snapshot rollback description into 072 | +| 244 | `5a750a862` | - | - | - | - | state_unit: fail a stop instead of panicking when a unit is lost (microsoft/openvmm#5013) | +| 245 | `a8a0e3517` | - | - | - | - | state_unit: fail a reset, save, restore, or time advance on a lost unit (microsoft/openvmm#5013) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -1306,7 +1313,25 @@ reset, save, or restore, and the time advance that PR10's position 019 adds, fai in it. It rewrites position 015's description of that state and adds tests in a new file. Position 242 has the VM worker tear the VM down when that restart's start fails, rather than keep serving a VM that it marks running over stopped units, and has the worker's teardown stop the state units -when they run, where upstream's microsoft/openvmm#4863 stops them when the VM runs. +when they run, where upstream's microsoft/openvmm#4863 stops them when the VM runs. Position 243 +changes position 015's rollback too: it stops the units that started again in reverse dependency +order, so that a started unit that a unit that is not stopped depends on keeps running, and leaves +a started unit whose task does not answer that stop in the terminal state, rather than panicking, +with the units kept running for a later stop. It rewrites position 015's description of the +rollback, adds tests beside position 015's, and extends the description of the snapshot rollback's +start, which PR17's position 072 adds to the VM worker. Position 244 has `StateUnits::stop` fail, +rather than panic, when a unit does not answer, leaving that unit in a new terminal state, and +moves position 243's ordered stop into a module that the stop and the rollback share. It changes +the callers that the chain adds or changes: the restart and the teardown that position 242 +changes; the snapshot quiesce and boundary of PR17's position 072, whose quiesce error it extends, +the boundary's request filter of position 073, and the VM controller's capture of position 076; +the restore readiness failure of PR12's position 037 and the gate timeout of PR20's position 095; +and, in OpenHCL, the stops of a pause, a save, a restart, and servicing, beside the start that +position 015 makes fallible. It makes `VmRpc::Pause` failable, as position 016 makes +`VmRpc::Resume`, and has position 241's tests check the stop's result. Position 245 has a reset, a +save, a restore, and the time advance of PR10's position 019 fail, rather than panic, when a unit +does not answer, leaving that unit in a terminal state of the operation, which position 241's +check of terminal units then refuses, and adds tests in a new file. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 177, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -1603,6 +1628,21 @@ its check that the units are stopped, and with a teardown stop that also tolerat terminal state whose task has ended. So they follow PR09 upstream, folded into position 015, and position 241's time advance check into PR10's position 019. +Positions 243-245 build on the chain too. Replaying position 243 on upstream main conflicts in all +three of its files: upstream main lacks the start rollback of position 015, its tests, and the +snapshot rollback of PR17's position 072, so it follows PR09 upstream, folded into position 015, +and its description of the snapshot rollback into position 072; microsoft/openvmm#4872's head +orders its rollback's stop the same way. Replaying position 244 conflicts in 12 of its 20 files, +since upstream main lacks, among others, the rollback and terminal states of positions 015, 241, +and 243, the fallible resume of position 016, and the microVM worker, snapshot boundary, and +controller of PR15 and PR17; its new module and tests and its changes to OpenHCL, the VMRS dump, +the ttrpc server, the VM controller's disk snapshot, and petri apply cleanly. +microsoft/openvmm#4937 proposes its change of `StateUnits::stop` for upstream's code, on +microsoft/openvmm#4872, so it follows PR09 upstream. Replaying position 245 conflicts in two of its +three files, all but its new test file: upstream main lacks the terminal state of position 015, +which `state_unit`'s `lib.rs` extends, and the time advance of PR10's position 019, so it follows +PR09 and PR10 upstream. + ## PR summary Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0 @@ -1621,7 +1661,7 @@ reorganizations folded into them; see [The v6.0 reorganization](#the-v60-reorgan | PR06 | U | Device fixes and hardening | 013-014 | `6ee09d624..5ed5f7a22` | 2 | Current upstream | | PR07 | U | Consomme resource bounding | - | all landed upstream | 0 | Current upstream | | PR08 | U | Preparatory refactors | - | all landed upstream | 0 | Current upstream | -| PR09 | U | Fallible VM lifecycle | 015-018; 241-242 (fixups) | `179106cd9..07a98b977`; `78e752de1..6ddd6ca50` | 6 | Current upstream | +| PR09 | U | Fallible VM lifecycle | 015-018; 241-243 (fixups) | `179106cd9..07a98b977`; `78e752de1..2fcd1573d` | 7 | Current upstream | | PR10 | U | Portable CPU and clock state | 019-029 | `a1e113d12..ab67f39e2` | 11 | PR09 | | PR11 | U | Snapshot publication protocol | 030-033 | `23e759e74..112512c35` | 4 | PR10 | | PR12 | U | Immutable private-CoW restore | 034-038 | `6c89f4c94..3da713626` | 5 | PR10 | @@ -1646,7 +1686,7 @@ reorganizations folded into them; see [The v6.0 reorganization](#the-v60-reorgan Each entry, at the position that the commit had in v5.0 and v5.1 too, names the v4.0 position of the commit, the v3.0 and v2.4 positions that v4.0 recorded, and the v2.4 and v5.1 fixups that it folds; -"(part)" marks a fixup that split across commits. Positions 241 and 242, which followed the v6.0 +"(part)" marks a fixup that split across commits. Positions 241-243, which followed the v6.0 reorganization, name their OpenVMM pull request instead. A v4.0 position counts in `feature/microvm/dev` at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4.0`. Landed entries name the commits and positions of the branch that @@ -1807,14 +1847,15 @@ Landed upstream and dropped from the chain:
-PR09 - Fallible VM lifecycle (U) - 6 commits + 1 landed upstream + 2 retired +PR09 - Fallible VM lifecycle (U) - 7 commits + 1 landed upstream + 2 retired -- 015 `179106cd9` - state_unit: roll back a partial start instead of panicking (v4.0 018, v3.0 026, v2.4 027; **open upstream**: microsoft/openvmm#4872, with review changes that the chain lacks, apart from those of positions 241 and 242; microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) +- 015 `179106cd9` - state_unit: roll back a partial start instead of panicking (v4.0 018, v3.0 026, v2.4 027; **open upstream**: microsoft/openvmm#4872, with review changes that the chain lacks, apart from those of positions 241-243; microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) - 016 `1ab89b3fe` - openvmm: make VM resume fallible (v4.0 019, v3.0 028, v2.4 029; **closed upstream**: microsoft/openvmm#4623 was closed in favor of microsoft/nvx#1232, but the resume stays fallible: the post-restore input gate, the restore readiness event, and the time ABI's TSC offset measurement fail it) - 017 `d51a1e8ef` - state_unit: record the exact unit inventory in saved state (v4.0 020, v3.0 029, v2.4 030; **open upstream**: microsoft/openvmm#4624) - 018 `07a98b977` - vmcore: add device input quiesce and resume (v4.0 022, v3.0 031, v2.4 032; **open upstream**: microsoft/openvmm#4626, stacked on microsoft/openvmm#4872, without the input gate's deadline, which the v5.0 reorganization removed here too, by folding v4.0 position 216) - 241 `78e752de1` - state_unit: handle a unit in a terminal state instead of panicking (microsoft/openvmm#5012; **fold into 015**, its check and test of a time advance into PR10's 019; microsoft/openvmm#4872's head makes the same change) - 242 `6ddd6ca50` - openvmm_core: tear the VM down when a restart cannot resume it (microsoft/openvmm#5012; **fold into 015**; microsoft/openvmm#4872's head makes the same change) +- 243 `2fcd1573d` - state_unit: roll back a partial start without panicking on a lost unit (microsoft/openvmm#5013; **fold into 015**, its description of the snapshot rollback into PR17's 072; microsoft/openvmm#4872's head orders its rollback's stop the same way) Landed upstream and dropped from the chain: diff --git a/openvmm b/openvmm index 6ddd6ca50..a8a0e3517 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 6ddd6ca505a7212d92e2a8e561a19635f7185d4a +Subproject commit a8a0e35178f61c0e82cbe465b7ebf70e0f0bb543