diff --git a/doc/ci.md b/doc/ci.md index 934231e2b..2e81747bf 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -233,6 +233,16 @@ settings, and it must restore them when the request returns (`nvx-time cancel-capture`). Afterwards, `nvx-time status` must exit 0 with a passing boot line at generation 0 and no restore line, and `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` must read 0. +The `snapshot-rollback` scenario fails a capture after its quiesce. The +guest's RAM has a file of its own, so the snapshot destination's parent holds +nothing, and it disappears after boot: the capture quiesces and saves the VM, +then fails to publish, and OpenVMM rolls it back. The request must return in +the source with status 0, after OpenVMM's capture profile records exactly one +quiesce and one save but no publication, and the source must pass the checks +that `snapshot-core` runs after its released request. Once the parent is back, +a second request of the same source must publish a snapshot and end the +source with status 0, leaving only the snapshot in the parent, and the +snapshot's restore must continue past that request exactly once. The `platform-zero-fill` scenario covers a platform restore whose `distro` image is shorter than the snapshot's and whose fresh scratch is absent. It captures a platform snapshot with a 1 MiB `distro` placeholder and a 1 MiB diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 69afe4af6..372af6da6 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -48,8 +48,9 @@ fallback, from automatic CPU-profile selection to a host-derived CPU profile, warns at every cold boot that takes it. The VMM uses checked address arithmetic, bounded buffers and tables, typed -validation errors, rate-limited guest-triggerable logs, and fallible quiesce -and restore operations. Native I/O callbacks only record bounded state or +validation errors, rate-limited guest-triggerable logs, and fallible restore +operations, and it rolls back a capture that fails after the quiesce and +before its commit point. Native I/O callbacks only record bounded state or enqueue notifications; they never perform blocking snapshot or host-resource work. Malformed saved device state is rejected before device workers start, and no virtual processor runs after a partial restore failure. A failure after diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index 216f04845..a261ee360 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -98,5 +98,7 @@ Writable scratch follows one of two policies: Transport and queue progress are saved with the other device state. Stopping a queue prevents new descriptor intake and drains every accepted I/O to one -used-ring completion within the bounded snapshot quiesce timeout. The +used-ring completion. The drain has no deadline of its own: a backend that +never completes an I/O stalls the capture until its launcher ends the VMM; see +[Capture Transaction](5.3-snapshot-protocol.md#533-capture-transaction). The controller flushes and copies scratch only after that drain. diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index 8731b583e..81e62151f 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -33,7 +33,7 @@ A capture requires up to three settings: a restore must supply the same one; see [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). -A capture also accepts four optional settings: +A capture also accepts three optional settings: - **Memory backing file.** The file that backs guest RAM. When none is supplied, the VMM creates one at launch in the parent of the destination @@ -44,8 +44,6 @@ A capture also accepts four optional settings: reserves so that a restore can give the VM more memory. Without it, a restore cannot add memory; see [Memory Activation](5.5-restore-protocol.md#554-memory-activation). -- **Quiesce timeout.** The longest the VMM may spend quiescing the VM, - including draining block I/O. - **Scratch restore mode.** How a restore recreates [paired scratch](5.5-restore-protocol.md#552-memory-and-scratch-materialization): a private copy; a copy-on-write clone, only for a `workload-start` snapshot; @@ -186,22 +184,31 @@ sequenceDiagram Units-->>Controller: State, inventory, time contract, and CPU profile Controller->>Storage: Copy state, memory, and optional scratch Controller->>Storage: Publish with no-replace rename - Controller->>Worker: Terminate committed source + alt Published + Controller->>Worker: Terminate committed source + else Failed before the rename + Controller->>Worker: Roll back + Worker->>Units: Start in dependency order and resume host input + Worker-->>Guest: Continue execution + end end ``` **Figure 5.2.** Capture transaction. -The VMM performs the capture as one bounded transaction: +The VMM performs the capture as one transaction: 1. Gate host input and defer completion of the snapshot-port write. 2. Stop the virtual processor at the I/O boundary while completing the write, so saved state starts at the instruction immediately after the output. 3. Preflight the machine profile, destination, backend, external attachments, and snapshot eligibility; a missing destination or a rejected preflight - releases the boundary and lets the guest continue. + releases the boundary and lets the guest continue. Otherwise, deliver the + console output that the guest wrote before its request; output accepted + while no endpoint is connected stays in the snapshot, and the restored VM + writes it. 4. Quiesce the remaining device workers and VM time in dependency order, - including the bounded block queue drain. + including the block queue drain. 5. Establish the time contract: require that no LAPIC timer is periodic or in TSC-deadline mode and that PIT channel 0 is not counting periodically, take the capture anchor and host identities, and record the declared rates, CPU @@ -213,8 +220,22 @@ The VMM performs the capture as one bounded transaction: 8. Publish the directory with one no-replace rename. 9. Terminate the source worker and process after publication. -A failure in step 5 is a rollback-safe capture failure. Deriving `Q` never -fails a capture; when it cannot be derived, it is omitted with a warning. +A failure in steps 5 through 8, before the rename, rolls the capture back: +the VMM starts the stopped state units again in dependency order, resumes host +input, and releases the boundary, so the guest continues. The source +terminates instead when a unit does not start again, or when the VMM cannot +remove the staging directory's alias of guest RAM. Deriving `Q` never fails a +capture; when it cannot be derived, it is omitted with a warning. + +The transaction has no deadline of its own. Gating host input in step 1 and +the quiesce in step 4 wait for every device, and delivering the console output +in step 3 waits for its endpoint to read it. A device that never finishes, or +an endpoint that stops reading, stalls the capture, with nothing published and +the source never resumed, until the process that launched the VMM ends it; the +VMM logs each state unit that has not answered after five seconds, so that the +log of such a capture names it. Every launcher therefore bounds a capture with +a deadline that ends the VMM, as the nvx crate's template store does with the +deadline that creating or refreshing a template requires. ## 5.3.4 State-Unit Ordering @@ -261,16 +282,16 @@ flowchart LR ## 5.3.5 Commit Point The directory rename is the commit point. Before it, a capture failure exposes -no final snapshot, and the guest resumes only when rollback is known to be -valid. After it, the source never resumes. This gives the guest an observable -boundary: code after the snapshot request runs once in each restored process -and never in the successfully captured source. A capture driven through the -management RPC likewise halts the source at the committed boundary and -terminates its process, so clients observe the transport closing. - -Establishing the boundary is itself fallible. If host input cannot be gated, -the worker rolls the gate back and releases the write so that the guest -continues; if that rollback is uncertain, the worker terminates. Once +no final snapshot, and the guest resumes only once the VMM has started the +stopped state units again. After it, the source never resumes. This gives the +guest an observable boundary: code after the snapshot request runs once in +each restored process and never in the successfully captured source. A capture +driven through the management RPC likewise halts the source at the committed +boundary and terminates its process, so clients observe the transport closing. + +Establishing the boundary is itself fallible. If a device fails to gate its +host input, the worker rolls the gate back and releases the write so that the +guest continues; if that rollback fails, the worker terminates. Once processor stopping has begun, a boundary failure is terminal: host input stays gated, and the VM is torn down rather than resumed. The post-restore acknowledgement boundary follows the same rule. @@ -284,9 +305,10 @@ As Table 5.3 shows, a published snapshot never coexists with a running source. | Outcome | Source | Snapshot | | --- | --- | --- | | Committed capture | Terminated | Published | -| Missing destination or rejected preflight | Continues | None | -| Failure before the commit point, with rollback known to be valid | Continues | None | -| Failure before the commit point, with rollback not known to be valid | Terminated | None | +| Missing destination, rejected preflight, or failed input gate | Continues | None | +| Failure after the quiesce, before the commit point, that rolls back | Continues | None | +| Failure before the commit point that cannot roll back | Terminated | None | +| Input gate, console delivery, or quiesce that never completes | Ended by its launcher | None | After a commit, execution continues only in virtual machines restored from the snapshot, either as clones or as one resumed instance; see diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 3fb7c63f7..82c9a07c4 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -7,18 +7,18 @@ `b2676222b383bf50c7022f6a5983920421469a47`; see [Upstream status](#upstream-status). - Mapped fork tip: `3cdf4fe3834cfb90b9ee685875493bc9bff1b7a4`, position 135. The graph, PR summary, and commit mapping below cover positions 001-135 and the PR20 fixup at position 152. -- NVX's pinned OpenVMM revision: `9cec2b89245f41f2126f9a787c5e43a0d84b7154`, position 214. - Position 214, which microsoft/openvmm#4916 proposes for `feature/microvm/dev`, is one commit on - position 213, `4713c539a`, the head of `feature/microvm/dev`. microsoft/openvmm#4894, - microsoft/openvmm#4886, and microsoft/openvmm#4887 landed positions 211-213 there by - fast-forward, on position 210, `eda13d80a`, the head of the `feature/microvm/release/v4.0` - branch of microsoft/openvmm. The v4.0 reorganization moved the 205 commits of - `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto `c6d3da0ff`. It - dropped the eight commits whose changes upstream had landed or superseded, retired the fallible - device start that microsoft/nvx#1232 replaces with awaited device restoration, folded that - restoration into the commits that it changes, and added 14 commits that give each ABI that the - chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The - pin is 79 commits past the mapped tip on the same first-parent chain. Positions 136-214, except +- NVX's pinned OpenVMM revision: `14dc4116ec7e57567633c4b7b07892aca3a58522`, position 216. Positions + 215 and 216, which microsoft/openvmm#4921 proposes for `feature/microvm/dev`, are two commits on + position 214, `9cec2b892`, the head of `feature/microvm/dev`. microsoft/openvmm#4916 landed + position 214 there by fast-forward, as microsoft/openvmm#4894, microsoft/openvmm#4886, and + microsoft/openvmm#4887 landed positions 211-213 on position 210, `eda13d80a`, the head of the + `feature/microvm/release/v4.0` branch of microsoft/openvmm. The v4.0 reorganization moved the 205 + commits of `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto + `c6d3da0ff`. It dropped the eight commits whose changes upstream had landed or superseded, retired + the fallible device start that microsoft/nvx#1232 replaces with awaited device restoration, folded + that restoration into the commits that it changes, and added 14 commits that give each ABI that + the chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The + pin is 81 commits past the mapped tip on the same first-parent chain. Positions 136-216, except the PR20 fixup at position 152, are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `ca6b2b177da5e33ae1bcac72986b31d6b039811c` (2026-11-02), one commit past the @@ -174,8 +174,8 @@ and 023 is in an open upstream pull request; the commits that upstream has since [Landed upstream](#landed-upstream), and PR08 has landed whole. Each pull request carries one commit, except microsoft/openvmm#4554, which carries positions 055-057 after the `boot_params` field that microsoft/openvmm#4661 has since landed; its ELF and initrd validation was split out -during review and landed as microsoft/openvmm#4667 (`a87971184`). One of the 21 pull requests, -microsoft/openvmm#4625, is a draft, and none has an approving review yet. +during review and landed as microsoft/openvmm#4667 (`a87971184`). One of those 21 pull requests, +microsoft/openvmm#4625, was a draft, and none had an approving review yet. microsoft/openvmm#4451, which proposed position 001, was closed without merging, and neither the remainder of position 006 nor position 023, which the v3.0 reorganization moved into PR09, has been proposed. microsoft/openvmm#4619, microsoft/openvmm#4622, and microsoft/openvmm#4623, which @@ -185,6 +185,17 @@ position 027 is retired, and position 019's resume stays fallible; see [The v4.0 reorganization](#the-v40-reorganization). PR10-PR13 and PR15-PR27 have no upstream pull requests yet. +On 2026-11-04, microsoft/nvx#1260 withdrew position 021 from PR09, since the deadline of the +snapshot quiesce added no safety: microsoft/openvmm#4625, which proposed position 021's bounded +quiesce transactions, was closed, and position 216 removes them, so the next reorganization drops +position 021. microsoft/openvmm#4872, opened on 2026-11-03, proposes position 018's rollback of a +partial start, with review changes that the chain lacks, and microsoft/openvmm#4626, which proposes +position 022's input gate without the deadline that position 216 removes from it, builds on it. So +does microsoft/openvmm#4937, which proposes no chain commit: it makes `StateUnits::stop` fail +instead of panicking when a unit's task has ended, so a rebase onto it adapts the chain's own +callers of `StateUnits::stop`, in the microVM worker's `dispatch/microvm.rs` and +`dispatch/restore.rs`. + | Roadmap PR | Positions | Upstream pull requests, in position order | | --- | ---: | --- | | PR01 | 002-005 | microsoft/openvmm#4595, microsoft/openvmm#4618, microsoft/openvmm#4598, microsoft/openvmm#4615 | @@ -194,13 +205,12 @@ requests yet. | PR05 | 011-013 | microsoft/openvmm#4536, microsoft/openvmm#4607, microsoft/openvmm#4613 | | PR06 | 014-016 | microsoft/openvmm#4597, microsoft/openvmm#4605, microsoft/openvmm#4609 | | PR07 | 017 | microsoft/openvmm#4600 | -| PR09 | 020-022 | microsoft/openvmm#4624, microsoft/openvmm#4625, microsoft/openvmm#4626 | +| PR09 | 018, 020, 022 | microsoft/openvmm#4872, microsoft/openvmm#4624, microsoft/openvmm#4626 | | PR14 | 055-058 | microsoft/openvmm#4554 (055-057), microsoft/openvmm#4611 | Stacked pull requests carry their prerequisites' commits: microsoft/openvmm#4609 builds on microsoft/openvmm#4585, which has merged; microsoft/openvmm#4613 builds on microsoft/openvmm#4536; -and microsoft/openvmm#4625 builds on the closed microsoft/openvmm#4619, as microsoft/openvmm#4626 -builds on microsoft/openvmm#4625, so both need a rebase that drops the fallible unit start. +and microsoft/openvmm#4626 builds on microsoft/openvmm#4872. Most of these pull requests have gained review commits that the fork chain does not carry. When one merges, the next rebase should drop its fork commit, as the `microvm/v2.2`, `feature/microvm/release/v2.3`, and `feature/microvm/release/v2.4` rebases and the v3.0 and v4.0 @@ -647,8 +657,8 @@ c6d3da0ff -> sandbox block roles [192] -> virtio feature masks [193-194] -> IOAPIC interrupt lines [195-196] -> ABI restart [197-210] -> boot console exit drain [211] -> portb without host input [212-213] - -> long Windows paths [214] - -> 9cec2b892 (NVX pin) + -> long Windows paths [214] -> snapshot quiesce without a deadline [215-216] + -> 14dc4116e (NVX pin) ``` For upstream review, PR01-PR07 and PR14 can be rebased independently onto current upstream main, @@ -664,12 +674,12 @@ see [The v3.0 reorganization](#the-v30-reorganization). ## Commits after the mapped tip -The NVX pin extends the chain past position 135 without merges. Of the 79 later commits, position +The NVX pin extends the chain past position 135 without merges. Of the 81 later commits, position 152 is a PR20 fixup, and the others are not yet classified or assigned to PRs. Each of positions 136-191 combines the v2.4 commits that the table lists, as v3.0 positions 145-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 192-196 followed the v3.0 reorganization, positions 197-210 are the ABI restart of the v4.0 -reorganization, and positions 211-214 followed it, so they combine none. +reorganization, and positions 211-216 followed it, so they combine none. | Position | Commit | v3.0 position | v2.4 positions | Topic | | ---: | --- | ---: | --- | --- | @@ -752,6 +762,8 @@ reorganization, and positions 211-214 followed it, so they combine none. | 212 | `de29849a4` | - | - | vmm_tests: drive the microVM guests without portb input (microsoft/openvmm#4886) | | 213 | `4713c539a` | - | - | microVM: remove portb's host input (microsoft/openvmm#4887) | | 214 | `9cec2b892` | - | - | pal: support long paths in secure file and directory creation (microsoft/openvmm#4916) | +| 215 | `1108c05c7` | - | - | state_unit: log the units that a state change waits for (microsoft/openvmm#4921) | +| 216 | `14dc4116e` | - | - | microVM: quiesce snapshots without a deadline (microsoft/openvmm#4921) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -788,7 +800,18 @@ then removes that input: the receive path of the portb console of PR15's positio PR20's position 098 shares with the restore records and PR20's position 099 gates, and the console input that PR15's position 068 gives the device on the CLI. Since those reads also noticed a departed peer of a listening endpoint, position 213 has the device poll the backend for -it instead. Assigning the later +it instead. Position 216 removes the bounded quiesce transactions that PR09's position 021 adds, +so that the next reorganization drops position 021, and the uncertain quiesce that PR17's +positions 076 and 080 handle in the worker and the controller, whose capture now quiesces and +rolls back with the start rollback of PR09's position 018. It also removes the capture's other +bounds. The input gate of PR09's position 022, whose module it makes private, loses its timeout, +and so do the snapshot-request port of PR17's position 075, its handles in OpenVMM and petri from +PR17's position 080, the worker of PR17's position 076, and the post-restore input gate of PR20's +position 099, which passed it on; the console flush of PR24's position 123 loses its own. The CLI +option of PR17's position 080 that set both goes, and the management RPC's field of PR25's +position 128 is reserved. The dispatcher test of PR17's position 077 builds the RPCs and the port +without durations, the boot console flush test of position 211 calls the flush without one, and +the input gate's tests reuse the log of position 215's test. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 182, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -846,6 +869,12 @@ publication file, whose new tests use the paired scratch of PR19's position 096 RAM of PR11's position 038. Its `pal` change alone applies to upstream main without conflicts, and `pal` builds there and passes its tests on Windows, so that change can go upstream on its own. +Position 215 changes only `state_unit`'s state change requests and its tests: replaying it on +upstream main conflicts only where its test module joins those of positions 018, 020, and 021 in +`lib.rs`. Position 216 changes the microVM capture of PR09, PR17, PR20, PR24, and PR25, its CLI +and management RPC, and their Guide pages: replaying it on upstream main conflicts in all 24 of +its files, 18 of which upstream main lacks, so it follows those PRs upstream. + ## PR summary Each count is of v4.0 commits, which include the fixups that the v3.0 reorganization folded into @@ -1040,11 +1069,11 @@ Landed upstream and dropped from the chain:
PR09 - Fallible and bounded VM lifecycle (U) - 6 commits + 1 retired -- 018 `a1a72fa96` - state_unit: roll back a partial start instead of panicking (v3.0 026, v2.4 027; **closed upstream**: microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) +- 018 `a1a72fa96` - state_unit: roll back a partial start instead of panicking (v3.0 026, v2.4 027; **open upstream**: microsoft/openvmm#4872, with review changes that the chain lacks; microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) - 019 `4b98c153d` - openvmm: make VM resume fallible (v3.0 028, v2.4 029; **closed upstream**: microsoft/openvmm#4623 was closed in favor of microsoft/nvx#1232, but the resume stays fallible: the post-restore input gate, the restore readiness event, and the time ABI's TSC offset measurement fail it) - 020 `6fedcc054` - state_unit: record the exact unit inventory in saved state (v3.0 029, v2.4 030; **open upstream**: microsoft/openvmm#4624) -- 021 `03291e1ea` - state_unit: add bounded quiesce transactions with rollback classification (v3.0 030, v2.4 031; **open upstream**: microsoft/openvmm#4625, a draft stacked on the closed microsoft/openvmm#4619; folds v2.4 371 (part)) -- 022 `cc2108a97` - vmcore: add device input quiesce and resume (v3.0 031, v2.4 032; **open upstream**: microsoft/openvmm#4626, stacked on microsoft/openvmm#4625) +- 021 `03291e1ea` - state_unit: add bounded quiesce transactions with rollback classification (v3.0 030, v2.4 031; **withdrawn**: microsoft/openvmm#4625 was closed in favor of microsoft/nvx#1260, and position 216 removes these transactions, so the next reorganization drops this commit; folds v2.4 371 (part)) +- 022 `cc2108a97` - vmcore: add device input quiesce and resume (v3.0 031, v2.4 032; **open upstream**: microsoft/openvmm#4626, stacked on microsoft/openvmm#4872, without the input gate's deadline, which position 216 removes here) - 023 `0692a461c` - openvmm_core: stop the devices before tearing down the VM (v3.0 032, v2.4 301) Retired by the v4.0 reorganization: diff --git a/openvmm b/openvmm index 9cec2b892..14dc4116e 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 9cec2b89245f41f2126f9a787c5e43a0d84b7154 +Subproject commit 14dc4116ec7e57567633c4b7b07892aca3a58522 diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index 6adba6bfe..dd15bf727 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -62,6 +62,7 @@ "smp": 21.2, "smp-snapshot": 26.1, "snapshot-core": 13.4, + "snapshot-rollback": 1.5, "snapshot-tiers": 6.8, "structured-outcome": 0.6, "time-abi-conformance": 0.7, @@ -98,7 +99,7 @@ ] }, { - "seconds": 267.8, + "seconds": 269.3, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -122,6 +123,7 @@ "sandbox-blocks", "scratch-snapshot", "smp", + "snapshot-rollback", "structured-outcome", "untiered-network-policy", "virtio-net", @@ -170,6 +172,7 @@ "smp": 22.6, "smp-snapshot": 28.5, "snapshot-core": 12.9, + "snapshot-rollback": 1.5, "snapshot-tiers": 6.3, "structured-outcome": 0.5, "time-abi-conformance": 0.8, @@ -181,7 +184,7 @@ }, "shards": [ { - "seconds": 248.8, + "seconds": 250.3, "scenarios": [ "command-line-limit", "console-snapshot", @@ -199,6 +202,7 @@ "smp", "smp-snapshot", "snapshot-core", + "snapshot-rollback", "snapshot-tiers", "time-host-correction", "untiered-network-policy" @@ -278,6 +282,7 @@ "smp": 30.9, "smp-snapshot": 46.6, "snapshot-core": 17.2, + "snapshot-rollback": 1.5, "snapshot-tiers": 12.0, "structured-outcome": 0.8, "time-abi-conformance": 1.6, @@ -313,7 +318,7 @@ ] }, { - "seconds": 356.0, + "seconds": 357.5, "scenarios": [ "command-line-limit", "console-exit", @@ -336,6 +341,7 @@ "private-network-ingress", "sandbox-blocks", "snapshot-core", + "snapshot-rollback", "structured-outcome", "time-abi-conformance", "time-host-correction", diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 8e7179478..c6e435601 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -139,6 +139,7 @@ "smp", "smp-snapshot", "snapshot-core", + "snapshot-rollback", "snapshot-tiers", "structured-outcome", "time-abi-conformance", @@ -8141,7 +8142,12 @@ def canceled_capture_checks() -> str: ) -def check_canceled_capture(output: bytes, command: Sequence[str]) -> None: +def check_canceled_capture( + output: bytes, + command: Sequence[str], + *, + context: str = "after the released snapshot request", +) -> None: """Check that a released snapshot request left the source as it was. The snapshot agent saves and overrides the stall detectors' settings @@ -8150,7 +8156,6 @@ def check_canceled_capture(output: bytes, command: Sequence[str]) -> None: source then reports a passing status at generation 0 with no restore, and RCU stall detection is no longer suppressed. """ - context = "after the released snapshot request" monitor = TimeAbiMonitor(command) try: monitor.feed(output) @@ -8362,6 +8367,171 @@ def run_snapshot_core( raise RuntimeError("restored clones reused a temporary-file ID") +SNAPSHOT_ROLLBACK_STATUS_PREFIX = b"NVX-SNAPSHOT-ROLLBACK-STATUS-" +SNAPSHOT_ROLLBACK_CONTINUED_MARKER = b"NVX-SNAPSHOT-ROLLBACK-CONTINUED" +SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX = b"NVX-SNAPSHOT-ROLLBACK-RESTORED-STATUS-" +SNAPSHOT_ROLLBACK_RESTORED_MARKER = b"NVX-SNAPSHOT-ROLLBACK-RESTORED" +CAPTURE_PROFILE_PHASE = re.compile( + rb"OPENVMM_SNAPSHOT_PROFILE_V1 operation=capture phase=([a-z_]+) " +) + + +def snapshot_request_line(status_prefix: bytes, done_marker: bytes) -> str: + """Return a console line that requests a snapshot, then prints + ``status_prefix`` with the request's exit status and, on a line of its + own, ``done_marker``. + + A quote pair splits each marker, so the console's echo of the line, which + the tty may wrap, never contains it. + """ + status = status_prefix.decode() + done = done_marker.decode() + return ( + f'nvx-snapshot; echo "{status[:4]}""{status[4:]}$?"; ' + f'echo "{done[:4]}""{done[4:]}"' + ) + + +def capture_profile_phases(output: bytes) -> list[str]: + """Return the phases of the capture profile records in ``output``, in the + order in which OpenVMM wrote them. + + OpenVMM writes a record and its newline separately, so console output can + share its line; the records are found wherever they are. + """ + return [match.group(1).decode() for match in CAPTURE_PROFILE_PHASE.finditer(output)] + + +def run_snapshot_rollback( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Fail a capture after a complete quiesce, and check that the source + resumes and can capture again. + + The snapshot destination's parent disappears after boot, so the capture + quiesces and saves the VM, and then fails to publish, which rolls it back: + the guest's request returns in the source, as for a request that OpenVMM + releases (``snapshot-core``), and the capture's profile records show the + quiesce and the save, but no publication. Once the parent is back, a + second request of the same source publishes a snapshot, whose restore + continues past that request. + """ + with tempfile.TemporaryDirectory(prefix="nvx-snapshot-rollback-") as temporary: + root = Path(temporary) + parent = root / "parent" + snapshot_path = parent / "snapshot" + parent.mkdir() + capture_command = [ + *workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + ), + "--snapshot-destination", + str(snapshot_path), + ] + # OpenVMM's own RAM backing would live in the destination's parent and + # keep it from going away, so the RAM has a file of its own. + memory = capture_command.index("--memory") + 1 + capture_command[memory] = f"size={memory_mib}M,file={root / 'memory.bin'}" + with OpenvmmProcess( + capture_command, + output_dir / "snapshot-rollback-capture.log", + environment={SNAPSHOT_PROFILE_ENV: "1"}, + ) as process: + process.wait_for(BOOT_MARKER, timeout) + parent.rmdir() + process.send_line( + snapshot_request_line( + SNAPSHOT_ROLLBACK_STATUS_PREFIX, SNAPSHOT_ROLLBACK_CONTINUED_MARKER + ) + ) + process.wait_for_line(SNAPSHOT_ROLLBACK_CONTINUED_MARKER, timeout) + failed = process.output + status = _single_marker_value(failed, SNAPSHOT_ROLLBACK_STATUS_PREFIX) + if status != b"0": + raise RuntimeError( + "the snapshot request that OpenVMM rolled back exited with " + f"{status.decode(errors='replace')}" + ) + phases = capture_profile_phases(failed) + if phases.count("quiesce") != 1 or phases.count("save_state") != 1: + raise RuntimeError( + "the failed capture did not quiesce and save the VM once " + f"before it returned: capture phases {phases}" + ) + if "publication" in phases or "publication_commit" in phases: + raise RuntimeError( + f"the failed capture published a snapshot: capture phases {phases}" + ) + if parent.exists(): + raise RuntimeError("the failed capture recreated its parent directory") + checks_start = len(process.output) + process.send_bytes(canceled_capture_checks().encode()) + process.wait_for_line( + CANCELED_CAPTURE_DONE_MARKER, max(timeout, STATUS_TIMEOUT_SECONDS) + ) + check_canceled_capture( + process.output[checks_start:], + capture_command, + context="after the rolled-back capture", + ) + parent.mkdir() + process.send_line( + snapshot_request_line( + SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX, + SNAPSHOT_ROLLBACK_RESTORED_MARKER, + ) + + "; nvx-exit 0" + ) + source = process.wait(timeout) + if source.returncode != 0: + raise RuntimeError( + f"snapshot source OpenVMM exited with {source.returncode}" + ) + if SNAPSHOT_ROLLBACK_RESTORED_MARKER in _output_lines(source.output): + raise RuntimeError("snapshot source crossed its terminal capture boundary") + phases = capture_profile_phases(source.output) + if phases.count("quiesce") != 2 or phases.count("publication_commit") != 1: + raise RuntimeError( + "the source did not quiesce twice and publish once: " + f"capture phases {phases}" + ) + if list(parent.iterdir()) != [snapshot_path]: + raise RuntimeError( + "the snapshot's parent holds more than the snapshot: " + f"{sorted(path.name for path in parent.iterdir())}" + ) + with OpenvmmProcess( + snapshot_restore_command(executable, backend, snapshot_path), + output_dir / "snapshot-rollback-restore.log", + ) as process: + process.wait_for_line(SNAPSHOT_ROLLBACK_RESTORED_MARKER, timeout) + restored = process.wait(timeout) + if restored.returncode != 0: + raise RuntimeError(f"snapshot restore exited with {restored.returncode}") + status = _single_marker_value( + restored.output, SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + ) + if status != b"0": + raise RuntimeError( + "the restored snapshot request exited with " + f"{status.decode(errors='replace')}" + ) + if _output_lines(restored.output).count(SNAPSHOT_ROLLBACK_RESTORED_MARKER) != 1: + raise RuntimeError("snapshot restore did not continue exactly once") + + def run_console_snapshot( executable: Path, kernel: Path, @@ -11308,6 +11478,20 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) + if "snapshot-rollback" in scenarios: + report.start( + "snapshot-rollback", + f"Running microVM snapshot rollback correctness on OpenVMM/{args.backend}", + ) + run_snapshot_rollback( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "snapshot-tiers" in scenarios: report.start( "snapshot-tiers", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 18e0edd36..d61da7316 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -7404,6 +7404,272 @@ def test_canceled_capture_requires_generation_zero_and_no_suppression(self): ) ) + def test_snapshot_request_line_prints_its_markers_whole(self): + status = microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX + done = microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER + line = microvm_tests.snapshot_request_line(status, done) + self.assertTrue(line.startswith("nvx-snapshot; ")) + self.assertNotIn("\n", line) + # The line never contains a whole marker, so the console's echo of + # it, wrapped or not, can't stand in for the guest's answers. + for marker in (status, done): + self.assertNotIn(marker.decode(), line) + shell = shutil.which("sh") + if shell is None: + self.skipTest("no POSIX shell") + for request, code in (("true", b"0"), ("false", b"1")): + with self.subTest(request=request): + answers = subprocess.run( + [shell, "-c", line.replace("nvx-snapshot", request, 1)], + capture_output=True, + check=True, + ).stdout + self.assertEqual(answers.splitlines(), [status + code, done]) + + def test_capture_profile_phases_find_records_that_share_a_line(self): + record = b"OPENVMM_SNAPSHOT_PROFILE_V1 operation=%s phase=%s exclusive=1 pid=7" + output = ( + record % (b"capture", b"input_gate") + + b"\n~ # " + + record % (b"capture", b"quiesce") + + b"guest bytes\r\n" + + record % (b"restore", b"artifact_open") + + b"\n" + + record % (b"capture", b"save_state") + + b"\n" + ) + self.assertEqual( + microvm_tests.capture_profile_phases(output), + ["input_gate", "quiesce", "save_state"], + ) + self.assertEqual(microvm_tests.capture_profile_phases(b"~ # quiesce\n"), []) + + def _run_snapshot_rollback( + self, + *, + status: int = 0, + failed_phases: Sequence[str] = ( + "input_gate", + "quiesce", + "save_state", + "memory_handle_flush", + ), + source_continues: bool = False, + ) -> list[tuple[str, object]]: + """Run snapshot-rollback against fake OpenVMM processes and return + what the scenario did, in order. + + The source's first request fails after `failed_phases` and returns + with `status`; its second publishes a snapshot, unless + `source_continues` makes it return in the source. + """ + events: list[tuple[str, object]] = [] + boot_command = [ + "openvmm", + "--hypervisor", + "kvm", + "--memory", + "128M", + "--kernel", + "vmlinux", + ] + + def profile(*phases: str) -> bytes: + return b"".join( + b"OPENVMM_SNAPSHOT_PROFILE_V1 operation=capture phase=%s " + b"exclusive=1 duration_ns=1 process_elapsed_ns=2 pid=3\n" + % phase.encode() + for phase in phases + ) + + class FakeProcess: + def __init__( + self, + command: Sequence[str], + log_path: Path, + *, + environment: Mapping[str, str] | None = None, + ) -> None: + self.command = list(command) + self.console = bytearray() + events.append(("start", log_path.name)) + if "--snapshot-destination" in command: + destination = command[command.index("--snapshot-destination") + 1] + self.destination: Path | None = Path(destination) + events.append(("environment", environment)) + events.append(("memory", command[command.index("--memory") + 1])) + else: + self.destination = None + restored = microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + self.console.extend( + restored + + b"0\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER + + b"\r\n" + ) + + def __enter__(self): + return self + + def __exit__( + self, + _exception_type: type[BaseException] | None, + _exception: BaseException | None, + _traceback: object | None, + ) -> None: + return None + + @property + def output(self) -> bytes: + return bytes(self.console) + + def wait_for(self, expected: bytes, _timeout: float) -> None: + events.append(("wait_for", expected)) + + def wait_for_line(self, expected: bytes, _timeout: float) -> None: + events.append(("wait_for_line", expected)) + + def send_line(self, line: str) -> None: + assert self.destination is not None + parent = self.destination.parent + events.append(("send_line", line)) + events.append( + ("parent", sorted(parent.iterdir()) if parent.exists() else None) + ) + if ( + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER.decode()[4:] + in line + ): + self.console.extend( + profile(*failed_phases) + + microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX + + str(status).encode() + + b"\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER + + b"\r\n" + ) + elif source_continues: + self.console.extend( + profile("input_gate", "quiesce", "save_state") + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + + b"0\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER + + b"\r\n" + ) + else: + self.destination.mkdir() + self.console.extend( + profile( + "input_gate", + "quiesce", + "save_state", + "publication_commit", + "publication", + ) + ) + + def send_bytes(self, data: bytes) -> None: + events.append(("send_bytes", data.decode())) + self.console.extend(_canceled_capture_output("kvm")) + + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: + events.append(("wait", None)) + return openvmm_process.OpenvmmProcessResult(0, bytes(self.console)) + + with ( + patch.object( + microvm_tests, "workload_boot_command", return_value=boot_command + ), + patch.object(microvm_tests, "OpenvmmProcess", side_effect=FakeProcess), + ): + microvm_tests.run_snapshot_rollback( + Path("openvmm"), + Path("vmlinux"), + Path("initramfs"), + "kvm", + memory_mib=128, + timeout=1, + output_dir=Path("logs"), + ) + return events + + def test_snapshot_rollback_fails_a_capture_after_its_quiesce_and_captures_again( + self, + ): + events = self._run_snapshot_rollback() + + memory = cast(str, dict(events)["memory"]) + self.assertTrue(memory.startswith("size=128M,file="), memory) + # The RAM file lies outside the destination's parent. + self.assertEqual( + Path(memory.removeprefix("size=128M,file=")).name, "memory.bin" + ) + first = microvm_tests.snapshot_request_line( + microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX, + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER, + ) + second = ( + microvm_tests.snapshot_request_line( + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX, + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER, + ) + + "; nvx-exit 0" + ) + self.assertEqual( + events, + [ + ("start", "snapshot-rollback-capture.log"), + ("environment", {"OPENVMM_STARTUP_PROFILE": "1"}), + ("memory", memory), + ("wait_for", microvm_tests.BOOT_MARKER), + # The parent is gone when the guest requests the first capture. + ("send_line", first), + ("parent", None), + ("wait_for_line", microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER), + ("send_bytes", microvm_tests.canceled_capture_checks()), + ("wait_for_line", microvm_tests.CANCELED_CAPTURE_DONE_MARKER), + # It is back, and empty, for the second. + ("send_line", second), + ("parent", []), + ("wait", None), + ("start", "snapshot-rollback-restore.log"), + ("wait_for_line", microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER), + ("wait", None), + ], + ) + + def test_snapshot_rollback_requires_a_rollback_after_a_complete_quiesce(self): + run = self._run_snapshot_rollback + cases: tuple[tuple[Callable[[], object], str], ...] = ( + ( + lambda: run(status=1), + "the snapshot request that OpenVMM rolled back exited with 1", + ), + ( + lambda: run(failed_phases=("input_gate", "output_flush")), + "the failed capture did not quiesce and save the VM once", + ), + ( + lambda: run( + failed_phases=( + "input_gate", + "quiesce", + "save_state", + "publication_commit", + ) + ), + "the failed capture published a snapshot", + ), + ( + lambda: run(source_continues=True), + "snapshot source crossed its terminal capture boundary", + ), + ) + for case, message in cases: + with self.subTest(message=message): + with self.assertRaisesRegex(RuntimeError, re.escape(message)): + case() + def test_snapshot_marker_parsers_require_single_well_formed_values(self): output = b"PREFIX-12\r\nPAIR-4-5\n" self.assertEqual(