From e347456144e8504054c2792d574cff475b649239 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 4 Nov 2026 14:04:07 -0800 Subject: [PATCH 1/3] test-microvm: roll back a capture that fails after its quiesce No scenario covered a capture that fails after a complete quiesce, which OpenVMM rolls back so that the source resumes: snapshot-core's request has no destination, so OpenVMM releases it before any quiesce. microsoft/nvx#1260, which removes the quiesce's deadline, makes that rollback the only way that a capture that quiesced the VM returns to its guest. The snapshot-rollback scenario gives the guest's RAM a file of its own, so that the snapshot destination's parent stays empty, and removes the parent after boot. The capture then quiesces and saves the VM, fails to publish, and rolls back. The scenario requires the request to return in the source with status 0, OpenVMM's capture profile to record one quiesce and one save but no publication, and the source to pass the checks that snapshot-core runs after its released request. It then restores the parent, and requires a second request of the same source to publish a snapshot and end the source, and the snapshot's restore to continue past that request once. The default suite runs it, with an estimate of its duration until CI's reports measure it, and doc/ci.md describes it. Tests: the request line hides its markers from the console's echo, the profile parser finds records that share a line with console output, and fake OpenVMM processes check the scenario's steps and that it fails when the request fails, when the failed capture did not quiesce or published, and when the second capture returns in the source. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 10 + scripts/nvx_tools/microvm_shards.json | 12 +- scripts/nvx_tools/microvm_tests.py | 188 +++++++++++++++++- scripts/test_microvm_tests.py | 266 ++++++++++++++++++++++++++ 4 files changed, 471 insertions(+), 5 deletions(-) diff --git a/doc/ci.md b/doc/ci.md index 934231e2b..2e81747bf 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -233,6 +233,16 @@ settings, and it must restore them when the request returns (`nvx-time cancel-capture`). Afterwards, `nvx-time status` must exit 0 with a passing boot line at generation 0 and no restore line, and `/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` must read 0. +The `snapshot-rollback` scenario fails a capture after its quiesce. The +guest's RAM has a file of its own, so the snapshot destination's parent holds +nothing, and it disappears after boot: the capture quiesces and saves the VM, +then fails to publish, and OpenVMM rolls it back. The request must return in +the source with status 0, after OpenVMM's capture profile records exactly one +quiesce and one save but no publication, and the source must pass the checks +that `snapshot-core` runs after its released request. Once the parent is back, +a second request of the same source must publish a snapshot and end the +source with status 0, leaving only the snapshot in the parent, and the +snapshot's restore must continue past that request exactly once. The `platform-zero-fill` scenario covers a platform restore whose `distro` image is shorter than the snapshot's and whose fresh scratch is absent. It captures a platform snapshot with a 1 MiB `distro` placeholder and a 1 MiB diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index 6adba6bfe..dd15bf727 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -62,6 +62,7 @@ "smp": 21.2, "smp-snapshot": 26.1, "snapshot-core": 13.4, + "snapshot-rollback": 1.5, "snapshot-tiers": 6.8, "structured-outcome": 0.6, "time-abi-conformance": 0.7, @@ -98,7 +99,7 @@ ] }, { - "seconds": 267.8, + "seconds": 269.3, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -122,6 +123,7 @@ "sandbox-blocks", "scratch-snapshot", "smp", + "snapshot-rollback", "structured-outcome", "untiered-network-policy", "virtio-net", @@ -170,6 +172,7 @@ "smp": 22.6, "smp-snapshot": 28.5, "snapshot-core": 12.9, + "snapshot-rollback": 1.5, "snapshot-tiers": 6.3, "structured-outcome": 0.5, "time-abi-conformance": 0.8, @@ -181,7 +184,7 @@ }, "shards": [ { - "seconds": 248.8, + "seconds": 250.3, "scenarios": [ "command-line-limit", "console-snapshot", @@ -199,6 +202,7 @@ "smp", "smp-snapshot", "snapshot-core", + "snapshot-rollback", "snapshot-tiers", "time-host-correction", "untiered-network-policy" @@ -278,6 +282,7 @@ "smp": 30.9, "smp-snapshot": 46.6, "snapshot-core": 17.2, + "snapshot-rollback": 1.5, "snapshot-tiers": 12.0, "structured-outcome": 0.8, "time-abi-conformance": 1.6, @@ -313,7 +318,7 @@ ] }, { - "seconds": 356.0, + "seconds": 357.5, "scenarios": [ "command-line-limit", "console-exit", @@ -336,6 +341,7 @@ "private-network-ingress", "sandbox-blocks", "snapshot-core", + "snapshot-rollback", "structured-outcome", "time-abi-conformance", "time-host-correction", diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 8e7179478..c6e435601 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -139,6 +139,7 @@ "smp", "smp-snapshot", "snapshot-core", + "snapshot-rollback", "snapshot-tiers", "structured-outcome", "time-abi-conformance", @@ -8141,7 +8142,12 @@ def canceled_capture_checks() -> str: ) -def check_canceled_capture(output: bytes, command: Sequence[str]) -> None: +def check_canceled_capture( + output: bytes, + command: Sequence[str], + *, + context: str = "after the released snapshot request", +) -> None: """Check that a released snapshot request left the source as it was. The snapshot agent saves and overrides the stall detectors' settings @@ -8150,7 +8156,6 @@ def check_canceled_capture(output: bytes, command: Sequence[str]) -> None: source then reports a passing status at generation 0 with no restore, and RCU stall detection is no longer suppressed. """ - context = "after the released snapshot request" monitor = TimeAbiMonitor(command) try: monitor.feed(output) @@ -8362,6 +8367,171 @@ def run_snapshot_core( raise RuntimeError("restored clones reused a temporary-file ID") +SNAPSHOT_ROLLBACK_STATUS_PREFIX = b"NVX-SNAPSHOT-ROLLBACK-STATUS-" +SNAPSHOT_ROLLBACK_CONTINUED_MARKER = b"NVX-SNAPSHOT-ROLLBACK-CONTINUED" +SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX = b"NVX-SNAPSHOT-ROLLBACK-RESTORED-STATUS-" +SNAPSHOT_ROLLBACK_RESTORED_MARKER = b"NVX-SNAPSHOT-ROLLBACK-RESTORED" +CAPTURE_PROFILE_PHASE = re.compile( + rb"OPENVMM_SNAPSHOT_PROFILE_V1 operation=capture phase=([a-z_]+) " +) + + +def snapshot_request_line(status_prefix: bytes, done_marker: bytes) -> str: + """Return a console line that requests a snapshot, then prints + ``status_prefix`` with the request's exit status and, on a line of its + own, ``done_marker``. + + A quote pair splits each marker, so the console's echo of the line, which + the tty may wrap, never contains it. + """ + status = status_prefix.decode() + done = done_marker.decode() + return ( + f'nvx-snapshot; echo "{status[:4]}""{status[4:]}$?"; ' + f'echo "{done[:4]}""{done[4:]}"' + ) + + +def capture_profile_phases(output: bytes) -> list[str]: + """Return the phases of the capture profile records in ``output``, in the + order in which OpenVMM wrote them. + + OpenVMM writes a record and its newline separately, so console output can + share its line; the records are found wherever they are. + """ + return [match.group(1).decode() for match in CAPTURE_PROFILE_PHASE.finditer(output)] + + +def run_snapshot_rollback( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Fail a capture after a complete quiesce, and check that the source + resumes and can capture again. + + The snapshot destination's parent disappears after boot, so the capture + quiesces and saves the VM, and then fails to publish, which rolls it back: + the guest's request returns in the source, as for a request that OpenVMM + releases (``snapshot-core``), and the capture's profile records show the + quiesce and the save, but no publication. Once the parent is back, a + second request of the same source publishes a snapshot, whose restore + continues past that request. + """ + with tempfile.TemporaryDirectory(prefix="nvx-snapshot-rollback-") as temporary: + root = Path(temporary) + parent = root / "parent" + snapshot_path = parent / "snapshot" + parent.mkdir() + capture_command = [ + *workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + ), + "--snapshot-destination", + str(snapshot_path), + ] + # OpenVMM's own RAM backing would live in the destination's parent and + # keep it from going away, so the RAM has a file of its own. + memory = capture_command.index("--memory") + 1 + capture_command[memory] = f"size={memory_mib}M,file={root / 'memory.bin'}" + with OpenvmmProcess( + capture_command, + output_dir / "snapshot-rollback-capture.log", + environment={SNAPSHOT_PROFILE_ENV: "1"}, + ) as process: + process.wait_for(BOOT_MARKER, timeout) + parent.rmdir() + process.send_line( + snapshot_request_line( + SNAPSHOT_ROLLBACK_STATUS_PREFIX, SNAPSHOT_ROLLBACK_CONTINUED_MARKER + ) + ) + process.wait_for_line(SNAPSHOT_ROLLBACK_CONTINUED_MARKER, timeout) + failed = process.output + status = _single_marker_value(failed, SNAPSHOT_ROLLBACK_STATUS_PREFIX) + if status != b"0": + raise RuntimeError( + "the snapshot request that OpenVMM rolled back exited with " + f"{status.decode(errors='replace')}" + ) + phases = capture_profile_phases(failed) + if phases.count("quiesce") != 1 or phases.count("save_state") != 1: + raise RuntimeError( + "the failed capture did not quiesce and save the VM once " + f"before it returned: capture phases {phases}" + ) + if "publication" in phases or "publication_commit" in phases: + raise RuntimeError( + f"the failed capture published a snapshot: capture phases {phases}" + ) + if parent.exists(): + raise RuntimeError("the failed capture recreated its parent directory") + checks_start = len(process.output) + process.send_bytes(canceled_capture_checks().encode()) + process.wait_for_line( + CANCELED_CAPTURE_DONE_MARKER, max(timeout, STATUS_TIMEOUT_SECONDS) + ) + check_canceled_capture( + process.output[checks_start:], + capture_command, + context="after the rolled-back capture", + ) + parent.mkdir() + process.send_line( + snapshot_request_line( + SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX, + SNAPSHOT_ROLLBACK_RESTORED_MARKER, + ) + + "; nvx-exit 0" + ) + source = process.wait(timeout) + if source.returncode != 0: + raise RuntimeError( + f"snapshot source OpenVMM exited with {source.returncode}" + ) + if SNAPSHOT_ROLLBACK_RESTORED_MARKER in _output_lines(source.output): + raise RuntimeError("snapshot source crossed its terminal capture boundary") + phases = capture_profile_phases(source.output) + if phases.count("quiesce") != 2 or phases.count("publication_commit") != 1: + raise RuntimeError( + "the source did not quiesce twice and publish once: " + f"capture phases {phases}" + ) + if list(parent.iterdir()) != [snapshot_path]: + raise RuntimeError( + "the snapshot's parent holds more than the snapshot: " + f"{sorted(path.name for path in parent.iterdir())}" + ) + with OpenvmmProcess( + snapshot_restore_command(executable, backend, snapshot_path), + output_dir / "snapshot-rollback-restore.log", + ) as process: + process.wait_for_line(SNAPSHOT_ROLLBACK_RESTORED_MARKER, timeout) + restored = process.wait(timeout) + if restored.returncode != 0: + raise RuntimeError(f"snapshot restore exited with {restored.returncode}") + status = _single_marker_value( + restored.output, SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + ) + if status != b"0": + raise RuntimeError( + "the restored snapshot request exited with " + f"{status.decode(errors='replace')}" + ) + if _output_lines(restored.output).count(SNAPSHOT_ROLLBACK_RESTORED_MARKER) != 1: + raise RuntimeError("snapshot restore did not continue exactly once") + + def run_console_snapshot( executable: Path, kernel: Path, @@ -11308,6 +11478,20 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) + if "snapshot-rollback" in scenarios: + report.start( + "snapshot-rollback", + f"Running microVM snapshot rollback correctness on OpenVMM/{args.backend}", + ) + run_snapshot_rollback( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "snapshot-tiers" in scenarios: report.start( "snapshot-tiers", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 18e0edd36..d61da7316 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -7404,6 +7404,272 @@ def test_canceled_capture_requires_generation_zero_and_no_suppression(self): ) ) + def test_snapshot_request_line_prints_its_markers_whole(self): + status = microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX + done = microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER + line = microvm_tests.snapshot_request_line(status, done) + self.assertTrue(line.startswith("nvx-snapshot; ")) + self.assertNotIn("\n", line) + # The line never contains a whole marker, so the console's echo of + # it, wrapped or not, can't stand in for the guest's answers. + for marker in (status, done): + self.assertNotIn(marker.decode(), line) + shell = shutil.which("sh") + if shell is None: + self.skipTest("no POSIX shell") + for request, code in (("true", b"0"), ("false", b"1")): + with self.subTest(request=request): + answers = subprocess.run( + [shell, "-c", line.replace("nvx-snapshot", request, 1)], + capture_output=True, + check=True, + ).stdout + self.assertEqual(answers.splitlines(), [status + code, done]) + + def test_capture_profile_phases_find_records_that_share_a_line(self): + record = b"OPENVMM_SNAPSHOT_PROFILE_V1 operation=%s phase=%s exclusive=1 pid=7" + output = ( + record % (b"capture", b"input_gate") + + b"\n~ # " + + record % (b"capture", b"quiesce") + + b"guest bytes\r\n" + + record % (b"restore", b"artifact_open") + + b"\n" + + record % (b"capture", b"save_state") + + b"\n" + ) + self.assertEqual( + microvm_tests.capture_profile_phases(output), + ["input_gate", "quiesce", "save_state"], + ) + self.assertEqual(microvm_tests.capture_profile_phases(b"~ # quiesce\n"), []) + + def _run_snapshot_rollback( + self, + *, + status: int = 0, + failed_phases: Sequence[str] = ( + "input_gate", + "quiesce", + "save_state", + "memory_handle_flush", + ), + source_continues: bool = False, + ) -> list[tuple[str, object]]: + """Run snapshot-rollback against fake OpenVMM processes and return + what the scenario did, in order. + + The source's first request fails after `failed_phases` and returns + with `status`; its second publishes a snapshot, unless + `source_continues` makes it return in the source. + """ + events: list[tuple[str, object]] = [] + boot_command = [ + "openvmm", + "--hypervisor", + "kvm", + "--memory", + "128M", + "--kernel", + "vmlinux", + ] + + def profile(*phases: str) -> bytes: + return b"".join( + b"OPENVMM_SNAPSHOT_PROFILE_V1 operation=capture phase=%s " + b"exclusive=1 duration_ns=1 process_elapsed_ns=2 pid=3\n" + % phase.encode() + for phase in phases + ) + + class FakeProcess: + def __init__( + self, + command: Sequence[str], + log_path: Path, + *, + environment: Mapping[str, str] | None = None, + ) -> None: + self.command = list(command) + self.console = bytearray() + events.append(("start", log_path.name)) + if "--snapshot-destination" in command: + destination = command[command.index("--snapshot-destination") + 1] + self.destination: Path | None = Path(destination) + events.append(("environment", environment)) + events.append(("memory", command[command.index("--memory") + 1])) + else: + self.destination = None + restored = microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + self.console.extend( + restored + + b"0\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER + + b"\r\n" + ) + + def __enter__(self): + return self + + def __exit__( + self, + _exception_type: type[BaseException] | None, + _exception: BaseException | None, + _traceback: object | None, + ) -> None: + return None + + @property + def output(self) -> bytes: + return bytes(self.console) + + def wait_for(self, expected: bytes, _timeout: float) -> None: + events.append(("wait_for", expected)) + + def wait_for_line(self, expected: bytes, _timeout: float) -> None: + events.append(("wait_for_line", expected)) + + def send_line(self, line: str) -> None: + assert self.destination is not None + parent = self.destination.parent + events.append(("send_line", line)) + events.append( + ("parent", sorted(parent.iterdir()) if parent.exists() else None) + ) + if ( + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER.decode()[4:] + in line + ): + self.console.extend( + profile(*failed_phases) + + microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX + + str(status).encode() + + b"\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER + + b"\r\n" + ) + elif source_continues: + self.console.extend( + profile("input_gate", "quiesce", "save_state") + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX + + b"0\r\n" + + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER + + b"\r\n" + ) + else: + self.destination.mkdir() + self.console.extend( + profile( + "input_gate", + "quiesce", + "save_state", + "publication_commit", + "publication", + ) + ) + + def send_bytes(self, data: bytes) -> None: + events.append(("send_bytes", data.decode())) + self.console.extend(_canceled_capture_output("kvm")) + + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: + events.append(("wait", None)) + return openvmm_process.OpenvmmProcessResult(0, bytes(self.console)) + + with ( + patch.object( + microvm_tests, "workload_boot_command", return_value=boot_command + ), + patch.object(microvm_tests, "OpenvmmProcess", side_effect=FakeProcess), + ): + microvm_tests.run_snapshot_rollback( + Path("openvmm"), + Path("vmlinux"), + Path("initramfs"), + "kvm", + memory_mib=128, + timeout=1, + output_dir=Path("logs"), + ) + return events + + def test_snapshot_rollback_fails_a_capture_after_its_quiesce_and_captures_again( + self, + ): + events = self._run_snapshot_rollback() + + memory = cast(str, dict(events)["memory"]) + self.assertTrue(memory.startswith("size=128M,file="), memory) + # The RAM file lies outside the destination's parent. + self.assertEqual( + Path(memory.removeprefix("size=128M,file=")).name, "memory.bin" + ) + first = microvm_tests.snapshot_request_line( + microvm_tests.SNAPSHOT_ROLLBACK_STATUS_PREFIX, + microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER, + ) + second = ( + microvm_tests.snapshot_request_line( + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_STATUS_PREFIX, + microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER, + ) + + "; nvx-exit 0" + ) + self.assertEqual( + events, + [ + ("start", "snapshot-rollback-capture.log"), + ("environment", {"OPENVMM_STARTUP_PROFILE": "1"}), + ("memory", memory), + ("wait_for", microvm_tests.BOOT_MARKER), + # The parent is gone when the guest requests the first capture. + ("send_line", first), + ("parent", None), + ("wait_for_line", microvm_tests.SNAPSHOT_ROLLBACK_CONTINUED_MARKER), + ("send_bytes", microvm_tests.canceled_capture_checks()), + ("wait_for_line", microvm_tests.CANCELED_CAPTURE_DONE_MARKER), + # It is back, and empty, for the second. + ("send_line", second), + ("parent", []), + ("wait", None), + ("start", "snapshot-rollback-restore.log"), + ("wait_for_line", microvm_tests.SNAPSHOT_ROLLBACK_RESTORED_MARKER), + ("wait", None), + ], + ) + + def test_snapshot_rollback_requires_a_rollback_after_a_complete_quiesce(self): + run = self._run_snapshot_rollback + cases: tuple[tuple[Callable[[], object], str], ...] = ( + ( + lambda: run(status=1), + "the snapshot request that OpenVMM rolled back exited with 1", + ), + ( + lambda: run(failed_phases=("input_gate", "output_flush")), + "the failed capture did not quiesce and save the VM once", + ), + ( + lambda: run( + failed_phases=( + "input_gate", + "quiesce", + "save_state", + "publication_commit", + ) + ), + "the failed capture published a snapshot", + ), + ( + lambda: run(source_continues=True), + "snapshot source crossed its terminal capture boundary", + ), + ) + for case, message in cases: + with self.subTest(message=message): + with self.assertRaisesRegex(RuntimeError, re.escape(message)): + case() + def test_snapshot_marker_parsers_require_single_well_formed_values(self): output = b"PREFIX-12\r\nPAIR-4-5\n" self.assertEqual( From 17869e0f8b7e30aa2c133e5470bcdd1a75f12dc7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 4 Nov 2026 18:19:59 -0800 Subject: [PATCH 2/3] openvmm: pin the snapshot quiesce without a deadline OpenVMM quiesced a guest-requested capture within --snapshot-quiesce-timeout-ms, 5 s by default, and terminated the source when a state unit missed the deadline; the same bound limited gating host input at the snapshot boundary and flushing console output before the capture. microsoft/nvx#1260 found that the deadline added no safety, since an unfinished quiesce never resumes the source and publishes nothing, and little liveness, since a VMM whose unit never stops runs until its launcher ends it anyway. It only made a slow quiesce fail, and caused the uncertain states of microsoft/openvmm#4625, which this change withdraws. The bounds of the input gate and of the console flush recovered nothing either: an expired gate released the guest no sooner, since its rollback waited for the unit whose answer it had stopped waiting for, and an expired flush saved the unsent output in the snapshot, which every restored VM then wrote again. Pin 14dc4116e, which microsoft/openvmm#4921 proposes for feature/microvm/dev: two commits on 9cec2b892, the current head of that branch. The first logs each state unit that has not answered a state change request after five seconds, and every five seconds until it answers, so that the log of a capture whose unit never stops names the unit. The second quiesces the capture with StateUnits::stop, without a deadline, and rolls back a failure that follows the quiesce with StateUnits::start, which stops the units that started again when one does not answer. Gating host input and flushing console output wait without a deadline too: a gate that a unit fails still rolls back and lets the guest continue, and only a failed rollback ends the VM. --snapshot-quiesce-timeout-ms goes, and the management RPC reserves field 3 with both of its names, quiesce_timeout_ms and boundary_timeout_ms. NVX never set either setting: the nvx crate's template store, the runtime adapter, and the test harness already bound every capture with a deadline that ends OpenVMM. The design now says that a capture has three optional settings, none of them a timeout (5.3.1); that neither the capture nor its input gate, console delivery, or quiesce has a deadline (5.3.3); that a failure after the quiesce rolls the capture back, while a capture that never completes ends with its launcher (5.3.3, 5.3.5, 5.3.6); that an input gate that fails, rather than one that is late, rolls back (5.3.5, 5.3.6); that the block queue's drain has no deadline of its own (3.4.5); and that the VMM rolls back a capture that fails after the quiesce rather than failing a quiesce (1.4.1). The control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/1.4-trust-boundaries.md | 5 ++- doc/design/3.4-block-storage.md | 4 +- doc/design/5.3-snapshot-protocol.md | 66 +++++++++++++++++++---------- openvmm | 2 +- 4 files changed, 51 insertions(+), 26 deletions(-) diff --git a/doc/design/1.4-trust-boundaries.md b/doc/design/1.4-trust-boundaries.md index 69afe4af6..372af6da6 100644 --- a/doc/design/1.4-trust-boundaries.md +++ b/doc/design/1.4-trust-boundaries.md @@ -48,8 +48,9 @@ fallback, from automatic CPU-profile selection to a host-derived CPU profile, warns at every cold boot that takes it. The VMM uses checked address arithmetic, bounded buffers and tables, typed -validation errors, rate-limited guest-triggerable logs, and fallible quiesce -and restore operations. Native I/O callbacks only record bounded state or +validation errors, rate-limited guest-triggerable logs, and fallible restore +operations, and it rolls back a capture that fails after the quiesce and +before its commit point. Native I/O callbacks only record bounded state or enqueue notifications; they never perform blocking snapshot or host-resource work. Malformed saved device state is rejected before device workers start, and no virtual processor runs after a partial restore failure. A failure after diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index 216f04845..a261ee360 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -98,5 +98,7 @@ Writable scratch follows one of two policies: Transport and queue progress are saved with the other device state. Stopping a queue prevents new descriptor intake and drains every accepted I/O to one -used-ring completion within the bounded snapshot quiesce timeout. The +used-ring completion. The drain has no deadline of its own: a backend that +never completes an I/O stalls the capture until its launcher ends the VMM; see +[Capture Transaction](5.3-snapshot-protocol.md#533-capture-transaction). The controller flushes and copies scratch only after that drain. diff --git a/doc/design/5.3-snapshot-protocol.md b/doc/design/5.3-snapshot-protocol.md index 8731b583e..81e62151f 100644 --- a/doc/design/5.3-snapshot-protocol.md +++ b/doc/design/5.3-snapshot-protocol.md @@ -33,7 +33,7 @@ A capture requires up to three settings: a restore must supply the same one; see [Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity). -A capture also accepts four optional settings: +A capture also accepts three optional settings: - **Memory backing file.** The file that backs guest RAM. When none is supplied, the VMM creates one at launch in the parent of the destination @@ -44,8 +44,6 @@ A capture also accepts four optional settings: reserves so that a restore can give the VM more memory. Without it, a restore cannot add memory; see [Memory Activation](5.5-restore-protocol.md#554-memory-activation). -- **Quiesce timeout.** The longest the VMM may spend quiescing the VM, - including draining block I/O. - **Scratch restore mode.** How a restore recreates [paired scratch](5.5-restore-protocol.md#552-memory-and-scratch-materialization): a private copy; a copy-on-write clone, only for a `workload-start` snapshot; @@ -186,22 +184,31 @@ sequenceDiagram Units-->>Controller: State, inventory, time contract, and CPU profile Controller->>Storage: Copy state, memory, and optional scratch Controller->>Storage: Publish with no-replace rename - Controller->>Worker: Terminate committed source + alt Published + Controller->>Worker: Terminate committed source + else Failed before the rename + Controller->>Worker: Roll back + Worker->>Units: Start in dependency order and resume host input + Worker-->>Guest: Continue execution + end end ``` **Figure 5.2.** Capture transaction. -The VMM performs the capture as one bounded transaction: +The VMM performs the capture as one transaction: 1. Gate host input and defer completion of the snapshot-port write. 2. Stop the virtual processor at the I/O boundary while completing the write, so saved state starts at the instruction immediately after the output. 3. Preflight the machine profile, destination, backend, external attachments, and snapshot eligibility; a missing destination or a rejected preflight - releases the boundary and lets the guest continue. + releases the boundary and lets the guest continue. Otherwise, deliver the + console output that the guest wrote before its request; output accepted + while no endpoint is connected stays in the snapshot, and the restored VM + writes it. 4. Quiesce the remaining device workers and VM time in dependency order, - including the bounded block queue drain. + including the block queue drain. 5. Establish the time contract: require that no LAPIC timer is periodic or in TSC-deadline mode and that PIT channel 0 is not counting periodically, take the capture anchor and host identities, and record the declared rates, CPU @@ -213,8 +220,22 @@ The VMM performs the capture as one bounded transaction: 8. Publish the directory with one no-replace rename. 9. Terminate the source worker and process after publication. -A failure in step 5 is a rollback-safe capture failure. Deriving `Q` never -fails a capture; when it cannot be derived, it is omitted with a warning. +A failure in steps 5 through 8, before the rename, rolls the capture back: +the VMM starts the stopped state units again in dependency order, resumes host +input, and releases the boundary, so the guest continues. The source +terminates instead when a unit does not start again, or when the VMM cannot +remove the staging directory's alias of guest RAM. Deriving `Q` never fails a +capture; when it cannot be derived, it is omitted with a warning. + +The transaction has no deadline of its own. Gating host input in step 1 and +the quiesce in step 4 wait for every device, and delivering the console output +in step 3 waits for its endpoint to read it. A device that never finishes, or +an endpoint that stops reading, stalls the capture, with nothing published and +the source never resumed, until the process that launched the VMM ends it; the +VMM logs each state unit that has not answered after five seconds, so that the +log of such a capture names it. Every launcher therefore bounds a capture with +a deadline that ends the VMM, as the nvx crate's template store does with the +deadline that creating or refreshing a template requires. ## 5.3.4 State-Unit Ordering @@ -261,16 +282,16 @@ flowchart LR ## 5.3.5 Commit Point The directory rename is the commit point. Before it, a capture failure exposes -no final snapshot, and the guest resumes only when rollback is known to be -valid. After it, the source never resumes. This gives the guest an observable -boundary: code after the snapshot request runs once in each restored process -and never in the successfully captured source. A capture driven through the -management RPC likewise halts the source at the committed boundary and -terminates its process, so clients observe the transport closing. - -Establishing the boundary is itself fallible. If host input cannot be gated, -the worker rolls the gate back and releases the write so that the guest -continues; if that rollback is uncertain, the worker terminates. Once +no final snapshot, and the guest resumes only once the VMM has started the +stopped state units again. After it, the source never resumes. This gives the +guest an observable boundary: code after the snapshot request runs once in +each restored process and never in the successfully captured source. A capture +driven through the management RPC likewise halts the source at the committed +boundary and terminates its process, so clients observe the transport closing. + +Establishing the boundary is itself fallible. If a device fails to gate its +host input, the worker rolls the gate back and releases the write so that the +guest continues; if that rollback fails, the worker terminates. Once processor stopping has begun, a boundary failure is terminal: host input stays gated, and the VM is torn down rather than resumed. The post-restore acknowledgement boundary follows the same rule. @@ -284,9 +305,10 @@ As Table 5.3 shows, a published snapshot never coexists with a running source. | Outcome | Source | Snapshot | | --- | --- | --- | | Committed capture | Terminated | Published | -| Missing destination or rejected preflight | Continues | None | -| Failure before the commit point, with rollback known to be valid | Continues | None | -| Failure before the commit point, with rollback not known to be valid | Terminated | None | +| Missing destination, rejected preflight, or failed input gate | Continues | None | +| Failure after the quiesce, before the commit point, that rolls back | Continues | None | +| Failure before the commit point that cannot roll back | Terminated | None | +| Input gate, console delivery, or quiesce that never completes | Ended by its launcher | None | After a commit, execution continues only in virtual machines restored from the snapshot, either as clones or as one resumed instance; see diff --git a/openvmm b/openvmm index 9cec2b892..14dc4116e 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 9cec2b89245f41f2126f9a787c5e43a0d84b7154 +Subproject commit 14dc4116ec7e57567633c4b7b07892aca3a58522 From 40187de3248e7a147b1fc59cf65e49763fdddf25 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 4 Nov 2026 18:22:57 -0800 Subject: [PATCH 3/3] doc: map the snapshot quiesce without a deadline in the OpenVMM roadmap The NVX pin moves to 14dc4116e, position 216, 81 commits past the mapped tip. Positions 215 and 216, which microsoft/openvmm#4921 proposes for feature/microvm/dev on its head, 9cec2b892, where microsoft/openvmm#4916 landed position 214 by fast-forward, are unassigned: position 215 logs the state units that a state change still waits for, and position 216 captures a microVM snapshot without a deadline, quiescing with StateUnits::stop and rolling back a failure after the quiesce with the start rollback of position 018. Position 216 removes the bounded quiesce transactions that PR09's position 021 adds, and the uncertain quiesce that PR17's positions 076 and 080 handle, so the roadmap marks position 021 withdrawn: microsoft/nvx#1260 closes microsoft/openvmm#4625, and the next reorganization drops the commit. It also removes the capture's other bounds: the timeout of the input gate of PR09's position 022, which PR17's positions 075, 076, and 080 and PR20's position 099 pass on, and that of the console flush of PR24's position 123, with the CLI option of PR17's position 080 that set both, and it reserves the management RPC's field of PR25's position 128. PR09's upstream pull requests are now microsoft/openvmm#4872, which proposes position 018 with review changes, microsoft/openvmm#4624, and microsoft/openvmm#4626, which builds on microsoft/openvmm#4872 and gates input without a deadline too. microsoft/openvmm#4937 also builds on it and proposes no chain commit: it makes `StateUnits::stop` fail instead of panicking when a unit's task has ended, so a rebase onto it adapts the chain's own callers of `StateUnits::stop`. Position 215 conflicts on upstream main only where its test module joins those of positions 018, 020, and 021; position 216 conflicts in all 24 of its files, 18 of which upstream main lacks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 79 ++++++++++++++++++++++----------- 1 file changed, 54 insertions(+), 25 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 3fb7c63f7..82c9a07c4 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -7,18 +7,18 @@ `b2676222b383bf50c7022f6a5983920421469a47`; see [Upstream status](#upstream-status). - Mapped fork tip: `3cdf4fe3834cfb90b9ee685875493bc9bff1b7a4`, position 135. The graph, PR summary, and commit mapping below cover positions 001-135 and the PR20 fixup at position 152. -- NVX's pinned OpenVMM revision: `9cec2b89245f41f2126f9a787c5e43a0d84b7154`, position 214. - Position 214, which microsoft/openvmm#4916 proposes for `feature/microvm/dev`, is one commit on - position 213, `4713c539a`, the head of `feature/microvm/dev`. microsoft/openvmm#4894, - microsoft/openvmm#4886, and microsoft/openvmm#4887 landed positions 211-213 there by - fast-forward, on position 210, `eda13d80a`, the head of the `feature/microvm/release/v4.0` - branch of microsoft/openvmm. The v4.0 reorganization moved the 205 commits of - `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto `c6d3da0ff`. It - dropped the eight commits whose changes upstream had landed or superseded, retired the fallible - device start that microsoft/nvx#1232 replaces with awaited device restoration, folded that - restoration into the commits that it changes, and added 14 commits that give each ABI that the - chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The - pin is 79 commits past the mapped tip on the same first-parent chain. Positions 136-214, except +- NVX's pinned OpenVMM revision: `14dc4116ec7e57567633c4b7b07892aca3a58522`, position 216. Positions + 215 and 216, which microsoft/openvmm#4921 proposes for `feature/microvm/dev`, are two commits on + position 214, `9cec2b892`, the head of `feature/microvm/dev`. microsoft/openvmm#4916 landed + position 214 there by fast-forward, as microsoft/openvmm#4894, microsoft/openvmm#4886, and + microsoft/openvmm#4887 landed positions 211-213 on position 210, `eda13d80a`, the head of the + `feature/microvm/release/v4.0` branch of microsoft/openvmm. The v4.0 reorganization moved the 205 + commits of `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto + `c6d3da0ff`. It dropped the eight commits whose changes upstream had landed or superseded, retired + the fallible device start that microsoft/nvx#1232 replaces with awaited device restoration, folded + that restoration into the commits that it changes, and added 14 commits that give each ABI that + the chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The + pin is 81 commits past the mapped tip on the same first-parent chain. Positions 136-216, except the PR20 fixup at position 152, are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `ca6b2b177da5e33ae1bcac72986b31d6b039811c` (2026-11-02), one commit past the @@ -174,8 +174,8 @@ and 023 is in an open upstream pull request; the commits that upstream has since [Landed upstream](#landed-upstream), and PR08 has landed whole. Each pull request carries one commit, except microsoft/openvmm#4554, which carries positions 055-057 after the `boot_params` field that microsoft/openvmm#4661 has since landed; its ELF and initrd validation was split out -during review and landed as microsoft/openvmm#4667 (`a87971184`). One of the 21 pull requests, -microsoft/openvmm#4625, is a draft, and none has an approving review yet. +during review and landed as microsoft/openvmm#4667 (`a87971184`). One of those 21 pull requests, +microsoft/openvmm#4625, was a draft, and none had an approving review yet. microsoft/openvmm#4451, which proposed position 001, was closed without merging, and neither the remainder of position 006 nor position 023, which the v3.0 reorganization moved into PR09, has been proposed. microsoft/openvmm#4619, microsoft/openvmm#4622, and microsoft/openvmm#4623, which @@ -185,6 +185,17 @@ position 027 is retired, and position 019's resume stays fallible; see [The v4.0 reorganization](#the-v40-reorganization). PR10-PR13 and PR15-PR27 have no upstream pull requests yet. +On 2026-11-04, microsoft/nvx#1260 withdrew position 021 from PR09, since the deadline of the +snapshot quiesce added no safety: microsoft/openvmm#4625, which proposed position 021's bounded +quiesce transactions, was closed, and position 216 removes them, so the next reorganization drops +position 021. microsoft/openvmm#4872, opened on 2026-11-03, proposes position 018's rollback of a +partial start, with review changes that the chain lacks, and microsoft/openvmm#4626, which proposes +position 022's input gate without the deadline that position 216 removes from it, builds on it. So +does microsoft/openvmm#4937, which proposes no chain commit: it makes `StateUnits::stop` fail +instead of panicking when a unit's task has ended, so a rebase onto it adapts the chain's own +callers of `StateUnits::stop`, in the microVM worker's `dispatch/microvm.rs` and +`dispatch/restore.rs`. + | Roadmap PR | Positions | Upstream pull requests, in position order | | --- | ---: | --- | | PR01 | 002-005 | microsoft/openvmm#4595, microsoft/openvmm#4618, microsoft/openvmm#4598, microsoft/openvmm#4615 | @@ -194,13 +205,12 @@ requests yet. | PR05 | 011-013 | microsoft/openvmm#4536, microsoft/openvmm#4607, microsoft/openvmm#4613 | | PR06 | 014-016 | microsoft/openvmm#4597, microsoft/openvmm#4605, microsoft/openvmm#4609 | | PR07 | 017 | microsoft/openvmm#4600 | -| PR09 | 020-022 | microsoft/openvmm#4624, microsoft/openvmm#4625, microsoft/openvmm#4626 | +| PR09 | 018, 020, 022 | microsoft/openvmm#4872, microsoft/openvmm#4624, microsoft/openvmm#4626 | | PR14 | 055-058 | microsoft/openvmm#4554 (055-057), microsoft/openvmm#4611 | Stacked pull requests carry their prerequisites' commits: microsoft/openvmm#4609 builds on microsoft/openvmm#4585, which has merged; microsoft/openvmm#4613 builds on microsoft/openvmm#4536; -and microsoft/openvmm#4625 builds on the closed microsoft/openvmm#4619, as microsoft/openvmm#4626 -builds on microsoft/openvmm#4625, so both need a rebase that drops the fallible unit start. +and microsoft/openvmm#4626 builds on microsoft/openvmm#4872. Most of these pull requests have gained review commits that the fork chain does not carry. When one merges, the next rebase should drop its fork commit, as the `microvm/v2.2`, `feature/microvm/release/v2.3`, and `feature/microvm/release/v2.4` rebases and the v3.0 and v4.0 @@ -647,8 +657,8 @@ c6d3da0ff -> sandbox block roles [192] -> virtio feature masks [193-194] -> IOAPIC interrupt lines [195-196] -> ABI restart [197-210] -> boot console exit drain [211] -> portb without host input [212-213] - -> long Windows paths [214] - -> 9cec2b892 (NVX pin) + -> long Windows paths [214] -> snapshot quiesce without a deadline [215-216] + -> 14dc4116e (NVX pin) ``` For upstream review, PR01-PR07 and PR14 can be rebased independently onto current upstream main, @@ -664,12 +674,12 @@ see [The v3.0 reorganization](#the-v30-reorganization). ## Commits after the mapped tip -The NVX pin extends the chain past position 135 without merges. Of the 79 later commits, position +The NVX pin extends the chain past position 135 without merges. Of the 81 later commits, position 152 is a PR20 fixup, and the others are not yet classified or assigned to PRs. Each of positions 136-191 combines the v2.4 commits that the table lists, as v3.0 positions 145-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 192-196 followed the v3.0 reorganization, positions 197-210 are the ABI restart of the v4.0 -reorganization, and positions 211-214 followed it, so they combine none. +reorganization, and positions 211-216 followed it, so they combine none. | Position | Commit | v3.0 position | v2.4 positions | Topic | | ---: | --- | ---: | --- | --- | @@ -752,6 +762,8 @@ reorganization, and positions 211-214 followed it, so they combine none. | 212 | `de29849a4` | - | - | vmm_tests: drive the microVM guests without portb input (microsoft/openvmm#4886) | | 213 | `4713c539a` | - | - | microVM: remove portb's host input (microsoft/openvmm#4887) | | 214 | `9cec2b892` | - | - | pal: support long paths in secure file and directory creation (microsoft/openvmm#4916) | +| 215 | `1108c05c7` | - | - | state_unit: log the units that a state change waits for (microsoft/openvmm#4921) | +| 216 | `14dc4116e` | - | - | microVM: quiesce snapshots without a deadline (microsoft/openvmm#4921) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -788,7 +800,18 @@ then removes that input: the receive path of the portb console of PR15's positio PR20's position 098 shares with the restore records and PR20's position 099 gates, and the console input that PR15's position 068 gives the device on the CLI. Since those reads also noticed a departed peer of a listening endpoint, position 213 has the device poll the backend for -it instead. Assigning the later +it instead. Position 216 removes the bounded quiesce transactions that PR09's position 021 adds, +so that the next reorganization drops position 021, and the uncertain quiesce that PR17's +positions 076 and 080 handle in the worker and the controller, whose capture now quiesces and +rolls back with the start rollback of PR09's position 018. It also removes the capture's other +bounds. The input gate of PR09's position 022, whose module it makes private, loses its timeout, +and so do the snapshot-request port of PR17's position 075, its handles in OpenVMM and petri from +PR17's position 080, the worker of PR17's position 076, and the post-restore input gate of PR20's +position 099, which passed it on; the console flush of PR24's position 123 loses its own. The CLI +option of PR17's position 080 that set both goes, and the management RPC's field of PR25's +position 128 is reserved. The dispatcher test of PR17's position 077 builds the RPCs and the port +without durations, the boot console flush test of position 211 calls the flush without one, and +the input gate's tests reuse the log of position 215's test. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 182, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -846,6 +869,12 @@ publication file, whose new tests use the paired scratch of PR19's position 096 RAM of PR11's position 038. Its `pal` change alone applies to upstream main without conflicts, and `pal` builds there and passes its tests on Windows, so that change can go upstream on its own. +Position 215 changes only `state_unit`'s state change requests and its tests: replaying it on +upstream main conflicts only where its test module joins those of positions 018, 020, and 021 in +`lib.rs`. Position 216 changes the microVM capture of PR09, PR17, PR20, PR24, and PR25, its CLI +and management RPC, and their Guide pages: replaying it on upstream main conflicts in all 24 of +its files, 18 of which upstream main lacks, so it follows those PRs upstream. + ## PR summary Each count is of v4.0 commits, which include the fixups that the v3.0 reorganization folded into @@ -1040,11 +1069,11 @@ Landed upstream and dropped from the chain:
PR09 - Fallible and bounded VM lifecycle (U) - 6 commits + 1 retired -- 018 `a1a72fa96` - state_unit: roll back a partial start instead of panicking (v3.0 026, v2.4 027; **closed upstream**: microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) +- 018 `a1a72fa96` - state_unit: roll back a partial start instead of panicking (v3.0 026, v2.4 027; **open upstream**: microsoft/openvmm#4872, with review changes that the chain lacks; microsoft/openvmm#4619 proposed the fallible unit start that the v4.0 reorganization retired, in favor of microsoft/nvx#1232; see [The v4.0 reorganization](#the-v40-reorganization)) - 019 `4b98c153d` - openvmm: make VM resume fallible (v3.0 028, v2.4 029; **closed upstream**: microsoft/openvmm#4623 was closed in favor of microsoft/nvx#1232, but the resume stays fallible: the post-restore input gate, the restore readiness event, and the time ABI's TSC offset measurement fail it) - 020 `6fedcc054` - state_unit: record the exact unit inventory in saved state (v3.0 029, v2.4 030; **open upstream**: microsoft/openvmm#4624) -- 021 `03291e1ea` - state_unit: add bounded quiesce transactions with rollback classification (v3.0 030, v2.4 031; **open upstream**: microsoft/openvmm#4625, a draft stacked on the closed microsoft/openvmm#4619; folds v2.4 371 (part)) -- 022 `cc2108a97` - vmcore: add device input quiesce and resume (v3.0 031, v2.4 032; **open upstream**: microsoft/openvmm#4626, stacked on microsoft/openvmm#4625) +- 021 `03291e1ea` - state_unit: add bounded quiesce transactions with rollback classification (v3.0 030, v2.4 031; **withdrawn**: microsoft/openvmm#4625 was closed in favor of microsoft/nvx#1260, and position 216 removes these transactions, so the next reorganization drops this commit; folds v2.4 371 (part)) +- 022 `cc2108a97` - vmcore: add device input quiesce and resume (v3.0 031, v2.4 032; **open upstream**: microsoft/openvmm#4626, stacked on microsoft/openvmm#4872, without the input gate's deadline, which position 216 removes here) - 023 `0692a461c` - openvmm_core: stop the devices before tearing down the VM (v3.0 032, v2.4 301) Retired by the v4.0 reorganization: