Repository navigation
[DO NOT REVIEW] microvm: enable governed ARM64 cold boot on WHP - #5045
Gautam Ahuja (ahuja-gautam) wants to merge 272 commits into
Conversation
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
|
@microsoft-github-policy-service agree company="Microsoft" |
|
Clarifying the unsafe-code notice: it scans whole modified Rust files. Relative to the rebased PR's base ( AI-authored clarification, posted at the author's request. |
1d6ddec to
6036d7e
Compare
…frequency MSHV partitions did not report their TSC frequency, so guests had to calibrate it, and nothing verified that a snapshot is restored on a host with the TSC frequency that it was saved with. The TSC of each VP is also restored at a different host time, so the counters of a restored SMP guest drift apart. Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and implement the partition clock hooks: report the TSC and local APIC timer frequencies, reject a requested TSC frequency that differs from that of the partition, since MSHV cannot change it, and, after a snapshot downtime, freeze partition time and align the counters of all VPs to the advanced BSP counter. Hide the TSC-deadline timer from the guest, in both the processor features of the partition and CPUID, since MSHV does not reliably deliver its events to direct-boot guests, and report that ordinary state access does not freeze the partition clock. It also includes these changes: - virt_mshv: adapt the chain to upstream's nested virtualization Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…erent WHP partitions did not report their TSC frequency, so guests had to calibrate it, and nothing verified that a snapshot is restored on a host with the TSC frequency that it was saved with. The TSC of each VP is also restored at a different host time, so the counters of a restored SMP guest disagree with each other. Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and implement the partition clock hooks: report the TSC frequency and the local APIC timer frequency, reject a requested TSC frequency that differs from that of the partition, and, after a snapshot downtime, align the counters of all VPs of an SMP partition with partition time suspended. To keep TSC reads coherent across the VPs, a restored SMP partition that does not use nested virtualization then derives the TSC of each VP from the partition-wide reference time. While this restored-TSC clock is active, WHP exits on RDTSC and on accesses to IA32_TSC, TSC_ADJUST, and IA32_TSC_DEADLINE, and the VP state accessors read and restore the TSC through the clock. A VP whose guest writes its TSC or changes its TSC_ADJUST leaves the clock and gets the native WHP semantics again, and resetting the partition disables the clock. Read IA32_TSC_DEADLINE before the APIC state when saving a VP, so that the deadline cannot expire between the two reads. Add the WHP properties and registers that this needs, and report a failure to inject a general protection fault from an MSR exit instead of panicking. It also includes these changes: - virt_whp: query the TSC frequency for the aarch64 hv1 emulator - virt: adopt upstream's TscDeadline VP state element Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
When a VM is restored after it was stopped for some time, the TSC of each VP and its local APIC timer continue from their saved values, so the guest does not observe the elapsed time and its timers fire late. Add `PartitionUnit::advance_tsc`, which advances the TSC of every stopped VP by the downtime at the guest TSC frequency. It also advances the local APIC timer of each VP at the APIC timer frequency, when the backend reports one, and fires an armed TSC deadline that the advanced TSC passed. Each VP adjusts its TSC through `Processor::advance_tsc`, reprograms its APIC and TSC deadline, and then verifies that its counter advanced by at least the requested amount, since a backend could discard the adjustment. The VP set tests' `TestVp` implements the new `ControlVp` method as unreachable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Measuring where snapshot capture and restore spend their time requires timing individual lifecycle phases in several OpenVMM components and processes, and correlating the results, without changing the behavior or the output of ordinary runs. Add a `profile` module to `openvmm_defs`. When the `OPENVMM_STARTUP_PROFILE` environment variable is set to a nonempty value other than `0`, a `ProfileSpan` measures an exclusive phase or a cumulative milestone and writes one machine-readable `OPENVMM_SNAPSHOT_PROFILE_V1` record to stderr, with its duration, the time since process start, the process ID, and optional byte and fault counters. Otherwise spans are inert. OpenVMM initializes the process-relative clock at startup; the lifecycle phases add their spans as they are implemented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ecoding The snapshot manifest identified its format only by a version number: nothing marked the file as an OpenVMM snapshot, or recorded which saved-state schema `state.bin` holds and how long it is. Reading a snapshot also decoded `manifest.bin` and read `state.bin` without any size limit. Introduce manifest version 3, which adds a format magic, the saved-state schema version and protobuf root type, and the length of `state.bin`, and move the format definitions and their validation into a `format` module. `SnapshotManifest` implements `Default` with the current format identifiers, and the callers that build manifests take the fields that they do not set from it. Manifests of the earlier version 2 format, which records SHA-256 digests of `state.bin` and `memory.bin` instead, are accepted as well; version 1 manifests are no longer accepted. Writing a snapshot records the length of `state.bin` and rejects saved state larger than 256 MiB. Reading a snapshot limits `manifest.bin` to 1 MiB, validates the manifest format before it reads `state.bin`, and requires `state.bin` to have the recorded length. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Saving a snapshot created the destination directory, wrote the manifest and the saved state into it, and hard-linked the memory backing file as `memory.bin`. A failure could leave a partial snapshot at the destination, nothing was flushed to stable storage, an existing directory was silently reused, and the hard link made the snapshot an alias of live guest RAM that any later write to the backing file corrupted. The backing file also had to be on the same filesystem as the snapshot. Publish snapshots through a private, uniquely named staging directory next to the destination. `state.bin`, `memory.bin`, and `manifest.bin` are written and flushed there, the staging directory is flushed, and it is then renamed to the destination without replacing an existing path. The rename is the commit point, after which the parent directory is flushed. A failure before the commit removes the staging directory and leaves the destination absent; `SnapshotWriteError` distinguishes it from a committed snapshot whose parent directory could not be flushed. `memory.bin` is now an independent copy of guest RAM, made from an open handle so that replacing the path of the backing file cannot substitute other bytes, and `write_snapshot_from_memory_file` accepts that handle directly. The copy clones the file when the filesystem supports it, and otherwise copies only allocated ranges or nonzero data on Linux and copies densely on Windows. The backing file must have the length that the manifest records. Add the snapshot file-system helpers that this needs: private directories, no-replace renames, directory flushes, regular-file access that rejects Windows reparse points, and sparse-aware copies. Update the Guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Copying guest RAM into `memory.bin` keeps a snapshot independent of the source VM, but it doubles the storage and the capture time for a VM whose backing file OpenVMM itself creates and owns, and whose source is terminated after capture anyway. Add `write_snapshot_from_owned_memory_file`, which publishes such RAM by hard-linking the exact open backing handle into the staging directory as `memory.bin`, after the state and the manifest are written and the handle is flushed. Linux links the descriptor with `linkat(AT_EMPTY_PATH)` or through `/proc/self/fd`, and Windows uses a handle-relative link; the link is then opened relative to the staging directory and must have the device and inode or file ID and the length of the source. Filesystems that cannot link fall back to the independent copy. Once the link may exist, the staging directory aliases live RAM: a failure removes it and flushes its parent before it reports a rollback-safe error, and a failed cleanup is reported as `SnapshotWriteError::CleanupUncertain`, which requires the source to terminate instead of resuming. Size new memory backing files through `initialize_snapshot_memory_backing_file`, which keeps them sparse except on Windows, where a dense file avoids slow copy-on-write faults, and add `open_memory_backing_file_handle`, which returns the sized backing file itself so that a caller can keep the exact handle for publication. It also includes these changes: - microVM: fix the macOS clippy build Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Restoring a snapshot maps `memory.bin` as guest RAM, but a file backing could only be mapped shared, so every guest write would go into the snapshot artifact, and the snapshot could be used only once. Add a file mapping mode to the memory manager. With `FileMappingMode::CopyOnWrite`, an existing backing file is mapped with private copy-on-write views, so the guest reads the contents of the file while its writes stay private to the process and never reach the file. Copy-on-write RAM requires an existing file backing, and it is not offered as a shared-memory backing to other processes. The default mode is shared, as before. Also let the memory manager count guest-memory faults and the bytes populated while resolving them, for diagnosing restore performance, and report the totals through `MemoryManager::fault_counters`. It also includes these changes: - openvmm: create copy-on-write mappables with upstream's signature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…he directory Restore read `manifest.bin` and `state.bin` by joining each name to the snapshot path, and the caller reopened `memory.bin` the same way. Every open resolved the pathname again, following symbolic links, so renaming or replacing the directory, or one of its files, between two opens could combine artifacts of different generations. Unexpected files in the directory were ignored, and the length of `memory.bin` was checked only by the caller. Add `OpenedSnapshot`, which opens the snapshot directory once and then opens each artifact relative to that directory handle. Linux uses `openat` with `O_NOFOLLOW`; Windows uses handle-relative, read-only opens that share only reads and reject reparse points. `OpenedSnapshot::open` reads the bounded manifest and saved state through the opened handles, requires the directory to contain exactly `manifest.bin`, `state.bin`, and `memory.bin` as regular files, checks the exact artifact lengths, and records the generation of the opened memory file: its identity, length, and, on Linux, its modification time. Accessors return the manifest and state, check that the memory generation is unchanged, duplicate the exact memory handle for a later mapping, and report logical and allocated artifact sizes for profiling. `into_parts` moves the open directory and artifact handles into the new `SnapshotRestoreGuards`, so a later caller can keep the generation open for the lifetime of a restored VM. The Linux generation leaves out the change time, which a change of the file's links, mode, or owner moves without changing its content. The source of a capture with automatic RAM publishes `memory.bin` as a hard link of its RAM file, whose temporary name it removes as it tears down, after the commit, and a restore that opened the snapshot before then must still find the generation that it opened. A write still moves the modification time, so a same-length change of the memory between the open and the mapping is still found. `restore::read_snapshot` now takes the expected guest memory size and validates all three artifacts through an `OpenedSnapshot`, and `read_snapshot_manifest`, `read_snapshot_with_memory`, and `read_snapshot_artifacts_with_memory` cover manifest-only reads and reads that also return the exact memory handle. The top-level `read_snapshot` re-export is removed; its callers use `restore::read_snapshot`. The command-line restore path still reopens `memory.bin` as shared, writable guest RAM, until restores map it privately copy-on-write. Document restore validation and the limits of the local format: it does not authenticate same-length payload changes, and Linux file descriptors do not provide mandatory write exclusion. Tests: a restore opens a snapshot whose `memory.bin` links the owned RAM file that it was published from, and that file's other name is then removed, which leaves `memory.bin` with one link and a new change time but its modification time; the restore still validates and duplicates the memory that it opened and reads back its content. It also includes these changes: - openvmm_helpers: leave the change time out of snapshot file generations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ion handles `--restore-snapshot` reopened `memory.bin` by path for reading and writing and mapped it as shared guest RAM. Every guest write after restore went into the snapshot artifact, so a snapshot could be restored only once, and copying the directory first was the only way to reuse it. The restored VM also held no handle to the generation it had validated, so the artifacts could be replaced while the VM ran. Restore now opens the snapshot through an `OpenedSnapshot` before the VM configuration is built, in the new `snapshot_restore` module of `openvmm_entry`. Just before the VM worker launches, the opened generation is validated against the VM configuration, a duplicate of the exact opened memory handle becomes a private copy-on-write memory handle (`file_to_copy_on_write_memory_fd`), and the memory generation is checked again after the mapping handle is created. The worker receives the handle with the new `shared_memory_copy_on_write` flag and maps guest RAM privately copy-on-write, so guest writes stay in the process and never reach `memory.bin`, and one snapshot can be restored any number of times. The open directory and artifact handles move to the worker as the new `snapshot_restore_guards` parameter, and the worker keeps them until it exits, which keeps the generation open for the lifetime of the restored VM; on Windows the read-only handles also deny writes to it. The worker takes its restore inputs from the parameters in the new `dispatch/restore` module, which also takes over the saved-state inventory check. The management and petri launches pass the defaults: a shared mapping and no guards. With `OPENVMM_STARTUP_PROFILE` set, restore records the `restore` phases `artifact_open`, `artifact_prepare`, and `cow_section_create`, and a launch records the `startup` milestones `worker_launch` and `worker_construct` and the worker phases `partition_prototype`, `cow_map_view`, `partition_build`, `gpa_registration`, and `partition_finalize`. Document that restores are private and repeatable, and describe the handles that restore retains. It also includes these changes: - openvmm: create copy-on-write mappables with upstream's signature - openvmm_helpers: leave the change time out of snapshot file generations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
An orchestrator that restores a VM from a snapshot has no reliable way to learn when the restore has finished: OpenVMM reports failures only in its log, and a restored vCPU may already be running before the orchestrator can tell whether devices started. Add `--restore-ready-path <PATH>`, which requires `--restore-snapshot`. Before the VM worker launches, OpenVMM connects to the named endpoint, an existing Unix domain socket on Linux or a `//./pipe/...` named pipe on Windows, and passes the connection to the worker as the new `restore_ready_sink` parameter. After the saved state is applied, the worker holds the restored VPs stopped. When the VM first starts, the worker starts its state units, writes exactly `RESTORE_READY_EVENT_V1` (`OPENVMM_RESTORE_READY_V1\n`) to the endpoint and flushes it, and only then releases the VPs, so the event always precedes guest execution. A failure to connect, write, or flush fails the launch, and a failure to write or flush also stops the started units, leaving the VPs stopped. The endpoint is used once and is never saved in a snapshot. The worker keeps this restore state in `dispatch/restore`, which now sequences the start of a restored VM. With `--paused`, the first successful `resume` in the REPL publishes the event. A resume that fails before then ends the REPL with an error and tears the VM down, because the single-use endpoint may already be consumed and a later resume would start the guest without the event. With `OPENVMM_STARTUP_PROFILE` set, restore also records the `restore` phases `saved_state_restore` and `device_start`, the latter with the guest-memory faults and populated bytes counted while the devices start. The management and petri launches pass no readiness endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
microsoft#4554 merged the firmware-less Linux direct boot of the dropped positions 051-053 as `loader::linux::mptable_boot`, where the chain had it in `loader::linux::microvm`. Load the microVM's kernel and compute its e820 RAM ranges there. Fold into positions 055 and 176 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nedSnapshot microsoft#4553's RestoreVM reads a snapshot with the path-based `openvmm_helpers::snapshot::read_snapshot`, which position 031 replaced with `OpenedSnapshot` and a `restore::read_snapshot` that also bounds the memory. RestoreVM now opens the snapshot once with `OpenedSnapshot`, which also checks that memory.bin has the manifest's length, checks the expected snapshot ID against its manifest, and closes its artifacts before the restore opens memory.bin as guest memory. Upstream's test of an existing snapshot reads it with the memory size of its manifest. Fold into position 031 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ures microsoft#4553 adds `snapshot_id` to `SnapshotManifest`, as field 18, and generates one for each SaveVM and for the command line's `save-snapshot`, neither of which captures a microVM. The microVM capture and the publication benchmark build their manifests field by field, so give them the zero ID, which the manifest defines as no snapshot ID. Nothing restores a microVM snapshot through RestoreVM, which alone checks an expected ID. Fold into positions 069 and 119 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microsoft#4553 adds `snapshot_saved` and `memory_backing_file` to `VmService` for SaveVM. Initialize them in the service that the guest exit test builds, as the other services are. Fold into position 118 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…publication microsoft#4553 documents SaveVM and the command line's restore for upstream's publication, which hard-links the memory backing file into the snapshot, and for a restore that runs memory.bin in place. Position 028 copies the backing file into a staged snapshot that a rename publishes without replacing an existing path, and position 032 restores the command line's snapshots through a private copy-on-write mapping. Give those reasons in SaveVM's documentation and comments and in the command line's save, and drop the snapshot guide's warning that a restore modifies the snapshot and can run only once, which still holds for RestoreVM, as its documentation says. The snapshot format guide names the snapshot ID of field 18, which a microVM capture leaves zero. Fold into positions 028, 032, and 069 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microsoft#4553 has iteration 1 of the ttrpc interface test save and restore its VM through SaveVM and RestoreVM, without the virtio consoles, which cannot be saved. That iteration also has the test's only virtio-frontend NIC, but position 044 lets virtio-net save and restore only a NIC bound to a static identity and feature contract, which the management RPC does not configure, so SaveVM refused the VM: "save state not supported". Give iteration 1 the VMBus NIC that the other iterations have, and cover the virtio frontend in iteration 2, which does not save. Fold into position 044 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RestoreVM, which microsoft#4553 adds, maps a snapshot's memory.bin in place, and position 028 publishes that file as an independent copy of the memory backing file, where upstream hard-links the backing file itself. On MSHV, with the snapshot on ext4, the VM that upstream's ttrpc interface test restores stopped as soon as it resumed: a VP took a memory intercept that an interrupt's delivery caused, which the emulator refuses ("an interrupt caused the memory access exit"). The test passes when the snapshot hard-links the backing file, or when the copy's pages, which its writes leave in the page cache, are dropped before the restore maps the file, and it still fails when the copy also writes the zero chunks that it skips. On tmpfs, it passes either way. Drop those pages once the copy is flushed. Fold into position 028 at the next reorganization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5ca20ee to
73cf403
Compare
Add the profile-owned ARM64 microVM boot, fixed device and platform-control integration using the existing loaders, device tree and shared device abstractions. Add governed ARM CPU/time admission with native WHP programming and readback, retaining the legacy fingerprint API and x86 behavior. Keep the qualified PA36 contract and cold-boot-only boundary; ARM snapshot capture and restore remain unavailable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Extend the existing --x-time-abi-verify lifecycle to ARM64 using one shared TimeAbiReport and the existing native admission checks. Report the selected profile, effective register checks and actual WHP address, feature-bank and counter-frequency readbacks without executing the guest. Keep x86 output compatible, reject unsupported ARM fault-injection hooks, and fail closed on missing evidence. This does not establish UTC synchronization or clock authority. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the shared CPU/time report and sample-field conversion within the architecture-selected adapter. Restore native type and clock-status import ownership so target-branch clock changes merge without conflict, retaining one shared report and unchanged field conversion and runtime behavior. No target commits or time-policy changes are incorporated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
6036d7e to
b09c588
Compare
|
Gautam Ahuja (@ahuja-gautam) We have validated your ARM64 WHP implementation on a MECHREVO N1X Spark with NVIDIA N1X. Our follow-up in #5096 adds a CPU profile for the two observed MIDRs, with focused tests and successful native boot validation. Since this PR is marked "[DO NOT REVIEW]", could you advise on the preferred contribution path? Should we keep #5096 as a draft until this lands, or would you prefer the two N1X-specific commits as a follow-up to your ARM branch? We want to keep the contribution narrowly scoped and avoid duplicating review of the prerequisite implementation. The N1X-only comparison contains the two adaptation commits and four changed files, excluding the later upstream merge in #5096. |
|
Thans (@QThans) please retarget #5096 to follow up Gautam Ahuja (@ahuja-gautam) branch. Keep "[DO NOT REVIEW]" |
fe61f9c to
1b25c5b
Compare
Pedro Henrique Penna (@ppenna) hi, I have opened ahuja-gautam#1 against Gautam's ARM branch, keeping “[DO NOT REVIEW]” and leaving #5096 open. Is this what you intended? |
Yes, this workstream is queued for review in the next days. |
Summary
hvc0console. Preserve x86/Standard defaults, legacy probing, PIO and existing state/wire contracts.CNTVCT_EL0reads around host UTC sampling, rejecting changed epochs, backwards counters and pairing uncertainty above 100 us. ReuseTimeSampleSource, the existing 48-byte v1 codec and shared host-clock publisher; unknown or unrepresentable calibration/correction/error bounds remain unknown, not synchronization guarantees.ARM support remains cold-boot only: no ARM snapshot capture or restore.
Current source
Published head
b09c58843ae310dfff9ae68e9e822f9c5846b914, tree0ab5ac89b10bf83d0a129d0a328abe682bc97835: three linear ARM feature/verifier/layout commits (30f64b509→b2304344b→b09c58843) above exact OpenVMM baseline73cf403574a8907ee94dab0ab9d648152b832b05, pinned by selected NVXc9817ae7(v6.1). Zero new merge commits.This base includes the upstream PIT, RTC, loader and state changes. Independent replay and integrated architecture reviews found no must-fix, required redesign or obsolete primitive: shared loaders/devices, CPU/time publisher, cold-only guards and unknown-bound semantics remain in use.
The later target
09cd771c061aa1d1b406b41eaf2eda851db87c15content-merges cleanly with this head, but its newer RTC/timer/parser runtime changes are not included or qualified by this candidate.OpenVMM validation of this head
0.5.0+gb09c58843, SHA256f171f776a76c7aeef5faae7221e7a508d4d3de48f7e62f227bc7fbc372eb7653. Its actual Cargo profile hasopt_level=0, debug information/assertions and overflow checks enabled; it is not an optimized release build, regardless of staging-directory names.guest_started=falsemeans no Linux guest instructions executed, not a guest-boot result.Final paired Phase3 execution
Runtime execution used NVX
2000aa078713467add77565acdae688a6d6ecfcfwith this exactb09c58843/f171f776product. Final published NVX24bdced073a24418c1ebbf1d0775bb5ba42266bais a test-only child; all 1,032 production-input rows are identical. Runtime results apply through that verified identity, not by claiming the VMs were re-executed at24bd.24bdPython suiteActual normal ARM capture refusal passed with the cold-only error. Normal-product ARM restore refusal is not proved: the retained fixture failed before manifest admission. The 37 native MMIO callback unit tests include generation/time-record and register permission/width/bounds cases; they are not raw guest-MMIO proof or deterministic active-VP0/AP-latch-overlap qualification.