Skip to content

[DO NOT REVIEW] microvm: enable governed ARM64 cold boot on WHP - #5045

Open
Gautam Ahuja (ahuja-gautam) wants to merge 272 commits into
microsoft:feature/microvm/devfrom
ahuja-gautam:openvmm-arm64-microvm-dev-20261111
Open

Gautam Ahuja (ahuja-gautam) wants to merge 272 commits into
microsoft:feature/microvm/devfrom
ahuja-gautam:openvmm-arm64-microvm-dev-20261111

Conversation

@ahuja-gautam

@ahuja-gautam Gautam Ahuja (ahuja-gautam) commented Nov 11, 2026 •

Copy link
Copy Markdown

Summary

  • Enable profile-owned ARM64/WHP microVM cold boot using the existing Image loader, device-tree, resources and devices, with fixed device/status/GIC bindings, MMIO platform control and hvc0 console. Preserve x86/Standard defaults, legacy probing, PIO and existing state/wire contracts.
  • Add governed CPU admission with actual WHP programming and effective-ID, feature-bank, PA-width and native-counter-frequency readback, retaining the PA36 profile.
  • Pair actual active guest CNTVCT_EL0 reads around host UTC sampling, rejecting changed epochs, backwards counters and pairing uncertainty above 100 us. Reuse TimeSampleSource, the existing 48-byte v1 codec and shared host-clock publisher; unknown or unrepresentable calibration/correction/error bounds remain unknown, not synchronization guarantees.

ARM support remains cold-boot only: no ARM snapshot capture or restore.

Current source

Published head b09c58843ae310dfff9ae68e9e822f9c5846b914, tree 0ab5ac89b10bf83d0a129d0a328abe682bc97835: three linear ARM feature/verifier/layout commits (30f64b509 → b2304344b → b09c58843) above exact OpenVMM baseline 73cf403574a8907ee94dab0ab9d648152b832b05, pinned by selected NVX c9817ae7 (v6.1). Zero new merge commits.

This base includes the upstream PIT, RTC, loader and state changes. Independent replay and integrated architecture reviews found no must-fix, required redesign or obsolete primitive: shared loaders/devices, CPU/time publisher, cold-only guards and unknown-bound semantics remain in use.

The later target 09cd771c061aa1d1b406b41eaf2eda851db87c15 content-merges cleanly with this head, but its newer RTC/timer/parser runtime changes are not included or qualified by this candidate.

OpenVMM validation of this head

  • Native Windows ARM64 focused units: 514 passed, 0 failed, 3 existing tests ignored; 740 filtered. x64 units executed under Windows ARM64 emulation: 210 passed, 0 failed, 0 ignored; 425 filtered. This is not physical-x86 WHP qualification.
  • All-target checks passed for 20 affected/coupled packages on both ISAs; native all-target clippy, no-dependency docs and all four full formatter passes passed. Formatting left the reviewed source unchanged.
  • The normal clean AA64 product reports 0.5.0+gb09c58843, SHA256 f171f776a76c7aeef5faae7221e7a508d4d3de48f7e62f227bc7fbc372eb7653. Its actual Cargo profile has opt_level=0, debug information/assertions and overflow checks enabled; it is not an optimized release build, regardless of staging-directory names.
  • Three normal-product verifier cases passed: effective-ID/native-counter readback, unknown-profile rejection and unsupported-hook rejection. guest_started=false means no Linux guest instructions executed, not a guest-boot result.

Final paired Phase3 execution

Runtime execution used NVX 2000aa078713467add77565acdae688a6d6ecfcf with this exact b09c58843/f171f776 product. Final published NVX 24bdced073a24418c1ebbf1d0775bb5ba42266ba is a test-only child; all 1,032 production-input rows are identical. Runtime results apply through that verified identity, not by claiming the VMs were re-executed at 24bd.

Executed scope Result
Maintained cold/repetition CLI campaign 173 CLI commands passed: 45 cold commands/53 scenario records and 128 repetition commands/135 records.
Runtime-adapter matrix 115 passed, across 9 targets; 46 snapshot cases excluded, not passed.
Actual guest units 833 passed, 0 failed; 12 exact known tests ignored.
Debug and explicit profile coverage 8 debug cells and 8 auto/host 1/2/4/8VP cells passed.
Final 24bd Python suite 2,255 methods run, zero assertion failures, 2,147 complete successes; 3 WinError 1314 error records from 2 Ubuntu setup methods, and 107 skip records (104 whole methods, 3 subcases).

Actual normal ARM capture refusal passed with the cold-only error. Normal-product ARM restore refusal is not proved: the retained fixture failed before manifest admission. The 37 native MMIO callback unit tests include generation/time-record and register permission/width/bounds cases; they are not raw guest-MMIO proof or deterministic active-VP0/AP-latch-overlap qualification.

@ahuja-gautam
Gautam Ahuja (ahuja-gautam) requested a review from a team as a code owner November 11, 2026 18:53
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code labels Nov 11, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@ahuja-gautam

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

@ahuja-gautam
Gautam Ahuja (ahuja-gautam) marked this pull request as draft November 11, 2026 18:59
@ahuja-gautam Gautam Ahuja (ahuja-gautam) changed the title microvm: enable governed ARM64 cold boot on WHP [DO NOT REVIEW] microvm: enable governed ARM64 cold boot on WHP Nov 11, 2026
@ahuja-gautam
Gautam Ahuja (ahuja-gautam) marked this pull request as ready for review November 11, 2026 19:07
@ppenna
Pedro Henrique Penna (ppenna) removed the request for review from a team November 11, 2026 20:01
@github-actions

Copy link
Copy Markdown

@ahuja-gautam

Gautam Ahuja (ahuja-gautam) commented Nov 12, 2026 •

Copy link
Copy Markdown
Author

Clarifying the unsafe-code notice: it scans whole modified Rust files. Relative to the rebased PR's base (5d656d85), the existing unsafe blocks and unsafe fn declarations in vm/whp/src/lib.rs and tmk/tmk_vmm/src/host_vmm.rs are unchanged; this PR adds no new unsafe blocks. The WHP additions here are safe property getters reusing the existing wrapper. The PR does add ARM ABI definitions and safe callers of those wrappers, so this is not a claim that the WHP changes need no review.

AI-authored clarification, posted at the author's request.

@github-actions

Copy link
Copy Markdown

@ahuja-gautam
Gautam Ahuja (ahuja-gautam) force-pushed the openvmm-arm64-microvm-dev-20261111 branch from 1d6ddec to 6036d7e Compare November 12, 2026 22:12
@github-actions

Copy link
Copy Markdown

…frequency

MSHV partitions did not report their TSC frequency, so guests had to
calibrate it, and nothing verified that a snapshot is restored on a host
with the TSC frequency that it was saved with. The TSC of each VP is also
restored at a different host time, so the counters of a restored SMP guest
drift apart.

Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and
implement the partition clock hooks: report the TSC and local APIC timer
frequencies, reject a requested TSC frequency that differs from that of
the partition, since MSHV cannot change it, and, after a snapshot downtime,
freeze partition time and align the counters of all VPs to the advanced
BSP counter.

Hide the TSC-deadline timer from the guest, in both the processor features
of the partition and CPUID, since MSHV does not reliably deliver its events
to direct-boot guests, and report that ordinary state access does not
freeze the partition clock.

It also includes these changes:

- virt_mshv: adapt the chain to upstream's nested virtualization

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…erent

WHP partitions did not report their TSC frequency, so guests had to
calibrate it, and nothing verified that a snapshot is restored on a host
with the TSC frequency that it was saved with. The TSC of each VP is also
restored at a different host time, so the counters of a restored SMP guest
disagree with each other.

Expose the exact TSC frequency of the partition in CPUID leaf 0x15, and
implement the partition clock hooks: report the TSC frequency and the local
APIC timer frequency, reject a requested TSC frequency that differs from
that of the partition, and, after a snapshot downtime, align the counters
of all VPs of an SMP partition with partition time suspended.

To keep TSC reads coherent across the VPs, a restored SMP partition that
does not use nested virtualization then derives the TSC of each VP from the
partition-wide reference time. While this restored-TSC clock is active, WHP
exits on RDTSC and on accesses to IA32_TSC, TSC_ADJUST, and
IA32_TSC_DEADLINE, and the VP state accessors read and restore the TSC
through the clock. A VP whose guest writes its TSC or changes its
TSC_ADJUST leaves the clock and gets the native WHP semantics again, and
resetting the partition disables the clock.

Read IA32_TSC_DEADLINE before the APIC state when saving a VP, so that the
deadline cannot expire between the two reads. Add the WHP properties and
registers that this needs, and report a failure to inject a general
protection fault from an MSR exit instead of panicking.

It also includes these changes:

- virt_whp: query the TSC frequency for the aarch64 hv1 emulator
- virt: adopt upstream's TscDeadline VP state element

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
When a VM is restored after it was stopped for some time, the TSC of each
VP and its local APIC timer continue from their saved values, so the guest
does not observe the elapsed time and its timers fire late.

Add `PartitionUnit::advance_tsc`, which advances the TSC of every stopped
VP by the downtime at the guest TSC frequency. It also advances the local
APIC timer of each VP at the APIC timer frequency, when the backend reports
one, and fires an armed TSC deadline that the advanced TSC passed. Each VP
adjusts its TSC through `Processor::advance_tsc`, reprograms its APIC and
TSC deadline, and then verifies that its counter advanced by at least the
requested amount, since a backend could discard the adjustment. The
VP set tests' `TestVp` implements the new `ControlVp` method as
unreachable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Measuring where snapshot capture and restore spend their time requires
timing individual lifecycle phases in several OpenVMM components and
processes, and correlating the results, without changing the behavior or
the output of ordinary runs.

Add a `profile` module to `openvmm_defs`. When the
`OPENVMM_STARTUP_PROFILE` environment variable is set to a nonempty value
other than `0`, a `ProfileSpan` measures an exclusive phase or a cumulative
milestone and writes one machine-readable
`OPENVMM_SNAPSHOT_PROFILE_V1` record to stderr, with its duration, the time
since process start, the process ID, and optional byte and fault counters.
Otherwise spans are inert. OpenVMM initializes the process-relative clock
at startup; the lifecycle phases add their spans as they are implemented.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ecoding

The snapshot manifest identified its format only by a version number:
nothing marked the file as an OpenVMM snapshot, or recorded which
saved-state schema `state.bin` holds and how long it is. Reading a snapshot
also decoded `manifest.bin` and read `state.bin` without any size limit.

Introduce manifest version 3, which adds a format magic, the saved-state
schema version and protobuf root type, and the length of `state.bin`, and
move the format definitions and their validation into a `format` module.
`SnapshotManifest` implements `Default` with the current format
identifiers, and the callers that build manifests take the fields that
they do not set from it. Manifests of the earlier version 2 format, which
records SHA-256 digests of `state.bin` and `memory.bin` instead, are
accepted as well; version 1 manifests are no longer accepted.

Writing a snapshot records the length of `state.bin` and rejects saved
state larger than 256 MiB. Reading a snapshot limits `manifest.bin` to
1 MiB, validates the manifest format before it reads `state.bin`, and
requires `state.bin` to have the recorded length.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Saving a snapshot created the destination directory, wrote the manifest and
the saved state into it, and hard-linked the memory backing file as
`memory.bin`. A failure could leave a partial snapshot at the destination,
nothing was flushed to stable storage, an existing directory was silently
reused, and the hard link made the snapshot an alias of live guest RAM that
any later write to the backing file corrupted. The backing file also had to
be on the same filesystem as the snapshot.

Publish snapshots through a private, uniquely named staging directory next
to the destination. `state.bin`, `memory.bin`, and `manifest.bin` are written
and flushed there, the staging directory is flushed, and it is then renamed
to the destination without replacing an existing path. The rename is the
commit point, after which the parent directory is flushed. A failure before
the commit removes the staging directory and leaves the destination absent;
`SnapshotWriteError` distinguishes it from a committed snapshot whose parent
directory could not be flushed.

`memory.bin` is now an independent copy of guest RAM, made from an open
handle so that replacing the path of the backing file cannot substitute
other bytes, and `write_snapshot_from_memory_file` accepts that handle
directly. The copy clones the file when the filesystem supports it, and
otherwise copies only allocated ranges or nonzero data on Linux and copies
densely on Windows. The backing file must have the length that the manifest
records.

Add the snapshot file-system helpers that this needs: private directories,
no-replace renames, directory flushes, regular-file access that rejects
Windows reparse points, and sparse-aware copies. Update the Guide.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Copying guest RAM into `memory.bin` keeps a snapshot independent of the
source VM, but it doubles the storage and the capture time for a VM whose
backing file OpenVMM itself creates and owns, and whose source is terminated
after capture anyway.

Add `write_snapshot_from_owned_memory_file`, which publishes such RAM by
hard-linking the exact open backing handle into the staging directory as
`memory.bin`, after the state and the manifest are written and the handle is
flushed. Linux links the descriptor with `linkat(AT_EMPTY_PATH)` or through
`/proc/self/fd`, and Windows uses a handle-relative link; the link is then
opened relative to the staging directory and must have the device and inode
or file ID and the length of the source. Filesystems that cannot link fall
back to the independent copy. Once the link may exist, the staging directory
aliases live RAM: a failure removes it and flushes its parent before it
reports a rollback-safe error, and a failed cleanup is reported as
`SnapshotWriteError::CleanupUncertain`, which requires the source to
terminate instead of resuming.

Size new memory backing files through
`initialize_snapshot_memory_backing_file`, which keeps them sparse except
on Windows, where a dense file avoids slow copy-on-write faults, and add
`open_memory_backing_file_handle`, which returns the sized backing file
itself so that a caller can keep the exact handle for publication.

It also includes these changes:

- microVM: fix the macOS clippy build

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
Restoring a snapshot maps `memory.bin` as guest RAM, but a file backing
could only be mapped shared, so every guest write would go into the
snapshot artifact, and the snapshot could be used only once.

Add a file mapping mode to the memory manager. With
`FileMappingMode::CopyOnWrite`, an existing backing file is mapped with
private copy-on-write views, so the guest reads the contents of the file
while its writes stay private to the process and never reach the file.
Copy-on-write RAM requires an existing file backing, and it is not offered
as a shared-memory backing to other processes. The default mode is shared,
as before.

Also let the memory manager count guest-memory faults and the bytes
populated while resolving them, for diagnosing restore performance, and
report the totals through `MemoryManager::fault_counters`.

It also includes these changes:

- openvmm: create copy-on-write mappables with upstream's signature

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…he directory

Restore read `manifest.bin` and `state.bin` by joining each name to the
snapshot path, and the caller reopened `memory.bin` the same way. Every
open resolved the pathname again, following symbolic links, so renaming
or replacing the directory, or one of its files, between two opens could
combine artifacts of different generations. Unexpected files in the
directory were ignored, and the length of `memory.bin` was checked only
by the caller.

Add `OpenedSnapshot`, which opens the snapshot directory once and then
opens each artifact relative to that directory handle. Linux uses
`openat` with `O_NOFOLLOW`; Windows uses handle-relative, read-only opens
that share only reads and reject reparse points. `OpenedSnapshot::open`
reads the bounded manifest and saved state through the opened handles,
requires the directory to contain exactly `manifest.bin`, `state.bin`,
and `memory.bin` as regular files, checks the exact artifact lengths, and
records the generation of the opened memory file: its identity, length,
and, on Linux, its modification time. Accessors return the
manifest and state, check that the memory generation is unchanged,
duplicate the exact memory handle for a later mapping, and report
logical and allocated artifact sizes for profiling. `into_parts` moves
the open directory and artifact handles into the new
`SnapshotRestoreGuards`, so a later caller can keep the generation open
for the lifetime of a restored VM.

The Linux generation leaves out the change time, which a change of the
file's links, mode, or owner moves without changing its content. The
source of a capture with automatic RAM publishes `memory.bin` as a hard
link of its RAM file, whose temporary name it removes as it tears down,
after the commit, and a restore that opened the snapshot before then
must still find the generation that it opened. A write still moves the
modification time, so a same-length change of the memory between the
open and the mapping is still found.

`restore::read_snapshot` now takes the expected guest memory size and
validates all three artifacts through an `OpenedSnapshot`, and
`read_snapshot_manifest`, `read_snapshot_with_memory`, and
`read_snapshot_artifacts_with_memory` cover manifest-only reads and reads
that also return the exact memory handle. The top-level `read_snapshot`
re-export is removed; its callers use `restore::read_snapshot`. The
command-line restore path still reopens `memory.bin` as shared, writable
guest RAM, until restores map it privately copy-on-write.

Document restore validation and the limits of the local format: it does
not authenticate same-length payload changes, and Linux file descriptors
do not provide mandatory write exclusion.

Tests: a restore opens a snapshot whose `memory.bin` links the owned RAM
file that it was published from, and that file's other name is then
removed, which leaves `memory.bin` with one link and a new change time
but its modification time; the restore still validates and duplicates
the memory that it opened and reads back its content.

It also includes these changes:

- openvmm_helpers: leave the change time out of snapshot file generations

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
…ion handles

`--restore-snapshot` reopened `memory.bin` by path for reading and writing
and mapped it as shared guest RAM. Every guest write after restore went
into the snapshot artifact, so a snapshot could be restored only once,
and copying the directory first was the only way to reuse it. The
restored VM also held no handle to the generation it had validated, so
the artifacts could be replaced while the VM ran.

Restore now opens the snapshot through an `OpenedSnapshot` before the VM
configuration is built, in the new `snapshot_restore` module of
`openvmm_entry`. Just before the VM worker launches, the opened generation
is validated against the VM configuration, a duplicate of the exact
opened memory handle becomes a private copy-on-write memory handle
(`file_to_copy_on_write_memory_fd`), and the memory generation is checked
again after the mapping handle is created. The worker receives the handle
with the new `shared_memory_copy_on_write` flag and maps guest RAM
privately copy-on-write, so guest writes stay in the process and never
reach `memory.bin`, and one snapshot can be restored any number of times.
The open directory and artifact handles move to the worker as the new
`snapshot_restore_guards` parameter, and the worker keeps them until it
exits, which keeps the generation open for the lifetime of the restored
VM; on Windows the read-only handles also deny writes to it.

The worker takes its restore inputs from the parameters in the new
`dispatch/restore` module, which also takes over the saved-state
inventory check. The management and petri launches pass the defaults: a
shared mapping and no guards.

With `OPENVMM_STARTUP_PROFILE` set, restore records the `restore` phases
`artifact_open`, `artifact_prepare`, and `cow_section_create`, and a
launch records the `startup` milestones `worker_launch` and
`worker_construct` and the worker phases `partition_prototype`,
`cow_map_view`, `partition_build`, `gpa_registration`, and
`partition_finalize`.

Document that restores are private and repeatable, and describe the
handles that restore retains.

It also includes these changes:

- openvmm: create copy-on-write mappables with upstream's signature
- openvmm_helpers: leave the change time out of snapshot file generations

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
An orchestrator that restores a VM from a snapshot has no reliable way to
learn when the restore has finished: OpenVMM reports failures only in its
log, and a restored vCPU may already be running before the orchestrator
can tell whether devices started.

Add `--restore-ready-path <PATH>`, which requires `--restore-snapshot`.
Before the VM worker launches, OpenVMM connects to the named endpoint, an
existing Unix domain socket on Linux or a `//./pipe/...` named pipe on
Windows, and passes the connection to the worker as the new
`restore_ready_sink` parameter. After the saved state is applied, the
worker holds the restored VPs stopped. When the VM first starts, the
worker starts its state units, writes exactly `RESTORE_READY_EVENT_V1`
(`OPENVMM_RESTORE_READY_V1\n`) to the endpoint and flushes it, and only
then releases the VPs, so the event always precedes guest execution. A
failure to connect, write, or flush fails the launch, and a failure to
write or flush also stops the started units, leaving the VPs stopped.
The endpoint is used once and is never saved in a snapshot. The worker
keeps this restore state in `dispatch/restore`, which now sequences the
start of a restored VM.

With `--paused`, the first successful `resume` in the REPL publishes the
event. A resume that fails before then ends the REPL with an error and
tears the VM down, because the single-use endpoint may already be
consumed and a later resume would start the guest without the event.

With `OPENVMM_STARTUP_PROFILE` set, restore also records the `restore`
phases `saved_state_restore` and `device_start`, the latter with the
guest-memory faults and populated bytes counted while the devices start.

The management and petri launches pass no readiness endpoint.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ef640993-0003-4e49-b1ed-1b8b4776d0ef
microsoft#4554 merged the firmware-less Linux direct boot of the
dropped positions 051-053 as `loader::linux::mptable_boot`, where the
chain had it in `loader::linux::microvm`. Load the microVM's kernel and
compute its e820 RAM ranges there.

Fold into positions 055 and 176 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nedSnapshot

microsoft#4553's RestoreVM reads a snapshot with the path-based
`openvmm_helpers::snapshot::read_snapshot`, which position 031 replaced
with `OpenedSnapshot` and a `restore::read_snapshot` that also bounds
the memory. RestoreVM now opens the snapshot once with `OpenedSnapshot`,
which also checks that memory.bin has the manifest's length, checks the
expected snapshot ID against its manifest, and closes its artifacts
before the restore opens memory.bin as guest memory. Upstream's test of
an existing snapshot reads it with the memory size of its manifest.

Fold into position 031 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ures

microsoft#4553 adds `snapshot_id` to `SnapshotManifest`, as
field 18, and generates one for each SaveVM and for the command line's
`save-snapshot`, neither of which captures a microVM. The microVM
capture and the publication benchmark build their manifests field by
field, so give them the zero ID, which the manifest defines as no
snapshot ID. Nothing restores a microVM snapshot through RestoreVM,
which alone checks an expected ID.

Fold into positions 069 and 119 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microsoft#4553 adds `snapshot_saved` and `memory_backing_file`
to `VmService` for SaveVM. Initialize them in the service that the
guest exit test builds, as the other services are.

Fold into position 118 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…publication

microsoft#4553 documents SaveVM and the command line's restore
for upstream's publication, which hard-links the memory backing file
into the snapshot, and for a restore that runs memory.bin in place.
Position 028 copies the backing file into a staged snapshot that a
rename publishes without replacing an existing path, and position 032
restores the command line's snapshots through a private copy-on-write
mapping. Give those reasons in SaveVM's documentation and comments and
in the command line's save, and drop the snapshot guide's warning that a
restore modifies the snapshot and can run only once, which still holds
for RestoreVM, as its documentation says. The snapshot format guide
names the snapshot ID of field 18, which a microVM capture leaves zero.

Fold into positions 028, 032, and 069 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microsoft#4553 has iteration 1 of the ttrpc interface test save
and restore its VM through SaveVM and RestoreVM, without the virtio
consoles, which cannot be saved. That iteration also has the test's only
virtio-frontend NIC, but position 044 lets virtio-net save and restore
only a NIC bound to a static identity and feature contract, which the
management RPC does not configure, so SaveVM refused the VM: "save state
not supported". Give iteration 1 the VMBus NIC that the other iterations
have, and cover the virtio frontend in iteration 2, which does not save.

Fold into position 044 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RestoreVM, which microsoft#4553 adds, maps a snapshot's
memory.bin in place, and position 028 publishes that file as an
independent copy of the memory backing file, where upstream hard-links
the backing file itself. On MSHV, with the snapshot on ext4, the VM that
upstream's ttrpc interface test restores stopped as soon as it resumed:
a VP took a memory intercept that an interrupt's delivery caused, which
the emulator refuses ("an interrupt caused the memory access exit"). The
test passes when the snapshot hard-links the backing file, or when the
copy's pages, which its writes leave in the page cache, are dropped
before the restore maps the file, and it still fails when the copy also
writes the zero chunks that it skips. On tmpfs, it passes either way.
Drop those pages once the copy is flushed.

Fold into position 028 at the next reorganization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the profile-owned ARM64 microVM boot, fixed device and platform-control integration using the existing loaders, device tree and shared device abstractions. Add governed ARM CPU/time admission with native WHP programming and readback, retaining the legacy fingerprint API and x86 behavior.

Keep the qualified PA36 contract and cold-boot-only boundary; ARM snapshot capture and restore remain unavailable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Extend the existing --x-time-abi-verify lifecycle to ARM64 using one shared TimeAbiReport and the existing native admission checks. Report the selected profile, effective register checks and actual WHP address, feature-bank and counter-frequency readbacks without executing the guest. Keep x86 output compatible, reject unsupported ARM fault-injection hooks, and fail closed on missing evidence. This does not establish UTC synchronization or clock authority.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the shared CPU/time report and sample-field conversion within the architecture-selected adapter. Restore native type and clock-status import ownership so target-branch clock changes merge without conflict, retaining one shared report and unchanged field conversion and runtime behavior. No target commits or time-policy changes are incorporated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@ahuja-gautam
Gautam Ahuja (ahuja-gautam) force-pushed the openvmm-arm64-microvm-dev-20261111 branch from 6036d7e to b09c588 Compare November 13, 2026 21:15
@github-actions

Copy link
Copy Markdown

@QThans

Copy link
Copy Markdown

Gautam Ahuja (@ahuja-gautam) We have validated your ARM64 WHP implementation on a MECHREVO N1X Spark with NVIDIA N1X. Our follow-up in #5096 adds a CPU profile for the two observed MIDRs, with focused tests and successful native boot validation.

Since this PR is marked "[DO NOT REVIEW]", could you advise on the preferred contribution path? Should we keep #5096 as a draft until this lands, or would you prefer the two N1X-specific commits as a follow-up to your ARM branch?

We want to keep the contribution narrowly scoped and avoid duplicating review of the prerequisite implementation. The N1X-only comparison contains the two adaptation commits and four changed files, excluding the later upstream merge in #5096.

@ppenna

Copy link
Copy Markdown
Contributor

Thans (@QThans) please retarget #5096 to follow up Gautam Ahuja (@ahuja-gautam) branch. Keep "[DO NOT REVIEW]"

@QThans

Copy link
Copy Markdown

Thans (Thans (@QThans)) please retarget #5096 to follow up Gautam Ahuja (Gautam Ahuja (@ahuja-gautam)) branch. Keep "[DO NOT REVIEW]"

Pedro Henrique Penna (@ppenna) hi, I have opened ahuja-gautam#1 against Gautam's ARM branch, keeping “[DO NOT REVIEW]” and leaving #5096 open. Is this what you intended?

@ppenna

Copy link
Copy Markdown
Contributor

Thans (Thans (@QThans)) please retarget #5096 to follow up Gautam Ahuja (Gautam Ahuja (@ahuja-gautam)) branch. Keep "[DO NOT REVIEW]"

Pedro Henrique Penna (@ppenna) hi, I have opened ahuja-gautam#1 against Gautam's ARM branch, keeping “[DO NOT REVIEW]” and leaving #5096 open. Is this what you intended?

Yes, this workstream is queued for review in the next days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants