Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,10 @@ describes the source definitions.
read-only; and every device but the NIC indirect descriptors and event-index
notification suppression. No device offers `VIRTIO_F_ACCESS_PLATFORM`, since
the microVM has no IOMMU, and no console offers its size. Firmware, ACPI,
SMBIOS, PCI, VMBus, UARTs, graphics, isolation, nested virtualization, and
other devices are rejected. Linux discovers contiguous APIC IDs, the IOAPIC,
and the routing of its 24 inputs from Intel MP 1.4 tables at `0x0` and
`0x400`; `boot_params` is at
SMBIOS, PCI, VMBus, UARTs, graphics, isolation, and other devices are
rejected. Linux discovers contiguous APIC IDs, the IOAPIC, and the routing
of its 24 inputs from Intel MP 1.4 tables at `0x0` and `0x400`;
`boot_params` is at
`0x2000`, the command line starts at `0x20000`, and no ACPI or SMBIOS data is
exposed. Host-driven save/restore, pulse-save/restore, and worker restart
remain unavailable.
Expand All @@ -91,7 +91,22 @@ describes the source definitions.
with one core per vCPU, no SMT, xAPIC mode, and contiguous APIC IDs from 0.
Guest-requested snapshot capture and new-process restore are available for
blockless and fixed-block machines on Linux/KVM, Linux/MSHV, and Windows/WHP,
except for the RAM-backed overlay layout.
except for the RAM-backed overlay layout and nested virtualization.

`--nested-virt` gives a `microvm` nested virtualization, so that its guest
can run VMs of its own, on KVM and WHP but never on MSHV, which offers
nested virtualization only with the Hyper-V guest interface that the NVX time
ABI excludes. The effective CPUID then sets VMX (`CPUID.1:ECX[5]`) over the
CPU profile, whose policy clears every virtualization extension, and the VMX
capability MSRs that describe the controls a guest hypervisor may use are
the backend's. It requires an Intel CPU profile, since no profile describes
SVM's feature leaf `0x8000000A` (`E_PROFILE_UNSUPPORTED` on AMD), and a
backend that offers nested virtualization on the host: KVM with `kvm_intel`
loaded with `nested=1`, or WHP with its nested virtualization feature;
otherwise the partition's creation fails before the guest runs. Such a
microVM is never captured or restored: OpenVMM refuses `--nested-virt` with
`--snapshot-destination` or `--restore-snapshot` before the VM starts. A
microVM without `--nested-virt` exposes no virtualization extension.

A non-empty `--microvm-sandbox-block` topology must end with the writable
`scratch` block unless the command line carries exactly one
Expand Down Expand Up @@ -807,7 +822,9 @@ guest state and protect the directory with host access controls.
such as Virtual Secure Mode (VSM), which can hurt performance and interfere
with VMBus devices; nested virt cannot currently be combined with
`--hv`/VMBus or
`--hypervisor whp:user_mode_apic`.
`--hypervisor whp:user_mode_apic`. A `--machine microvm` takes it on KVM and
WHP only, with an Intel CPU profile, and refuses it with a snapshot capture
or restore; see `--machine` above.
* `--uefi [OPTIONS]`: Boot using `mu_msvm` UEFI. Options are comma-separated:
* `firmware=<FILE>`: Path to the UEFI firmware file (`MSVM.fd`). If omitted, the default is read from `OPENVMM_UEFI_FIRMWARE`, then from `X86_64_OPENVMM_UEFI_FIRMWARE` or `AARCH64_OPENVMM_UEFI_FIRMWARE`.
* `debug`: Enable UEFI debugging on COM1.
Expand Down
6 changes: 6 additions & 0 deletions Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,12 @@ value 2 offered `VIRTIO_F_ACCESS_PLATFORM` on every device but the NIC, and the
console size on the consoles, and put the virtio devices on ISA IRQs, where
value 3 gives each one its own IOAPIC input past them.

`VMConfig.nested_virt` gives a microVM nested virtualization, as
[`--nested-virt`](cli.md) does: on KVM and WHP, never on MSHV, and under an
Intel CPU profile, with VMX set in its effective CPUID. A request refuses it
with `microvm_snapshot.destination_path`, and a restore refuses it, since a
microVM with nested virtualization is never captured or restored.

`VMConfig.cpu_profile` selects the microVM's CPU profile as
[`--cpu-profile`](cli.md) does, with the same values and checks: `auto`, the
default, which an empty value also selects, for the built-in profile of the
Expand Down
13 changes: 13 additions & 0 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,17 @@ profile. Their error thus names every violation of each, leads with the code
of the first that fails, in that order, and goes on with the failures of the
others.

A microVM with nested virtualization (`--nested-virt`), whose effective CPUID
sets VMX, is never captured or restored. KVM cannot save the processor state
of a nested guest, and the VMX capabilities on which a guest hypervisor's
state depends are the backend's, which no CPU profile pins. OpenVMM therefore
refuses `--nested-virt` with `--snapshot-destination` or
`--restore-snapshot`, and the management RPC refuses `nested_virt` with
`destination_path` or `restore_path`, before the VM starts; a guest's
snapshot request in such a VM is also refused before anything stops. No
snapshot records VMX, so a restore, which computes the effective CPUID
without it, never has nested virtualization.

Restore also requires:

- a snapshot of the same time ABI version, so a snapshot of time ABI 1 must
Expand Down Expand Up @@ -614,3 +625,5 @@ immediately with a clear error if any active device does not support it.
model and stepping whose backend supports the profile.
Snapshots from manifest versions before 6, or from a build that computes a
different effective CPUID, must be recaptured.
- A microVM with nested virtualization (`--nested-virt`) supports neither
capture nor restore.
4 changes: 4 additions & 0 deletions openvmm/openvmm_core/src/worker/dispatch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1331,9 +1331,13 @@ impl InitializedVm {
cfg.microvm.restored_host_profile.as_ref(),
&parameters.hooks,
)?;
// Nested virtualization sets VMX over the profile's surface; a
// restore never has it, so a snapshot's effective CPUID never
// does either.
let effective_cpuid = Arc::new(virt::time_abi::cpuid::effective_cpuid(
&profile,
&processor_topology,
cfg.hypervisor.nested_virt,
)?);
let (msrs, config) = time_abi::partition_config(&profile, &effective_cpuid);
(
Expand Down
25 changes: 24 additions & 1 deletion openvmm/openvmm_core/src/worker/dispatch/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,14 @@ impl MicrovmParameters {
&parameters.cfg,
Some(parameters.hypervisor.id()),
)?;
// A snapshot never records nested virtualization, since no microVM
// with it is captured, so its restore never has it either.
anyhow::ensure!(
parameters.cfg.machine_profile != MachineProfile::Microvm
|| !parameters.cfg.hypervisor.nested_virt
|| parameters.saved_state.is_none(),
"a microVM restore never has nested virtualization"
);
Ok(Self { snapshot_boundary })
}

Expand Down Expand Up @@ -197,7 +205,9 @@ impl LoadedVm {
&mut self,
request: MicrovmSnapshotBoundaryRequest,
) -> bool {
let Some(request) = snapshot_rpc::filter_boundary_request(request) else {
let Some(request) =
snapshot_rpc::filter_boundary_request(request, self.inner.hypervisor_cfg.nested_virt)
else {
return true;
};
if self.snapshot_boundary.stop_guard.is_some() {
Expand Down Expand Up @@ -327,6 +337,19 @@ impl LoadedVm {
RemoteError::new(anyhow::anyhow!("VM is already stopped")),
));
}
// A microVM with nested virtualization is never captured: KVM cannot
// save a nested guest's processor state, and the VMX capabilities on
// which a guest hypervisor's state depends are the host backend's,
// which no CPU profile pins. The frontends refuse such a capture at
// launch; refuse it here too, before anything stops, rather than
// during the save.
if self.inner.hypervisor_cfg.nested_virt {
return Err(openvmm_defs::rpc::SnapshotQuiesceError::Rejected(
Comment thread
ppenna marked this conversation as resolved.
Resolved
RemoteError::new(anyhow::anyhow!(
"microVM snapshot capture does not support nested virtualization"
)),
));
}

let quiesce = openvmm_defs::profile::ProfileSpan::start();
if let Err(error) = self.state_units.quiesce_for_save(timeout).await {
Expand Down
45 changes: 41 additions & 4 deletions openvmm/openvmm_core/src/worker/dispatch/snapshot_rpc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,14 @@ use openvmm_defs::rpc::VmRpc;
#[error("VM mutation is unavailable while a microVM snapshot boundary is active")]
struct SnapshotBoundaryActive;

/// Returns the guest's snapshot boundary request that the worker must
/// establish, or `None` for one that ends before host input is gated or a vCPU
/// stops: one that stop or reset cancelled, and any of a microVM with nested
/// virtualization (`nested_virt`), which is never captured, which completes at
/// once.
pub(super) fn filter_boundary_request(
request: MicrovmSnapshotBoundaryRequest,
nested_virt: bool,
) -> Option<MicrovmSnapshotBoundaryRequest> {
// Stop/reset drops the release receiver but leaves the request queued.
// The worker serializes reset with boundary establishment, so checking
Expand All @@ -22,6 +28,14 @@ pub(super) fn filter_boundary_request(
tracelimit::warn_ratelimited!("dropping cancelled microVM snapshot boundary request");
return None;
}
if nested_virt {
tracelimit::warn_ratelimited!(
"refusing microVM snapshot boundary request: a microVM with nested virtualization is never captured"
);
request.release_write.send(());
request.transaction_complete.complete(());
return None;
}
Some(request)
}

Expand Down Expand Up @@ -113,9 +127,9 @@ mod tests {
// Cancellation leaves the old request ahead of the new one in the queue.
let old_request = requests.try_recv().unwrap();
assert!(old_request.release_write.is_closed());
assert!(filter_boundary_request(old_request).is_none());
assert!(filter_boundary_request(old_request, false).is_none());

let request = filter_boundary_request(requests.try_recv().unwrap())
let request = filter_boundary_request(requests.try_recv().unwrap(), false)
.expect("a cancelled predecessor must not discard the next request");
request.release_write.send(());
device.poll_device(&mut cx);
Expand All @@ -137,7 +151,7 @@ mod tests {
let IoResult::Defer(mut write) = device.io_write(SNAPSHOT_PORT, &[0]) else {
panic!("snapshot write was not deferred");
};
let request = filter_boundary_request(requests.try_recv().unwrap())
let request = filter_boundary_request(requests.try_recv().unwrap(), false)
.expect("live snapshot request was discarded");
request.release_write.send(());
let mut cx = Context::from_waker(Waker::noop());
Expand All @@ -156,7 +170,30 @@ mod tests {
IoResult::Defer(_)
));
drop(device);
assert!(filter_boundary_request(requests.try_recv().unwrap()).is_none());
assert!(filter_boundary_request(requests.try_recv().unwrap(), false).is_none());
}

#[test]
fn snapshot_boundary_completes_a_nested_microvms_request_at_once() {
let (send, mut requests) = mesh::channel();
let mut device = MicrovmSnapshotRequest::new(Some(send), Duration::from_secs(1));
device.start();
let IoResult::Defer(mut write) = device.io_write(SNAPSHOT_PORT, &[0]) else {
panic!("snapshot write was not deferred");
};
// The request ends before the worker gates input or stops a vCPU,
// and the guest's write completes.
assert!(filter_boundary_request(requests.try_recv().unwrap(), true).is_none());
let mut cx = Context::from_waker(Waker::noop());
device.poll_device(&mut cx);
assert!(matches!(write.poll_write(&mut cx), Poll::Ready(Ok(()))));

// The device takes the next request, which the transaction's
// completion released.
let IoResult::Defer(_) = device.io_write(SNAPSHOT_PORT, &[0]) else {
panic!("the next snapshot write was not deferred");
};
assert!(filter_boundary_request(requests.try_recv().unwrap(), true).is_none());
}

#[test]
Expand Down
11 changes: 7 additions & 4 deletions openvmm/openvmm_core/src/worker/dispatch/time_abi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1457,7 +1457,7 @@ mod tests {
fn test_config(vp_count: u32) -> (Arc<TimeAbiMsrs>, TimeAbiConfig, EffectiveCpuid) {
let profile = test_profile();
let effective =
effective_cpuid(profile, &topology(vp_count, X2ApicState::Supported)).unwrap();
effective_cpuid(profile, &topology(vp_count, X2ApicState::Supported), false).unwrap();
let (msrs, config) = partition_config(&partition_profile(profile), &effective);
(msrs, config, effective)
}
Expand Down Expand Up @@ -1594,7 +1594,8 @@ mod tests {
for profile in cpu_profile::pinned_profiles() {
for vp_count in [1, 2, 8] {
for x2apic in [X2ApicState::Unsupported, X2ApicState::Supported] {
let effective = effective_cpuid(profile, &topology(vp_count, x2apic)).unwrap();
let effective =
effective_cpuid(profile, &topology(vp_count, x2apic), false).unwrap();
let (_, config) = partition_config(&partition_profile(profile), &effective);
assert_eq!(config.cpu_profile.id(), profile.id());

Expand Down Expand Up @@ -1630,7 +1631,8 @@ mod tests {
for profile in cpu_profile::pinned_profiles() {
for vp_count in [1, 2, 8, 64] {
for x2apic in [X2ApicState::Unsupported, X2ApicState::Supported] {
let effective = effective_cpuid(profile, &topology(vp_count, x2apic)).unwrap();
let effective =
effective_cpuid(profile, &topology(vp_count, x2apic), false).unwrap();
let (_, config) = partition_config(&partition_profile(profile), &effective);
check_presented(profile, &effective, config.cpuid.leaves().to_vec())
.unwrap_or_else(|err| {
Expand Down Expand Up @@ -1691,7 +1693,8 @@ mod tests {
assert!(profile.lookup(0, 0)[0] < 0x1f, "{}", profile.id());
for vp_count in [1, 2, 8] {
let effective =
effective_cpuid(profile, &topology(vp_count, X2ApicState::Supported)).unwrap();
effective_cpuid(profile, &topology(vp_count, X2ApicState::Supported), false)
.unwrap();
let subleaves = |function| {
effective
.results()
Expand Down
52 changes: 49 additions & 3 deletions openvmm/openvmm_defs/src/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1950,6 +1950,30 @@ fn validate_machine_load_mode(
}
}

/// The backends whose microVMs may have nested virtualization.
pub const MICROVM_NESTED_VIRTUALIZATION_BACKENDS: [&str; 2] = ["kvm", "whp"];

/// Checks that a microVM with nested virtualization runs on a backend that
/// supports it, KVM or WHP, when `hypervisor_id` names the backend. MSHV
/// exposes nested virtualization only to a partition with the Hyper-V guest
/// interface, which the NVX time ABI excludes, so a microVM on MSHV never has
/// it.
///
/// Whether the host itself offers nested virtualization, and whether the
/// microVM's CPU profile is Intel's, whose VMX nested virtualization exposes,
/// is checked when the partition is created, before the guest runs.
pub fn validate_microvm_nested_virtualization(hypervisor_id: Option<&str>) -> anyhow::Result<()> {
if let Some(hypervisor_id) = hypervisor_id {
anyhow::ensure!(
MICROVM_NESTED_VIRTUALIZATION_BACKENDS.contains(&hypervisor_id),
"microVM nested virtualization requires KVM or WHP, not {hypervisor_id}: MSHV \
exposes nested virtualization only with the Hyper-V guest interface, which the \
NVX time ABI excludes"
);
}
Ok(())
}

/// Validates the microVM machine contract. Standard-machine configurations are unchanged.
pub fn validate_machine_config(config: &Config, hypervisor_id: Option<&str>) -> anyhow::Result<()> {
let MachineProfile::Microvm = config.machine_profile else {
Expand Down Expand Up @@ -2036,10 +2060,12 @@ pub fn validate_machine_config(config: &Config, hypervisor_id: Option<&str>) ->
anyhow::ensure!(
!config.hypervisor.with_hv
&& config.hypervisor.with_vtl2.is_none()
&& config.hypervisor.with_isolation.is_none()
&& !config.hypervisor.nested_virt,
"microVM does not support Hyper-V enlightenments, VTL2, isolation, or nested virtualization"
&& config.hypervisor.with_isolation.is_none(),
"microVM does not support Hyper-V enlightenments, VTL2, or isolation"
);
if config.hypervisor.nested_virt {
validate_microvm_nested_virtualization(hypervisor_id)?;
}

let expected_chipset = BaseChipsetManifest {
with_generic_cmos_rtc: true,
Expand Down Expand Up @@ -2332,6 +2358,26 @@ mod tests {
);
}

/// A microVM may have nested virtualization on KVM and WHP, and on a
/// backend that is not yet selected, but never on MSHV.
#[test]
fn microvm_nested_virtualization_requires_kvm_or_whp() {
for hypervisor_id in [None, Some("kvm"), Some("whp")] {
validate_microvm_nested_virtualization(hypervisor_id).unwrap();
}
for hypervisor_id in ["mshv", "hvf", ""] {
let error = validate_microvm_nested_virtualization(Some(hypervisor_id))
.unwrap_err()
.to_string();
assert!(
error.starts_with(&format!(
"microVM nested virtualization requires KVM or WHP, not {hypervisor_id}:"
)),
"{error}"
);
}
}

#[test]
fn microvm_processor_limit_is_host_owned() {
let mut cmdline = build_microvm_command_line(&[], false).unwrap();
Expand Down
Loading
Loading