Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 65 additions & 29 deletions Guide/src/dev_guide/snapshot_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,19 +254,26 @@ or group of MSRs that a VP holds and a fresh VP does not:
| 18 | `debug_ctl` | `IA32_DEBUGCTL` | the backend has it (KVM) |
| 19 | `misc_msrs` | `IA32_MISC_ENABLE`, `MSR_MISC_FEATURES_ENABLES` (CPUID faulting), and `IA32_POWER_CTL` | the backend has them (KVM) |
| 20 | `machine_check` | `IA32_MCG_STATUS`, `IA32_MCG_CTL`, and the control, status, address, miscellaneous, and CMCI control registers of each machine-check bank | the backend has them (KVM) |
| 21 | `bndcfgs` | `IA32_BNDCFGS` | the guest's CPUID enumerates MPX |
| 22 | `svm_msrs` | AMD's `VM_CR` and `VM_HSAVE_PA` | the backend has them (KVM on an AMD host) |
| 23 | `osvw` | AMD's `OSVW_ID_LENGTH` and `OSVW_STATUS` | the guest's CPUID enumerates OSVW, and the backend keeps them (KVM on an AMD host) |

A partition has an element only where the guest can have the state and the
backend can access it, which the partition's capabilities record. The guest's
CPUID decides the first for `spec_ctrl` and `umwait_control`; no CPUID bit
decides it for the other elements, whose MSRs every CPU has. For the KVM
backend:

- It can access an MSR only if the host's `KVM_GET_MSR_INDEX_LIST` lists it, so
a host without the MSR, such as one without WAITPKG, holds no element for it
instead of failing every save.
CPUID decides the first for `spec_ctrl`, `umwait_control`, `bndcfgs`, and
`osvw`; no CPUID bit decides it for the other elements, whose MSRs every CPU of
the vendor has. For the KVM backend:

- It can access an MSR only if the host's `KVM_GET_MSR_INDEX_LIST` lists it,
except for the ones below that KVM keeps without listing them, so a host
without the MSR, such as one without WAITPKG, holds no element for it instead
of failing every save. KVM lists `IA32_BNDCFGS` where it exposes MPX,
which it does on a host with MPX: the kernel still enables MPX's state
components in the host's XCR0 for KVM's guests, although it dropped MPX for
its own processes in Linux 5.6.
- It can access none of them for an SNP guest, whose registers live in an
encrypted VMSA.
- `IA32_DEBUGCTL` is the exception to the list. Every KVM virtualizes it, and
- `IA32_DEBUGCTL` is an exception to the list. Every KVM virtualizes it, and
`KVM_GET_MSRS` and `KVM_SET_MSRS` reach it on every kernel that the time ABI
supports, Linux 5.16 and later: the vendor code fails a write only for a bit
that KVM does not support (VMX, which drops BTF and LBR instead) or that is
Expand All @@ -282,9 +289,36 @@ backend:
- KVM does not list the registers of the machine-check banks, so `machine_check`
follows `IA32_MCG_STATUS`, and the number of banks is the one that the VP's
`IA32_MCG_CAP` reports.

The MSHV, WHP, and OpenHCL backends do not have elements 17 through 20, and
clear their capabilities.
- On an AMD host, KVM keeps a guest's `VM_CR` and `VM_HSAVE_PA` whatever the
guest's CPUID says, though no CPU profile enumerates SVM. It lists
`VM_HSAVE_PA`, and not `VM_CR`, but it lists `VM_HSAVE_PA` wherever the host
can read it, as an Intel host can under a hypervisor that reads it as 0,
where KVM has no `VM_CR`. So `svm_msrs` needs KVM to run on SVM, as the vendor
of its supported CPUID shows, as well as the MSR in its list. A guest can
only write a page address that it has to `VM_HSAVE_PA`, while KVM takes any
value from the host, and cannot set `VM_CR`'s `SVMDIS` while `EFER.SVME` is
set, which KVM refuses from the host only after it has set `SVMDIS`, so a
restore refuses either before it writes either MSR. Once a guest sets
`VM_CR`'s `SVMDIS`, KVM keeps `SVMDIS` and `LOCK` for the life of the vCPU,
so a VM reset leaves them set, as it did before the element; a restore into
a new VP sets them as the guest saved them.
- KVM keeps the OSVW MSRs for a vCPU whose CPUID enumerates OSVW, as every AMD
CPU profile does, where KVM supports OSVW, as it does on an AMD host, and
lists neither, so `osvw` follows the guest's CPUID and KVM's supported CPUID.
A new VP holds values that KVM derives from the host's, which a snapshot
replaces with the ones that the guest saw, and which a VM reset puts back.

The MSHV and WHP backends hold `umwait_control` and `bndcfgs` as registers of
the VP, as they hold `spec_ctrl`, where the guest's CPUID enumerates WAITPKG or
MPX, which the hypervisor then presents to the guest; MSHV clears them for an
SNP guest, as KVM does. They do not have elements 18 through 20, and clear their
capabilities: the Microsoft hypervisor keeps none of those registers for a guest
to write (it drops the guest's writes of `IA32_DEBUGCTL` and `IA32_MISC_ENABLE`,
raises #GP for `MSR_MISC_FEATURES_ENABLES` and `IA32_POWER_CTL`, and gives the
guest no machine-check banks). They do not have elements 22 and 23 either: on
WHP, OpenVMM serves a guest's accesses of the OSVW MSRs itself, reading them as
0 and refusing writes, and neither backend reads AMD's SVM MSRs. OpenHCL does
not have elements 17 through 23.

A saved state restores only into a partition that has every element that it
holds. A destination whose backend lacks one, such as a host whose KVM does not
Expand All @@ -306,26 +340,19 @@ also fails for a state that holds a value that KVM refuses, such as a bank
control other than 0 or all ones (bit 0 and bit 10 may be clear), which no guest
writes, after it has written the registers that precede it.

Some guest-writable state has no element:

- `IA32_BNDCFGS` belongs to MPX, which only the Skylake-SP CPU profile
enumerates. Since Linux 5.6, the kernel does not enable the MPX state
components in the host's XCR0, which KVM needs to expose MPX, so that CPU
profile cannot run on a KVM host that the time ABI supports, which needs
Linux 5.16.
- `MSR_VM_HSAVE_PA` belongs to SVM, which no CPU profile enumerates. A guest
without SVM cannot run VMRUN, so nothing uses the MSR.
- AMD's legacy performance counters, `0xc0010000` through `0xc0010007`, hold
state only on an AMD host whose KVM virtual PMU is on. A time ABI guest has no
PMU, so the KVM backend raises #GP for them instead (see [Time and CPU
compatibility](../user_guide/openvmm/snapshots.md#time-and-cpu-compatibility)),
and a snapshot of another guest does not hold their state.

A snapshot from an earlier build holds none of elements 17 through 20, so a
Some guest-writable state has no element: AMD's legacy performance counters,
`0xc0010000` through `0xc0010007`, hold state only on an AMD host whose KVM
virtual PMU is on. A time ABI guest has no PMU, so the KVM backend raises #GP
for them instead (see [Time and CPU
compatibility](../user_guide/openvmm/snapshots.md#time-and-cpu-compatibility)),
and a snapshot of another guest does not hold their state.

A snapshot from an earlier build holds none of elements 17 through 23, so a
restore leaves those MSRs as `bind` set them, as it did before:
`IA32_MISC_ENABLE` at the Microsoft hypervisor's value, the machine-check banks
as a new VP has them, and the other MSRs at 0. An earlier build ignores the
elements of a later snapshot in the same way.
as a new VP has them, the OSVW MSRs at KVM's values for the host, and the other
MSRs at 0. An earlier build ignores the elements of a later snapshot in the same
way.

### Fields of an element

Expand Down Expand Up @@ -372,6 +399,15 @@ save records it as the activity's interruption alone and holds no event for it:
A restore refuses any other pending exception, which KVM would drop, before it
changes the activity. Only a damaged or crafted snapshot holds one.

Nor does a software interruption need a field, since no save holds one. A
Hyper-V VP that the hypervisor stops while it delivers a software interrupt
(`INT n`), a software exception (`INT3` or `INTO`), or a privileged software
exception (`INT1`) holds the event with the length of the instruction that
raised it, which the activity has no place for, so MSHV and WHP refuse to save
it rather than drop it, as they refuse a pending event other than an exception
or an extint, or one whose vector is not 8 bits. KVM reports no such event: it
has the guest run the instruction again.

KVM on an Intel CPU can hold an interrupt shadow of either kind, which the
Boolean `interrupt_shadow` cannot say: the shadow that a `MOV SS` or `POP SS`
leaves also blocks NMIs and suppresses certain debug exceptions at the next
Expand Down
12 changes: 9 additions & 3 deletions Guide/src/dev_guide/tests/vmm.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,17 @@ The other guest-writable MSRs that the VP state holds, where the backend can
access them, have cases of their own in `msr_state.rs`:
`misc_msrs_survive_a_save_and_restore`,
`umwait_control_survives_a_save_and_restore` (which needs a host with WAITPKG),
`bndcfgs_survives_a_save_and_restore` (which needs a host with MPX),
`svm_msrs_survive_a_save_and_restore` and `osvw_survives_a_save_and_restore`
(which need an AMD host),
`debug_ctl_survives_a_save_and_restore`, and
`machine_check_state_survives_a_save_and_restore`, each of which sets the state,
saves it, changes the VP, and restores it, and two that check that a reset
returns the feature MSRs and the machine-check state to what a new VP holds.
MSHV and WHP do not hold these elements, so these cases report ignored there.
saves it, changes the VP, and restores it, and three that check that a reset
returns the feature MSRs, the machine-check state, and the OSVW MSRs to what a
new VP holds.
MSHV and WHP hold `IA32_UMWAIT_CONTROL` and `IA32_BNDCFGS`, whose cases run
there on a host with WAITPKG or MPX, and none of the other elements, so their
cases report ignored there.
A restore refuses a saved state that holds an element that the partition lacks,
such as SynIC state for a partition without Hyper-V enlightenments: it fails
with an error that names the element, before it changes any state.
Expand Down
21 changes: 17 additions & 4 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,15 @@ SEV-SNP VM, keeps its RAM in the normal address space alone, since KVM does not
support SMM with private memory. A capture that meets a vCPU in its handler
fails, and the guest can request it again once the handler has executed `RSM`.

A save on MSHV or WHP fails for a vCPU that the hypervisor stopped while it
delivered a software interrupt (`INT n`), a software exception (`INT3` or
`INTO`), or a privileged software exception (`INT1`): the hypervisor holds the
event with the length of the instruction that raised it, which the saved state
has no place for, so a restore would lose it. KVM holds no such event, and runs
the instruction again instead. A guest-requested capture that fails this way
rolls back, so the guest continues, takes the event, and can request the
capture again.

While a guest-requested snapshot boundary is held, VM-worker management RPCs
that can change VM state are rejected rather than queued. This includes memory
writes, pause/resume, reset, interrupt injection, hotplug, and state dumps.
Expand Down Expand Up @@ -831,7 +840,11 @@ immediately with a clear error if any active device does not support it.
elements](../../dev_guide/snapshot_format.md#vp-state-elements) for the list.
A snapshot from a build that predates one of them leaves that state as a new
VP has it, and a build that predates it ignores it in a later snapshot, so
neither restore fails, and neither keeps what the guest wrote. Only a host
whose KVM can access an MSR saves it, and a snapshot restores only on a host
that can access every MSR that it holds: on another host, the restore is
refused, with an error that names the element, before it changes the VP.
neither restore fails, and neither keeps what the guest wrote. Only a backend
that can access an MSR saves it: KVM where the host lists the MSR, or, for
AMD's `VM_CR` and OSVW MSRs, which KVM does not list, where it keeps them, and
MSHV and WHP only `IA32_SPEC_CTRL`, `IA32_UMWAIT_CONTROL`, and
`IA32_BNDCFGS`, the only ones of these MSRs that their hypervisor keeps for a
guest; a snapshot restores only on a host that can access every MSR that it
holds: on another host, the restore is refused, with an error that names the
element, before it changes the VP.
6 changes: 4 additions & 2 deletions openhcl/virt_mshv_vtl/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2508,10 +2508,12 @@ impl UhPartition {
// Hardware-isolated guests use our software APIC, which does not
// implement the architectural TSC-deadline MSR.
caps.tsc_deadline &= !isolation.is_hardware_isolated();
// The VP state access of OpenHCL does not hold SPEC_CTRL or
// UMWAIT_CONTROL.
// The VP state access of OpenHCL does not hold SPEC_CTRL,
// UMWAIT_CONTROL, BNDCFGS, or AMD's OSVW MSRs.
caps.spec_ctrl = false;
caps.umwait_control = false;
caps.bndcfgs = false;
caps.osvw = false;
match isolation {
IsolationType::Tdx => {
assert_eq!(caps.vtom.is_some(), !hide_isolation);
Expand Down
24 changes: 24 additions & 0 deletions openhcl/virt_mshv_vtl/src/processor/mshv/x64.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1717,6 +1717,30 @@ impl AccessVpState for UhVpStateAccess<'_, '_, HypervisorBackedX86> {
Err(vp_state::Error::Unimplemented("machine_check"))
}

fn bndcfgs(&mut self) -> Result<vp::Bndcfgs, Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn set_bndcfgs(&mut self, _value: &vp::Bndcfgs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn svm_msrs(&mut self) -> Result<vp::SvmMsrs, Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn set_svm_msrs(&mut self, _value: &vp::SvmMsrs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn osvw(&mut self) -> Result<vp::Osvw, Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn set_osvw(&mut self, _value: &vp::Osvw) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn synic_msrs(&mut self) -> Result<vp::SyntheticMsrs, Self::Error> {
self.get_register_state()
}
Expand Down
24 changes: 24 additions & 0 deletions openhcl/virt_mshv_vtl/src/processor/snp/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3007,6 +3007,30 @@ impl AccessVpState for UhVpStateAccess<'_, '_, SnpBacked> {
Err(vp_state::Error::Unimplemented("machine_check"))
}

fn bndcfgs(&mut self) -> Result<vp::Bndcfgs, Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn set_bndcfgs(&mut self, _value: &vp::Bndcfgs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn svm_msrs(&mut self) -> Result<vp::SvmMsrs, Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn set_svm_msrs(&mut self, _value: &vp::SvmMsrs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn osvw(&mut self) -> Result<vp::Osvw, Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn set_osvw(&mut self, _value: &vp::Osvw) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn cet(&mut self) -> Result<vp::Cet, Self::Error> {
let vmsa = self.vp.runner.vmsa(self.vtl);
Ok(vp::Cet { scet: vmsa.s_cet() })
Expand Down
24 changes: 24 additions & 0 deletions openhcl/virt_mshv_vtl/src/processor/tdx/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4266,6 +4266,30 @@ impl AccessVpState for UhVpStateAccess<'_, '_, TdxBacked> {
Err(vp_state::Error::Unimplemented("machine_check"))
}

fn bndcfgs(&mut self) -> Result<vp::Bndcfgs, Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn set_bndcfgs(&mut self, _value: &vp::Bndcfgs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("bndcfgs"))
}

fn svm_msrs(&mut self) -> Result<vp::SvmMsrs, Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn set_svm_msrs(&mut self, _value: &vp::SvmMsrs) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("svm_msrs"))
}

fn osvw(&mut self) -> Result<vp::Osvw, Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn set_osvw(&mut self, _value: &vp::Osvw) -> Result<(), Self::Error> {
Err(vp_state::Error::Unimplemented("osvw"))
}

fn cet(&mut self) -> Result<vp::Cet, Self::Error> {
Err(vp_state::Error::Unimplemented("cet"))
}
Expand Down
4 changes: 4 additions & 0 deletions vm/x86/x86defs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,8 @@ pub const X86X_MSR_MTRR_DEF_TYPE: u32 = 0x2ff;
/// has four, `CTL`, `STATUS`, `ADDR`, and `MISC`, in that order.
pub const X86X_MSR_MC0_CTL: u32 = 0x400;

/// `IA32_BNDCFGS`, MPX's supervisor-mode configuration.
pub const X86X_MSR_BNDCFGS: u32 = 0xd90;
pub const X86X_MSR_XSS: u32 = 0xda0;

pub const X86X_IA32_MSR_RAPL_POWER_UNIT: u32 = 0x606;
Expand Down Expand Up @@ -346,6 +348,8 @@ pub const X86X_AMD_MSR_HW_CFG: u32 = 0xC0010015;
pub const X86X_AMD_MSR_NB_CFG: u32 = 0xC001001F;
pub const X86X_AMD_MSR_VM_CR: u32 = 0xC0010114;
pub const X86X_AMD_MSR_IGNNE: u32 = 0xC0010115;
/// `VM_HSAVE_PA`, the physical address of SVM's host save area.
pub const X86X_AMD_MSR_VM_HSAVE_PA: u32 = 0xC0010117;
pub const X86X_AMD_MSR_GHCB: u32 = 0xC0010130;
pub const X86X_AMD_MSR_SEV: u32 = 0xC0010131;
pub const X86X_AMD_MSR_SECURE_AVIC_CONTROL: u32 = 0xc0010138;
Expand Down
Loading
Loading