Skip to content

openvmm: KVM snapshots lose MXCSR on hosts without XSAVEC or XSAVES #1389

Closed as not planned

Description

@tangruize

On a host whose kernel uses neither XSAVEC nor XSAVES, KVM_GET_XSAVE and KVM_GET_XSAVE2 report MXCSR as 0x1f80 whenever the guest's XMM registers and the upper halves of its YMM registers are in their init state, whatever the guest's real MXCSR. Such hosts are Intel before Skylake, AMD before Zen, an L1 host whose hypervisor hides both instructions, or a kernel with both disabled on its command line. A snapshot taken then restores the guest with the default MXCSR: its rounding control, FTZ, DAZ and exception masks are lost. Without a snapshot the guest keeps its MXCSR. This is a Linux regression that is not fixed upstream; hosts whose kernel uses XSAVEC or XSAVES, which is most current ones, are not affected.

Affected host kernels

Host kernel MXCSR that KVM_GET_XSAVE reports when XSTATE_BV[1] and XSTATE_BV[2] are clear
using XSAVEC or XSAVES, any version the guest's (the CPU sets XSTATE_BV[1] whenever MXCSR is not 0x1f80)
using XSAVE or XSAVEOPT, 5.16 and later (7.3 included) 0x1f80
using XSAVE or XSAVEOPT, before 5.16 the guest's (KVM copied the legacy area as it was)

ptrace (NT_X86_XSTATE, NT_PRFPREG) and core dumps have reported 0x1f80 the same way since 5.14.

Reproduction

Not reproduced; it needs a host kernel without XSAVEC and XSAVES. On one, run a guest that sets MXCSR to a value other than 0x1f80 (for example 0x7f80, rounding toward zero), with its XMM registers zero and its AVX upper halves in their init state, then snapshot and restore it and read MXCSR in the restored guest. Expected: 0x7f80. Actual: 0x1f80.

Cause

The kernel builds the uabi copy's header from the fpstate's XSTATE_BV and copies MXCSR only when it has SSE or YMM set:

// arch/x86/kernel/fpu/xstate.c (Linux 7.3)
	/* Copy MXCSR when SSE or YMM are set in the feature mask */
	copy_feature(header.xfeatures & (XFEATURE_MASK_SSE | XFEATURE_MASK_YMM),
		     &to, &xsave->i387.mxcsr, &xinit->i387.mxcsr,
		     MXCSR_AND_FLAGS_SIZE);

That rule holds for the compacted format, where XSAVEC and XSAVES set XSTATE_BV[1] whenever MXCSR is not 0x1f80. In the standard format, which the kernel uses without XSAVEC and XSAVES, XSTATE_BV[1] tracks the XMM registers only, while XSAVE stores MXCSR and the standard form of XRSTOR loads it in any case. So the fpstate holds the live MXCSR and the copy reports 0x1f80. KVM_GET_XSAVE and KVM_GET_XSAVE2 go through this copy (fpu_copy_guest_fpstate_to_uabi). It regressed when KVM's own copy of the legacy area was replaced by it (bf5d00470787, 5.16), and ptrace and core dumps when they moved to it (3a3351126ee8 and adc997b3d66d, 5.14).

OpenVMM cannot work around it: KvmVpStateAccess::xsave (vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs) saves the area that get_xsave returns, and no other KVM interface reports MXCSR (KVM_GET_FPU leaves struct kvm_fpu's mxcsr unset).

Fix

A patch for the x86/fpu code is attached below; it has not been sent upstream yet. On kernels that use the standard format only, it marks SSE as in use in the uabi header when MXCSR is not 0x1f80, as XSAVEC and XSAVES do in hardware. The XMM registers are still taken from the fpstate only if its own XSTATE_BV[1] is set, because XSAVEOPT leaves stale values in an area whose XMM registers are in their init state. A standard-format kernel then exports what a compacted-format kernel already exports for the same state, and KVM_SET_XSAVE takes MXCSR back from it. The change is +20/−2 in __copy_xstate_to_uabi_buf. It applies to tip master, v7.3, 6.18.y and 6.12.y; it is compile-tested only.

Related: #1360.

Activity

  1. tangruize commented on Nov 8, 2026

    @tangruize
    ContributorAuthor

    Proposed Linux patch

    Patch for the x86/fpu code (tip tree), against tip master 4f08fc2dfebccc28b7a7645dd81d28a755815e47; it also applies to v7.3. Compile-tested only, and not yet sent upstream. Save the contents below as linux.patch and run git apply linux.patch in a kernel tree.

    linux.patch
    diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
    index da303714379c..7ef755416114 100644
    --- a/arch/x86/kernel/fpu/xstate.c
    +++ b/arch/x86/kernel/fpu/xstate.c
    @@ -1172,6 +1172,19 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
     	memset(&header, 0, sizeof(header));
     	header.xfeatures = xsave->header.xfeatures;
     
    +	/*
    +	 * XSAVEC and XSAVES set XSTATE_BV[1] when MXCSR is not the default
    +	 * value. XSAVE and XSAVEOPT set it only when the XMM registers are
    +	 * not in their init state, but save MXCSR in any case, and the
    +	 * standard form of XRSTOR loads it in any case. Mark SSE as in use
    +	 * for a non-default MXCSR in the standard format as well, so that
    +	 * the live MXCSR is copied out instead of the default value and is
    +	 * taken back when the buffer is copied in again.
    +	 */
    +	if (!cpu_feature_enabled(X86_FEATURE_XCOMPACTED) &&
    +	    xsave->i387.mxcsr != MXCSR_DEFAULT)
    +		header.xfeatures |= XFEATURE_MASK_SSE;
    +
     	/* Mask out the feature bits depending on copy mode */
     	switch (copy_mode) {
     	case XSTATE_COPY_FP:
    @@ -1201,8 +1214,13 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,
     		     &to, &xsave->i387.st_space, &xinit->i387.st_space,
     		     sizeof(xsave->i387.st_space));
     
    -	/* Copy the SSE state - shared with YMM, but independently managed */
    -	copy_feature(header.xfeatures & XFEATURE_MASK_SSE,
    +	/*
    +	 * Copy the SSE state - shared with YMM, but independently managed.
    +	 * Copy the XMM registers from the buffer only if XSTATE_BV[1] is set
    +	 * there, as XSAVEOPT does not save them in their init state.
    +	 */
    +	copy_feature(header.xfeatures & xsave->header.xfeatures &
    +		     XFEATURE_MASK_SSE,
     		     &to, &xsave->i387.xmm_space, &xinit->i387.xmm_space,
     		     sizeof(xsave->i387.xmm_space));
     
    
  2. ppenna commented on Nov 15, 2026

    @ppenna
    Contributor

    Closing as not planned. This is not only a failed reproduction: the kernel defect is confirmed in the Linux source (v5.15, v5.16, v6.18, and master at fa2933aac170, 2026-11-15), where the uabi export drops MXCSR when XSTATE_BV has neither SSE nor AVX set; it lies outside NVX and OpenVMM, since no KVM interface reports the value and no refusal in check_saveable can be exact; and on the pinned-profile CPUs that I could measure, a guest's own instructions do not leave SSE clear with a non-default MXCSR. The fix belongs in the kernel. Two things need adding to the issue's analysis: which guests can reach the case on real CPUs, and which hosts use the standard format.

    The kernel

    • KVM_GET_XSAVE and KVM_GET_XSAVE2 both end in kvm_vcpu_ioctl_x86_get_xsave2(), fpu_copy_guest_fpstate_to_uabi(), and __copy_xstate_to_uabi_buf(..., XSTATE_COPY_XSAVE). It copies the fpstate's MXCSR only if (header.xfeatures & (XFEATURE_MASK_SSE | XFEATURE_MASK_YMM)) and writes 0x1f80 otherwise. Linux 5.16 moved KVM onto this path (bf5d00470787); before it, fill_xsave() copied the legacy area as it was. master has the same code.
    • No other KVM interface reports MXCSR. kvm_arch_vcpu_ioctl_get_fpu() fills fpr, fcw, fsw, ftwx, last_opcode, last_ip, last_dp, and xmm, and never mxcsr; the ioctl kzallocs struct kvm_fpu, so its mxcsr reads 0 (the same in 5.15). KVM_GET_ONE_REG takes MSRs only, and MXCSR is not an MSR. KVM_GET_REGS, KVM_GET_SREGS, KVM_GET_SREGS2, KVM_GET_MSRS, KVM_GET_XCRS, KVM_GET_DEBUGREGS, KVM_GET_VCPU_EVENTS, KVM_GET_LAPIC, and KVM_GET_NESTED_STATE hold none either. The live MXCSR exists only in the kernel's fpstate, behind the copy above, so KvmVpStateAccess::xsave has no second source to read it from.
    • The kernel uses the standard format when neither XSAVEC nor XSAVES is enabled (X86_FEATURE_XCOMPACTED is set when either is). Both xsavec and xsaves are then missing from the flags in /proc/cpuinfo, and the kernel logs its choice at boot: x86/fpu: Enabled xstate features ..., using 'standard' format. The kernel has used XSAVEC since 5.19 (8ad7e8f69695). In 5.16 to 5.18 the compacted format meant XSAVES only, so a CPU with XSAVEC but not XSAVES ran the standard format there.

    Which guests lose it

    The issue's table is right about what the kernel exports when XSTATE_BV[1] and [2] are clear. Whether a guest reaches that case with a non-default MXCSR depends on the CPU. XSAVE and XSAVEOPT write XSTATE_BV[1] from XINUSE[1], which the SDM defines for the XMM registers only ("XSTATE_BV[1] may be written with 0 even if MXCSR does not have its RESET value of 1F80H", Vol. 1 section 13.9), while XSAVEC and XSAVES set it for a non-default MXCSR (sections 13.10 and 13.11). The first program below changes MXCSR in user space, saves with each instruction, and prints what it wrote. It ran on a Core i9-12900H (Alder Lake, Windows 11), a Xeon Platinum 8370C (Ice Lake-SP, Windows build 26100), and a Xeon Platinum 8573C (Emerald Rapids, Linux 6.18), the CPUs of three pinned profiles, with the same result on all three. xsave, xsaveopt, and xsavec wrote this to the area's XSTATE_BV after MXCSR became 0x7f80 with the XMM registers zero:

    MXCSR set by xsave and xsaveopt write XSTATE_BV xsavec writes
    LDMXCSR, VLDMXCSR 0x2 0x2
    FXRSTOR 0x3 0x3
    XRSTOR, standard form, XSTATE_BV=0 0x0 0x2
    XRSTOR, standard form, XSTATE_BV=1 0x1 0x3

    On these CPUs the instructions that a guest ordinarily uses to change MXCSR mark SSE in use, so a standard-format kernel exports the guest's MXCSR. The SSE bit stays clear only when MXCSR came from an XRSTOR of an area whose SSE bit is clear, which those instructions do not do. KVM does it on a standard-format kernel when KVM_SET_XSAVE takes an area with none of x87, SSE, and AVX: it keeps the MXCSR that its vCPU held, and the standard form of XRSTOR loads it. A VM reset left a stale MXCSR that way (#1360) until #1641 changed OpenVMM's restore. I could not measure AMD CPUs or Intel CPUs before Skylake-SP, so the issue's list of standard-format hosts is an upper bound on the affected ones that I can neither confirm nor exclude.

    The issue's reproduction, on nvx-kvm-test (Xeon 8573C, Linux 6.18.57, XSAVES), with the second program below, which uses raw KVM ioctls: a real-mode guest runs LDMXCSR with 0x7f80 and leaves XMM and AVX alone; KVM_GET_XSAVE returns XSTATE_BV=0x2 and MXCSR=0x7f80; a guest in a fresh vCPU that the area is restored into reads 0x7f80. As the issue says, a host that uses XSAVES is not affected, and no available host uses the standard format.

    Why OpenVMM cannot work around it

    • Recovering the value: no interface reports it, as above. OpenVMM's saved area is the compact format, in which MXCSR counts only when SSE or AVX is set (Xsave::normalize clears it otherwise, and write_standard restores 0x1f80), the same contract as the kernel's export, so the value is gone before OpenVMM reads the area.
    • Refusing exactly: with both bits clear, the area's MXCSR is 0x1f80 by construction, so a check cannot tell a default MXCSR from a lost one. It could only refuse every capture whose SSE and AVX state is in init state on a host that it takes for standard-format. That rejects the common case, a guest with the default MXCSR, to guard a case that the CPUs measured above do not produce.
    • Telling the hosts apart: user space sees the format only through the /proc/cpuinfo flags and dmesg, and the format does not decide the outcome: the CPU's XINUSE tracking does. Only a probe guest in a scratch VM would separate the hosts.

    NVX's pinned CPU profiles cover Intel Skylake-SP, Ice Lake-SP, Emerald Rapids, Granite Rapids, and Alder Lake, and AMD Milan, Genoa, and Turin. Those CPUs have both instructions, and every profile enumerates them to the guest (CPUID.(0xd,1).EAX is 0xf), so a standard-format host among them needs a hypervisor that hides both instructions or a kernel command line that disables both. Any other Intel or AMD CPU runs on a host profile, which the host checks of doctor do not qualify.

    Where the fix belongs

    __copy_xstate_to_uabi_buf() in arch/x86/kernel/fpu/xstate.c, and copy_uabi_to_xstate() for the write direction. The x86/fpu code goes through the tip tree: x86@kernel.org and linux-kernel@vger.kernel.org, with kvm@vger.kernel.org and the KVM/x86 maintainers on cc, since KVM_GET_XSAVE is their ABI. Intel's Chang S. Bae posted the same format-dependent correction in 2022, x86/fpu: Fix MXCSR handling and SSE component definition (v3), after gVisor's report of the write direction on hosts without XSAVES. It did not reach mainline: master still has the code above. The patch in this issue and that series fix the same dependence on the format, so a report to the kernel should cite both.

    The restore direction differs by format too, and there user space can shape what it gives KVM_SET_XSAVE. OpenVMM now does: #1641 pins OpenVMM 23fb678876c9 (microsoft/openvmm#5118), whose xsave::restore zeroes the x87 and XMM bytes of a legacy component that the area's XSTATE_BV leaves out and, where KVM would keep the vCPU's MXCSR or PKRU through the area, writes a seed buffer first, so that KVM reads the area's clear bits as init state on every host. That fixes #1360. The description of #1641 also records a case that it leaves alone and that belongs to this family: an area with AVX set, SSE clear, and a non-default MXCSR, which only a standard-format host's save can hold, loses its MXCSR when it is restored onto a host that uses the compacted format.

    What would reopen this

    I could not measure AMD CPUs or Intel CPUs before Skylake-SP, and no such host is available, so this half stays open to evidence. Reopen the issue with the output of both programs from a host on which the first writes XSTATE_BV 0x0 in its LDMXCSR rows, or on which the second reports XSTATE_BV=0x0 and MXCSR=0x1f80 for the guest that ran LDMXCSR and 0x1f80 for the restored guest, together with that host's lscpu, uname -r, the x86/fpu: line of dmesg, and CPUID.(0xd,1).EAX. A refusal in check_saveable would then be the way to fail closed: the capture rolls back, and the saved state does not change. The host test would have to follow the CPU's behavior, not the kernel's format.

    xsave_mxcsr_probe.rs
    // Does XSAVE, XSAVEOPT or XSAVEC report SSE as in use (XSTATE_BV[1]) when the
    // XMM registers are in their init state and MXCSR is not 0x1f80?
    //
    // Each case starts from the init state (XRSTOR of an all-init area), changes
    // MXCSR one way, and saves into a fresh area. The XSAVE area's XSTATE_BV and
    // MXCSR print for each instruction. A kernel in the standard format, which uses
    // XSAVE or XSAVEOPT, exports MXCSR through KVM_GET_XSAVE only when XSTATE_BV
    // has SSE or AVX set.
    //
    //   rustc -O --edition 2021 xsave_mxcsr_probe.rs && ./xsave_mxcsr_probe
    use std::arch::asm;
    use std::arch::x86_64::__cpuid_count;
    
    #[repr(C, align(64))]
    #[derive(Clone)]
    struct Area([u8; 4096]);
    
    const MXCSR: usize = 24;
    const XSTATE_BV: usize = 512;
    
    impl Area {
        fn with(xstate_bv: u64, mxcsr: u32) -> Self {
            let mut area = Area([0; 4096]);
            area.0[MXCSR..MXCSR + 4].copy_from_slice(&mxcsr.to_le_bytes());
            area.0[XSTATE_BV..XSTATE_BV + 8].copy_from_slice(&xstate_bv.to_le_bytes());
            area
        }
        fn xstate_bv(&self) -> u64 {
            u64::from_le_bytes(self.0[XSTATE_BV..XSTATE_BV + 8].try_into().unwrap())
        }
        fn mxcsr(&self) -> u32 {
            u32::from_le_bytes(self.0[MXCSR..MXCSR + 4].try_into().unwrap())
        }
    }
    
    #[derive(Clone, Copy)]
    enum Save {
        Xsave,
        Xsaveopt,
        Xsavec,
    }
    
    enum Load {
        Ldmxcsr,
        Vldmxcsr,
        Fxrstor(Area),
        Xrstor(Area),
    }
    
    unsafe fn xgetbv0() -> u64 {
        let (lo, hi): (u32, u32);
        asm!("xgetbv", in("ecx") 0, out("eax") lo, out("edx") hi, options(nomem, nostack, preserves_flags));
        (u64::from(hi) << 32) | u64::from(lo)
    }
    
    unsafe fn xrstor(area: &Area, rfbm: u64) {
        asm!("xrstor [{p}]", p = in(reg) area.0.as_ptr(), in("eax") rfbm as u32, in("edx") (rfbm >> 32) as u32, options(nostack));
    }
    
    unsafe fn save(how: Save, area: &mut Area, rfbm: u64) {
        let p = area.0.as_mut_ptr();
        let (lo, hi) = (rfbm as u32, (rfbm >> 32) as u32);
        match how {
            Save::Xsave => asm!("xsave [{p}]", p = in(reg) p, in("eax") lo, in("edx") hi, options(nostack)),
            Save::Xsaveopt => asm!("xsaveopt [{p}]", p = in(reg) p, in("eax") lo, in("edx") hi, options(nostack)),
            Save::Xsavec => asm!("xsavec [{p}]", p = in(reg) p, in("eax") lo, in("edx") hi, options(nostack)),
        }
    }
    
    unsafe fn load(how: &Load, rfbm: u64) {
        static MXCSR_RZ: u32 = 0x7f80;
        match how {
            Load::Ldmxcsr => asm!("ldmxcsr [{p}]", p = in(reg) &MXCSR_RZ, options(nostack, readonly)),
            Load::Vldmxcsr => asm!("vldmxcsr [{p}]", p = in(reg) &MXCSR_RZ, options(nostack, readonly)),
            Load::Fxrstor(area) => asm!("fxrstor [{p}]", p = in(reg) area.0.as_ptr(), options(nostack)),
            Load::Xrstor(area) => xrstor(area, rfbm),
        }
    }
    
    fn main() {
        unsafe {
            let ext = __cpuid_count(0xd, 1).eax;
            let xcr0 = xgetbv0();
            // Leave out AMX's components and PKRU.
            let rfbm = xcr0 & 0xe7;
            println!("CPUID.(0xd,1).EAX={ext:#x} (XSAVEOPT bit 0, XSAVEC bit 1, XSAVES bit 3), XCR0={xcr0:#x}");
    
            let mut saves = vec![Save::Xsave];
            let mut names = vec!["xsave"];
            if ext & 1 != 0 {
                saves.push(Save::Xsaveopt);
                names.push("xsaveopt");
            }
            if ext & 2 != 0 {
                saves.push(Save::Xsavec);
                names.push("xsavec");
            }
    
            let init = Area::with(0, 0x1f80);
            let mut cases = vec![("MXCSR := 0x7f80 by LDMXCSR", Load::Ldmxcsr)];
            if xcr0 & 4 != 0 {
                cases.push(("MXCSR := 0x7f80 by VLDMXCSR", Load::Vldmxcsr));
            }
            cases.push(("FXRSTOR with MXCSR 0x7f80, XMM zero", Load::Fxrstor(Area::with(0, 0x7f80))));
            cases.push(("XRSTOR, XSTATE_BV=0, MXCSR 0x7f80", Load::Xrstor(Area::with(0, 0x7f80))));
            cases.push(("XRSTOR, XSTATE_BV=1 (x87), MXCSR 0x7f80", Load::Xrstor(Area::with(1, 0x7f80))));
    
            println!("{:<42} {:<9} {:>10} {:>8}", "case", "save", "XSTATE_BV", "MXCSR");
            for (name, how) in &cases {
                for (&s, &n) in saves.iter().zip(&names) {
                    let mut area = init.clone();
                    xrstor(&area, rfbm);
                    load(how, rfbm);
                    save(s, &mut area, rfbm);
                    xrstor(&init, rfbm);
                    println!(
                        "{:<42} {:<9} {:>10} {:>8}",
                        name,
                        n,
                        format!("{:#x}", area.xstate_bv()),
                        format!("{:#x}", area.mxcsr())
                    );
                }
            }
        }
    }
    
    kvm_mxcsr_e2e.rs
    // End-to-end check of what KVM_GET_XSAVE / KVM_SET_XSAVE do with a guest's MXCSR
    // on this host, with no VMM: a real-mode guest runs LDMXCSR, the program saves
    // the vCPU's XSAVE area, restores it into a fresh vCPU, and a guest in that vCPU
    // reads MXCSR back with STMXCSR.
    //
    //   rustc -O --edition 2021 kvm_mxcsr_e2e.rs && ./kvm_mxcsr_e2e
    use std::ffi::{c_char, c_int, c_ulong, c_void};
    
    extern "C" {
        fn open(path: *const c_char, flags: c_int, ...) -> c_int;
        fn ioctl(fd: c_int, request: c_ulong, ...) -> c_int;
        fn mmap(addr: *mut c_void, len: usize, prot: c_int, flags: c_int, fd: c_int, off: i64) -> *mut c_void;
    }
    
    const KVM_GET_API_VERSION: c_ulong = 0xae00;
    const KVM_CREATE_VM: c_ulong = 0xae01;
    const KVM_GET_VCPU_MMAP_SIZE: c_ulong = 0xae04;
    const KVM_CREATE_VCPU: c_ulong = 0xae41;
    const KVM_SET_USER_MEMORY_REGION: c_ulong = 0x4020_ae46;
    const KVM_RUN: c_ulong = 0xae80;
    const KVM_SET_REGS: c_ulong = 0x4090_ae82;
    const KVM_GET_SREGS: c_ulong = 0x8138_ae83;
    const KVM_SET_SREGS: c_ulong = 0x4138_ae84;
    const KVM_GET_XSAVE: c_ulong = 0x9000_aea4;
    const KVM_SET_XSAVE: c_ulong = 0x5000_aea5;
    const KVM_EXIT_HLT: u32 = 5;
    
    #[repr(C)]
    struct MemoryRegion {
        slot: u32,
        flags: u32,
        guest_phys_addr: u64,
        memory_size: u64,
        userspace_addr: u64,
    }
    
    #[repr(C)]
    #[derive(Default, Clone, Copy)]
    struct Regs([u64; 18]);
    
    #[repr(C)]
    #[derive(Clone, Copy)]
    struct Sregs([u64; 39]);
    
    const RIP: usize = 16;
    const RFLAGS: usize = 17;
    // kvm_sregs: eight 24-byte segments, two 16-byte tables, then cr0, cr2, cr3, cr4.
    const CS_BASE: usize = 0;
    const CS_SELECTOR_WORD: usize = 1; // limit (u32) then selector (u16) in word 1
    const CR4: usize = (8 * 24 + 2 * 16) / 8 + 3;
    
    #[repr(C, align(64))]
    struct Xsave([u8; 4096]);
    
    fn check(r: c_int, what: &str) -> c_int {
        if r < 0 {
            panic!("{what} failed: {}", std::io::Error::last_os_error());
        }
        r
    }
    
    struct Vm {
        vcpu: c_int,
        run: *mut u8,
        mem: *mut u8,
    }
    
    impl Vm {
        unsafe fn new(kvm: c_int, code: &[u8], data: u32) -> Vm {
            let vm = check(ioctl(kvm, KVM_CREATE_VM, 0 as c_ulong), "KVM_CREATE_VM");
            let mem = mmap(std::ptr::null_mut(), 0x10000, 3, 0x22, -1, 0) as *mut u8;
            assert!(mem as isize != -1);
            let region = MemoryRegion { slot: 0, flags: 0, guest_phys_addr: 0, memory_size: 0x10000, userspace_addr: mem as u64 };
            check(ioctl(vm, KVM_SET_USER_MEMORY_REGION, &region), "KVM_SET_USER_MEMORY_REGION");
            std::ptr::copy_nonoverlapping(code.as_ptr(), mem.add(0x1000), code.len());
            std::ptr::copy_nonoverlapping(data.to_le_bytes().as_ptr(), mem.add(0x1100), 4);
            let vcpu = check(ioctl(vm, KVM_CREATE_VCPU, 0 as c_ulong), "KVM_CREATE_VCPU");
            let size = check(ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, 0 as c_ulong), "KVM_GET_VCPU_MMAP_SIZE") as usize;
            let run = mmap(std::ptr::null_mut(), size, 3, 1, vcpu, 0) as *mut u8;
            assert!(run as isize != -1);
    
            // Real mode at 0:0x1000 with CR4.OSFXSR, so that SSE instructions run.
            let mut sregs = Sregs([0; 39]);
            check(ioctl(vcpu, KVM_GET_SREGS, &mut sregs), "KVM_GET_SREGS");
            sregs.0[CS_BASE] = 0;
            // limit (low 32 bits) and selector (next 16 bits)
            sregs.0[CS_SELECTOR_WORD] &= !(0xffff_u64 << 32);
            sregs.0[CR4] |= 0x200;
            check(ioctl(vcpu, KVM_SET_SREGS, &sregs), "KVM_SET_SREGS");
            let mut regs = Regs::default();
            regs.0[RIP] = 0x1000;
            regs.0[RFLAGS] = 2;
            check(ioctl(vcpu, KVM_SET_REGS, &regs), "KVM_SET_REGS");
            Vm { vcpu, run, mem }
        }
    
        unsafe fn run_to_hlt(&self) {
            check(ioctl(self.vcpu, KVM_RUN, 0 as c_ulong), "KVM_RUN");
            let reason = *(self.run.add(8) as *const u32);
            assert_eq!(reason, KVM_EXIT_HLT, "exit reason {reason}");
        }
    
        unsafe fn xsave(&self) -> Xsave {
            let mut area = Xsave([0; 4096]);
            check(ioctl(self.vcpu, KVM_GET_XSAVE, area.0.as_mut_ptr()), "KVM_GET_XSAVE");
            area
        }
    
        unsafe fn set_xsave(&self, area: &Xsave) -> c_int {
            ioctl(self.vcpu, KVM_SET_XSAVE, area.0.as_ptr())
        }
    
        unsafe fn data(&self) -> u32 {
            *(self.mem.add(0x1100) as *const u32)
        }
    }
    
    fn header(area: &Xsave) -> (u64, u32) {
        (
            u64::from_le_bytes(area.0[512..520].try_into().unwrap()),
            u32::from_le_bytes(area.0[24..28].try_into().unwrap()),
        )
    }
    
    // ldmxcsr [0x1100]; hlt
    const LDMXCSR_HLT: [u8; 6] = [0x0f, 0xae, 0x16, 0x00, 0x11, 0xf4];
    // stmxcsr [0x1100]; hlt
    const STMXCSR_HLT: [u8; 6] = [0x0f, 0xae, 0x1e, 0x00, 0x11, 0xf4];
    
    fn main() {
        unsafe {
            let kvm = check(open(c"/dev/kvm".as_ptr(), 2 | 0o2000000), "open /dev/kvm");
            assert_eq!(check(ioctl(kvm, KVM_GET_API_VERSION, 0 as c_ulong), "KVM_GET_API_VERSION"), 12);
    
            // 1. A guest sets MXCSR with LDMXCSR and leaves XMM and AVX in init state.
            let source = Vm::new(kvm, &LDMXCSR_HLT, 0x7f80);
            source.run_to_hlt();
            let saved = source.xsave();
            let (bv, mxcsr) = header(&saved);
            println!("guest ran LDMXCSR 0x7f80, XMM untouched: KVM_GET_XSAVE XSTATE_BV={bv:#x} MXCSR={mxcsr:#x}");
    
            // 2. The saved area, restored into a fresh vCPU, and read back by a guest.
            let target = Vm::new(kvm, &STMXCSR_HLT, 0);
            assert_eq!(target.set_xsave(&saved), 0);
            target.run_to_hlt();
            println!("restored into a fresh vCPU, guest STMXCSR: {:#x}", target.data());
    
            // 3. An area with x87 set, SSE and AVX clear, and 0x7f80 in its MXCSR
            // field: a standard-format kernel loads that MXCSR, and a compacted-format
            // kernel initializes it to 0x1f80.
            let mut crafted = Xsave([0; 4096]);
            crafted.0[24..28].copy_from_slice(&0x7f80u32.to_le_bytes());
            crafted.0[512..520].copy_from_slice(&1u64.to_le_bytes());
            let fresh = Vm::new(kvm, &STMXCSR_HLT, 0);
            let status = fresh.set_xsave(&crafted);
            fresh.run_to_hlt();
            println!(
                "KVM_SET_XSAVE with XSTATE_BV=1 (x87) and MXCSR 0x7f80 (status {status}), guest STMXCSR: {:#x}",
                fresh.data()
            );
        }
    }
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions