Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Guide/src/reference/devices/virtio/virtio-fs.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ fixes the remaining guest-visible configuration:
| Transport | virtio-mmio at `0xd0001000` |
| Interrupt | IRQ 6 |
| Queues | One high-priority and one request queue |
| Features | Indirect descriptors, event index, version 1, access-platform |
| Features | Indirect descriptors, event index, version 1 |
| DAX window | None |
| Cache policy | Zero entry and attribute lifetimes |
| File policy | Direct I/O |
Expand Down
23 changes: 16 additions & 7 deletions Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,18 @@ describes the source definitions.
`nvx_control_tty=hvc2` token. The reserved slot and resource identity do not
expose a live control endpoint. Both consoles
remain virtio-console devices from the guest's perspective. Ordinary
`--virtio-blk` is rejected. All use split rings. Firmware, ACPI, SMBIOS, PCI,
VMBus, UARTs, graphics, isolation, nested virtualization, and other devices
are rejected. Linux discovers contiguous APIC IDs and the IOAPIC from Intel
MP 1.4 tables at `0x0` and `0x400`; `boot_params` is at `0x2000`, the command
line starts at `0x20000`, and no ACPI or SMBIOS data is exposed. Host-driven
save/restore, pulse-save/restore, and worker restart remain unavailable.
`--virtio-blk` is rejected. All use split rings, and each offers an exact
feature mask of virtio 1.0 features: the NIC its MAC address; the block
roles a segment limit, block size, flush, and topology, and `distro`
read-only; and every device but the NIC indirect descriptors and event-index
notification suppression. No device offers `VIRTIO_F_ACCESS_PLATFORM`, since
the microVM has no IOMMU, and no console offers its size. Firmware, ACPI,
SMBIOS, PCI, VMBus, UARTs, graphics, isolation, nested virtualization, and
other devices are rejected. Linux discovers contiguous APIC IDs and the
IOAPIC from Intel MP 1.4 tables at `0x0` and `0x400`; `boot_params` is at
`0x2000`, the command line starts at `0x20000`, and no ACPI or SMBIOS data is
exposed. Host-driven save/restore, pulse-save/restore, and worker restart
remain unavailable.

`microvm` may also expose a dedicated control virtio-console at MMIO
`0xd0007000`, IRQ 3. It requires the boot virtio-console, preserves
Expand All @@ -94,7 +100,10 @@ describes the source definitions.
```admonish warning title="microVM migration"
The canonical `microvm` spelling now selects the contract formerly exposed
as `microvm-v2`; the `microvm-v2` selector and the former ABI-v1 behavior are
removed. Snapshot ABI and boot layout remain numeric value 2.
removed. Snapshots record microVM ABI value 3 and boot layout value 2. ABI
value 3 offers no device `VIRTIO_F_ACCESS_PLATFORM` and no console
`VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, so restore refuses a
snapshot of value 2, which must be recaptured.
```
* `--net <IPv4/PREFIX>`: With `--machine microvm`, attach one virtio-net NIC
at MMIO `0xd0000000`. KVM and MSHV use IRQ 10; WHP uses IRQ 5. Prefixes
Expand Down
8 changes: 5 additions & 3 deletions Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,9 +260,11 @@ opened. Clients must regenerate bindings because `machine_profile` moved from
field 15 to field 17 when upstream assigned field 15 to `iommufds`; the
colliding legacy encoding is rejected. Clients that used the former
`MICROVM_V2` source name must also update their bindings. RPC construction is
currently blockless; role-bearing sandbox blocks remain CLI-only. Snapshot ABI
and boot layout remain value 2, which identifies the Linux-direct MP-table
layout. Value 1 snapshots are unsupported.
currently blockless; role-bearing sandbox blocks remain CLI-only. Snapshots
record microVM ABI value 3 and boot layout value 2, which identifies the
Linux-direct MP-table layout. Snapshots of ABI values 1 and 2 are unsupported:
value 2 offered `VIRTIO_F_ACCESS_PLATFORM` on every device but the NIC, and the
console size on the consoles.

`VMConfig.cpu_profile` selects the microVM's CPU profile as
[`--cpu-profile`](cli.md) does, with the same values and checks: `auto`, the
Expand Down
4 changes: 3 additions & 1 deletion Guide/src/user_guide/openvmm/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,9 @@ NUMA, xAPIC mode, and contiguous APIC IDs `0..N-1`; APIC ID 0 is the BSP.
Custom socket, SMT, APIC, x2APIC, and NUMA options are rejected.

The microVM uses fixed virtio device slots and sandbox block roles. Its
persisted ABI and boot layout remain value 2. The
persisted ABI is value 3 and its boot layout value 2. Each fixed virtio device
offers an exact feature mask, which never includes `VIRTIO_F_ACCESS_PLATFORM`,
since the microVM has no IOMMU, nor packed virtqueues. The
MP floating pointer begins at `0x0`, the MP configuration table at `0x400`,
the boot GDT at `0x1000`, and the Linux zero page at `0x2000`. No ACPI MADT or
SMBIOS data is exposed.
Expand Down
5 changes: 3 additions & 2 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -604,8 +604,9 @@ immediately with a clear error if any active device does not support it.
- VMs using PCAT firmware do not support save/restore
- Standard-machine restore still requires matching `--memory` and
`--processors`. MicroVM restore reads them authoritatively from the manifest
and rejects overrides. Persisted microVM ABI and boot-layout value 2 are
supported.
and rejects overrides. Persisted microVM ABI value 3 and boot-layout value 2
are supported; a snapshot of ABI value 2, whose devices offered
`VIRTIO_F_ACCESS_PLATFORM`, must be recaptured.
- MicroVM restore requires the backend, CPU generation, and CPU profile of
the capture, and an OpenVMM build that computes the same effective CPUID.
A host profile's snapshot, including one of a cold boot whose `auto` fell
Expand Down
195 changes: 177 additions & 18 deletions openvmm/openvmm_core/src/worker/dispatch/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -472,40 +472,53 @@ impl LoadedVm {
}
}

/// Returns the fixed slot of the microVM virtio device `id`: its MMIO window,
/// interrupt, exact guest-visible feature mask, and interrupt-status delivery.
fn virtio_mmio_config(
id: &str,
sandbox_blocks: &[openvmm_defs::microvm::MicrovmSandboxBlockConfig],
sandbox_block_index: &mut usize,
filesystem_index: &mut usize,
chipset_mmio: ChipsetMmioRanges,
) -> anyhow::Result<(u64, u64, u32, u64, VirtioMmioInterruptMode)> {
let (start, irq) = match id {
let (start, irq, features) = match id {
"virtio-net" => (
openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE,
openvmm_defs::microvm::microvm_virtio_net_irq(None)?,
openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES,
),
// The configuration orders virtio-fs devices by slot.
"virtiofs" => {
let slot = openvmm_defs::microvm::MICROVM_FILESYSTEM_SLOTS
.get(*filesystem_index)
.context("microVM has more virtio-fs devices than fixed slots")?;
*filesystem_index += 1;
(slot.mmio_base, slot.irq)
(
slot.mmio_base,
slot.irq,
openvmm_defs::microvm::MICROVM_VIRTIO_FS_FEATURES,
)
}
"virtio-console" => (
openvmm_defs::microvm::MICROVM_VIRTIO_CONSOLE_MMIO_BASE,
openvmm_defs::microvm::MICROVM_VIRTIO_CONSOLE_IRQ,
openvmm_defs::microvm::MICROVM_VIRTIO_CONSOLE_FEATURES,
),
openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID => (
openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE,
openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ,
openvmm_defs::microvm::MICROVM_VIRTIO_CONSOLE_FEATURES,
),
"virtio-blk" => {
let block = sandbox_blocks
.get(*sandbox_block_index)
.context("microVM virtio-blk device has no sandbox role")?;
*sandbox_block_index += 1;
(block.role.mmio_base(), block.role.irq())
(
block.role.mmio_base(),
block.role.irq(),
openvmm_defs::microvm::microvm_sandbox_block_features(block.role),
)
}
_ => anyhow::bail!("unsupported microVM virtio device '{id}' reached worker construction"),
};
Expand All @@ -517,23 +530,11 @@ fn virtio_mmio_config(
.is_some_and(|end| end <= chipset_mmio.low.end()),
"microVM virtio slot for '{id}' is outside the fixed low-MMIO aperture"
);
let disabled_features = match id {
"virtio-net" => !openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES,
"virtiofs" => !openvmm_defs::microvm::MICROVM_VIRTIO_FS_FEATURES,
"virtio-blk" => {
let block = sandbox_blocks
.iter()
.find(|block| block.role.mmio_base() == start)
.context("microVM block slot has no role")?;
!openvmm_defs::microvm::microvm_sandbox_block_features(block.role)
}
_ => 1 << 34,
};
let interrupt_mode = VirtioMmioInterruptMode::SharedStatus {
status_gpa: openvmm_defs::microvm::microvm_virtio_status_gpa(start)
.context("microVM slot has no shared-status word")?,
};
Ok((start, len, irq, disabled_features, interrupt_mode))
Ok((start, len, irq, features, interrupt_mode))
}

/// The guest memory that the microVM e820 map reserves inside RAM besides the
Expand Down Expand Up @@ -602,7 +603,7 @@ impl<'a> VirtioMmioSlots<'a> {
id: &str,
device: ResolvedVirtioDevice,
) -> anyhow::Result<()> {
let (mmio_start, mmio_len, irq, disabled_features, interrupt_mode) = virtio_mmio_config(
let (mmio_start, mmio_len, irq, features, interrupt_mode) = virtio_mmio_config(
id,
self.sandbox_blocks,
&mut self.sandbox_block_index,
Expand All @@ -625,7 +626,7 @@ impl<'a> VirtioMmioSlots<'a> {
partition.clone().into_doorbell_registration(Vtl::Vtl0),
mmio_start,
mmio_len,
disabled_features,
!features,
interrupt_mode,
)
.context("failed to create the virtio-mmio transport")?;
Expand Down Expand Up @@ -803,3 +804,161 @@ pub(super) fn x86_topology_builder(
Ok(vm_topology::processor::TopologyBuilder::from_host_topology()?)
}
}

// The microVM runs on Linux and Windows hosts alone, where its NIC has an
// interrupt.
#[cfg(all(test, any(target_os = "linux", windows)))]
mod tests {
use super::virtio_mmio_config;
use crate::worker::memory_layout::ChipsetMmioRanges;
use chipset_device::io::IoResult;
use chipset_device::mmio::MmioIntercept;
use guestmem::GuestMemory;
use inspect::InspectMut;
use memory_range::MemoryRange;
use openvmm_defs::microvm::MicrovmSandboxBlockConfig;
use openvmm_defs::microvm::MicrovmSandboxBlockRole;
use pal_async::DefaultDriver;
use pal_async::async_test;
use virtio::DeviceTraits;
use virtio::QueueResources;
use virtio::VirtioDevice;
use virtio::VirtioMmioDevice;
use virtio::queue::QueueState;
use virtio::spec::VirtioDeviceFeatures;
use virtio::spec::mmio::VirtioMmioRegister;
use vmcore::line_interrupt::LineInterrupt;
use vmcore::vm_task::SingleDriverBackend;
use vmcore::vm_task::VmTaskDriverSource;

/// A device that offers every feature, so that its transport offers the
/// guest exactly the features that the microVM configuration admits.
#[derive(InspectMut)]
struct EveryFeatureDevice;

impl VirtioDevice for EveryFeatureDevice {
fn traits(&self) -> DeviceTraits {
DeviceTraits {
device_features: VirtioDeviceFeatures::from_bits(u64::MAX),
max_queues: 1,
..Default::default()
}
}

async fn read_registers_u32(&mut self, _offset: u16) -> u32 {
0
}

async fn write_registers_u32(&mut self, _offset: u16, _val: u32) {}

async fn start_queue(
&mut self,
_idx: u16,
_resources: QueueResources,
_features: &VirtioDeviceFeatures,
_initial_state: Option<QueueState>,
) -> anyhow::Result<()> {
anyhow::bail!("the test never starts a queue")
}

async fn stop_queue(&mut self, _idx: u16) -> Option<QueueState> {
None
}
}

/// Returns the numbers of the feature bits that the transport at
/// `mmio_base` offers the guest, read as a driver reads them, one 32-bit
/// bank at a time.
fn offered_feature_bits(device: &mut VirtioMmioDevice, mmio_base: u64) -> Vec<u32> {
let select = mmio_base + u64::from(VirtioMmioRegister::DEVICE_FEATURES_SEL.0);
let register = mmio_base + u64::from(VirtioMmioRegister::DEVICE_FEATURES.0);
let mut features = 0;
for bank in 0..2u32 {
assert!(matches!(
device.mmio_write(select, &bank.to_le_bytes()),
IoResult::Ok
));
let mut value = [0; 4];
assert!(matches!(
device.mmio_read(register, &mut value),
IoResult::Ok
));
features |= u64::from(u32::from_le_bytes(value)) << (32 * bank);
}
(0..u64::BITS)
.filter(|bit| features & (1 << bit) != 0)
.collect()
}

#[async_test]
async fn microvm_devices_offer_exactly_their_feature_masks(driver: DefaultDriver) {
let driver_source = VmTaskDriverSource::new(SingleDriverBackend::new(driver));
// The memory holds the page of the slots' shared interrupt-status
// words.
let gm = GuestMemory::allocate(
(openvmm_defs::microvm::MICROVM_SHARED_STATUS_PAGE_GPA
+ openvmm_defs::microvm::MICROVM_SHARED_STATUS_PAGE_SIZE) as usize,
);
let chipset_mmio = ChipsetMmioRanges {
low: MemoryRange::new(0xc000_0000..0x1_0000_0000),
high: MemoryRange::EMPTY,
vtl2: MemoryRange::EMPTY,
};
let blocks = [
MicrovmSandboxBlockConfig {
role: MicrovmSandboxBlockRole::Distro,
read_only: true,
},
MicrovmSandboxBlockConfig {
role: MicrovmSandboxBlockRole::Scratch,
read_only: false,
},
];
let mut sandbox_block_index = 0;
let mut filesystem_index = 0;
// Each device in the order of the slots of its kind, with its window
// and the features that the guest sees: never access-platform, which
// the transport offers every device, nor the console size, which the
// virtio-console device offers.
let devices: [(&str, u64, &[u32]); 6] = [
("virtio-net", 0xd000_0000, &[5, 32]),
("virtiofs", 0xd000_1000, &[28, 29, 32]),
("virtio-console", 0xd000_2000, &[28, 29, 32]),
("virtio-blk", 0xd000_3000, &[2, 5, 6, 9, 10, 28, 29, 32]),
("virtio-blk", 0xd000_6000, &[2, 6, 9, 10, 28, 29, 32]),
(
openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID,
0xd000_7000,
&[28, 29, 32],
),
];
for (id, mmio_base, bits) in devices {
let (start, len, _irq, features, interrupt_mode) = virtio_mmio_config(
id,
&blocks,
&mut sandbox_block_index,
&mut filesystem_index,
chipset_mmio,
)
.unwrap();
assert_eq!(start, mmio_base, "{id}");
let mut device = VirtioMmioDevice::new_with_disabled_features_and_interrupt_mode(
Box::new(EveryFeatureDevice),
&driver_source.simple(),
gm.clone(),
LineInterrupt::detached(),
None,
start,
len,
!features,
interrupt_mode,
)
.unwrap();
assert_eq!(
offered_feature_bits(&mut device, start),
bits,
"{id} at {start:#x}"
);
}
}
}
Loading
Loading