diff --git a/Guide/src/reference/devices/firmware/linux_direct.md b/Guide/src/reference/devices/firmware/linux_direct.md index 06c34ba94e6..e8bd5196c28 100644 --- a/Guide/src/reference/devices/firmware/linux_direct.md +++ b/Guide/src/reference/devices/firmware/linux_direct.md @@ -27,7 +27,9 @@ uncompressed ELF kernel, but selects an ACPI-free platform mode. OpenVMM places an Intel MP 1.4 floating pointer at `0x0`, its configuration table at `0x400`, the bootstrap GDT at `0x1000`, `boot_params` at `0x2000`, and the command line in the 64-KiB region starting at `0x20000`. The MP table describes contiguous -APIC IDs, the ISA bus, the IOAPIC, and legacy IRQ routing. No RSDP, ACPI table, +APIC IDs, the ISA bus, the IOAPIC, and the routing of each of the IOAPIC's 24 +inputs: the ISA IRQs, and inputs 16 through 23, which the PIC does not see, as +active-high, edge-triggered interrupts of their own numbers. No RSDP, ACPI table, SMBIOS anchor, or SMBIOS structure table is imported. On AArch64, pass the uncompressed `Image` file (not `Image.gz`). diff --git a/Guide/src/reference/devices/virtio/virtio-fs.md b/Guide/src/reference/devices/virtio/virtio-fs.md index 95113e1a992..755e15de214 100644 --- a/Guide/src/reference/devices/virtio/virtio-fs.md +++ b/Guide/src/reference/devices/virtio/virtio-fs.md @@ -45,7 +45,7 @@ fixes the remaining guest-visible configuration: | Stable ID | `fs:microvm0` | | Tag | `microvm` | | Transport | virtio-mmio at `0xd0001000` | -| Interrupt | IRQ 6 | +| Interrupt | IRQ 17, an IOAPIC input past the ISA IRQs | | Queues | One high-priority and one request queue | | Features | Indirect descriptors, event index, version 1 | | DAX window | None | diff --git a/Guide/src/reference/openvmm/management/cli.md b/Guide/src/reference/openvmm/management/cli.md index c995220c21e..8da4addf608 100644 --- a/Guide/src/reference/openvmm/management/cli.md +++ b/Guide/src/reference/openvmm/management/cli.md @@ -52,17 +52,20 @@ describes the source definitions. rejects a longer command line, or one with an embedded NUL, rather than let the guest truncate it. - One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 6 and remains + One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 17 and remains dormant when neither `--mount` nor `--mount-aggregate` is given; either one binds HostFs to it; one optional `--virtio-console ` is exposed at MMIO `0xd0002000`, - IRQ 7 as the boot/log console (`hvc1`); and `--microvm-sandbox-block` - exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 4 and the fixed - `scratch` slot at MMIO `0xd0006000`, IRQ 11. The windows between them and - IRQs 9 and 12 stay unassigned. The profile also reserves MMIO `0xd0007000`, - IRQ 3 for the dedicated control console (`hvc2`), selected by the host-owned - `nvx_control_tty=hvc2` token. The reserved slot and resource identity do not - expose a live control endpoint. Both consoles + IRQ 18 as the boot/log console (`hvc1`); and `--microvm-sandbox-block` + exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 19 and the fixed + `scratch` slot at MMIO `0xd0006000`, IRQ 22. The windows between them and + IRQs 20 and 21 stay unassigned. The profile also reserves MMIO `0xd0007000`, + IRQ 23 for the dedicated control console (`hvc2`), selected by the host-owned + `nvx_control_tty=hvc2` token. Each virtio-mmio window, `0xd0000000 + n * + 0x1000`, interrupts on its own IOAPIC input, `16 + n`, with active-high edges + on every backend; the PIC sees none of these inputs, and the ISA IRQs carry + only the PIT (IRQ 0) and the RTC (IRQ 8). The reserved slot and resource + identity do not expose a live control endpoint. Both consoles remain virtio-console devices from the guest's perspective. Ordinary `--virtio-blk` is rejected. All use split rings, and each offers an exact feature mask of virtio 1.0 features: the NIC its MAC address; the block @@ -71,14 +74,15 @@ describes the source definitions. notification suppression. No device offers `VIRTIO_F_ACCESS_PLATFORM`, since the microVM has no IOMMU, and no console offers its size. Firmware, ACPI, SMBIOS, PCI, VMBus, UARTs, graphics, isolation, nested virtualization, and - other devices are rejected. Linux discovers contiguous APIC IDs and the - IOAPIC from Intel MP 1.4 tables at `0x0` and `0x400`; `boot_params` is at + other devices are rejected. Linux discovers contiguous APIC IDs, the IOAPIC, + and the routing of its 24 inputs from Intel MP 1.4 tables at `0x0` and + `0x400`; `boot_params` is at `0x2000`, the command line starts at `0x20000`, and no ACPI or SMBIOS data is exposed. Host-driven save/restore, pulse-save/restore, and worker restart remain unavailable. `microvm` may also expose a dedicated control virtio-console at MMIO - `0xd0007000`, IRQ 3. It requires the boot virtio-console, preserves + `0xd0007000`, IRQ 23. It requires the boot virtio-console, preserves `console=hvc1`, and publishes `nvx_control_tty=hvc2`. The profile fixes boot-before-control discovery order and rejects user overrides that could change it. @@ -102,11 +106,13 @@ describes the source definitions. as `microvm-v2`; the `microvm-v2` selector and the former ABI-v1 behavior are removed. Snapshots record microVM ABI value 3 and boot layout value 2. ABI value 3 offers no device `VIRTIO_F_ACCESS_PLATFORM` and no console - `VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, so restore refuses a + `VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, and gives each virtio-mmio + slot its own IOAPIC input past the ISA IRQs on every backend, where value 2 + used ISA IRQs and put the NIC on another one on WHP, so restore refuses a snapshot of value 2, which must be recaptured. ``` * `--net `: With `--machine microvm`, attach one virtio-net NIC - at MMIO `0xd0000000`. KVM and MSHV use IRQ 10; WHP uses IRQ 5. Prefixes + at MMIO `0xd0000000`, IRQ 16, on every backend. Prefixes `/1` through `/30` are accepted. The first usable subnet address becomes the gateway; network, broadcast, and gateway addresses cannot be assigned to the guest. Guest and gateway MAC addresses are derived as @@ -266,7 +272,7 @@ describes the source definitions. protocol does not retain pre-capture endpoint completions; restored guest software must establish new host-side flows. * `--mount `: With `--machine microvm`, attach - one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 6, with tag `microvm`. + one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 17, with tag `microvm`. `--mount` may appear once. The default mode is read-only; `rw` must be explicit. The guest target ends at the first comma, and the mode follows the last comma, so a host path may contain commas when the mode is given. @@ -1006,7 +1012,7 @@ Serial devices can be configured to appear as different devices inside the guest behaves normally. * `--virtio-console `: Expose a virtio console device. It normally appears as `/dev/hvc0`. Under `--machine microvm`, it occupies fixed MMIO - `0xd0002000`, IRQ 7, and is selected as `/dev/hvc1`; the raw portb path + `0xd0002000`, IRQ 18, and is selected as `/dev/hvc1`; the raw portb path remains available as `hvc0` for early output and recovery. A microVM accepts these explicit attachment policies: @@ -1034,7 +1040,7 @@ Serial devices can be configured to appear as different devices inside the guest * `--microvm-control-console `: With `--machine microvm`, expose a second independent single-port virtio console at fixed MMIO `0xd0007000`, IRQ - 3. `--virtio-console` is required on a fresh boot and remains the only kernel + 23. `--virtio-console` is required on a fresh boot and remains the only kernel console. The control device normally appears as the profile-owned `nvx_control_tty=hvc2`. diff --git a/Guide/src/reference/openvmm/management/grpc.md b/Guide/src/reference/openvmm/management/grpc.md index caeb7d9bb5d..71b0c058761 100644 --- a/Guide/src/reference/openvmm/management/grpc.md +++ b/Guide/src/reference/openvmm/management/grpc.md @@ -219,7 +219,7 @@ On cold boot, `DevicesConfig.virtio_consoles` may hold one entry, which configures a microVM Unix socket or named-pipe endpoint. Listener mode recreates the path on restore. Client mode is required and uses a five-second connection timeout -before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 7, and selects +before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 18, and selects `hvc1`; its canonical path and policy become the stable restore attachment. `DevicesConfig.virtiofs_config` may bind one HostFs attachment to the fixed @@ -232,7 +232,7 @@ target, and access mode, and the live root must retain its saved identity; the snapshot's denied, allowed, and writable paths apply again. A snapshot captured with the slot dormant may omit the attachment or supply a new one; after `ResumeVM`, the guest explicitly -mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 6, one +mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 17, one request queue, and no DAX window. The management API binds only a single directory: it cannot attach an aggregate of several host directories, as the command line's `--mount-aggregate` does, and it cannot restore a snapshot of @@ -264,7 +264,8 @@ currently blockless; role-bearing sandbox blocks remain CLI-only. Snapshots record microVM ABI value 3 and boot layout value 2, which identifies the Linux-direct MP-table layout. Snapshots of ABI values 1 and 2 are unsupported: value 2 offered `VIRTIO_F_ACCESS_PLATFORM` on every device but the NIC, and the -console size on the consoles. +console size on the consoles, and put the virtio devices on ISA IRQs, where +value 3 gives each one its own IOAPIC input past them. `VMConfig.cpu_profile` selects the microVM's CPU profile as [`--cpu-profile`](cli.md) does, with the same values and checks: `auto`, the diff --git a/Guide/src/user_guide/openvmm/run.md b/Guide/src/user_guide/openvmm/run.md index 0da0ac7efde..84f8d99b9a6 100644 --- a/Guide/src/user_guide/openvmm/run.md +++ b/Guide/src/user_guide/openvmm/run.md @@ -44,11 +44,12 @@ locations: | Role | Access | MMIO address | IRQ | | --- | --- | ---: | ---: | -| `distro` | read-only | `0xd0003000` | 4 | -| `scratch` | writable | `0xd0006000` | 11 | +| `distro` | read-only | `0xd0003000` | 19 | +| `scratch` | writable | `0xd0006000` | 22 | -The windows `0xd0004000` and `0xd0005000` between them, and IRQs 9 and 12, -stay unassigned, so that no slot moves. +The windows `0xd0004000` and `0xd0005000` between them, and their IRQs 20 and +21, stay unassigned, so that no slot moves. Each IRQ is an IOAPIC input past +the ISA IRQs, which the PIC does not see. Use `--microvm-sandbox-block ROLE:DISK`, in the order shown. The `distro` role requires the normal disk `,ro` option and a non-empty topology must end @@ -94,8 +95,10 @@ Custom socket, SMT, APIC, x2APIC, and NUMA options are rejected. The microVM uses fixed virtio device slots and sandbox block roles. Its persisted ABI is value 3 and its boot layout value 2. Each fixed virtio device offers an exact feature mask, which never includes `VIRTIO_F_ACCESS_PLATFORM`, -since the microVM has no IOMMU, nor packed virtqueues. The -MP floating pointer begins at `0x0`, the MP configuration table at `0x400`, +since the microVM has no IOMMU, nor packed virtqueues, and interrupts on an +IOAPIC input of its own past the ISA IRQs, the same on every backend. The +MP floating pointer begins at `0x0`, the MP configuration table, which routes +all 24 IOAPIC inputs, at `0x400`, the boot GDT at `0x1000`, and the Linux zero page at `0x2000`. No ACPI MADT or SMBIOS data is exposed. diff --git a/Guide/src/user_guide/openvmm/snapshots.md b/Guide/src/user_guide/openvmm/snapshots.md index 0b2b07665bf..835b5c31afc 100644 --- a/Guide/src/user_guide/openvmm/snapshots.md +++ b/Guide/src/user_guide/openvmm/snapshots.md @@ -606,7 +606,7 @@ immediately with a clear error if any active device does not support it. `--processors`. MicroVM restore reads them authoritatively from the manifest and rejects overrides. Persisted microVM ABI value 3 and boot-layout value 2 are supported; a snapshot of ABI value 2, whose devices offered - `VIRTIO_F_ACCESS_PLATFORM`, must be recaptured. + `VIRTIO_F_ACCESS_PLATFORM` and used ISA IRQs, must be recaptured. - MicroVM restore requires the backend, CPU generation, and CPU profile of the capture, and an OpenVMM build that computes the same effective CPUID. A host profile's snapshot, including one of a cold boot whose `auto` fell diff --git a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs index 11c680f22bd..7cee87dd756 100644 --- a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs +++ b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs @@ -187,6 +187,7 @@ pub(super) fn load_linux_x86_mptable( &vm.gm, &apic_ids, &openvmm_defs::microvm::MICROVM_LEVEL_TRIGGERED_IRQS, + openvmm_defs::microvm::MICROVM_IOAPIC_INPUTS, &reserved_memory_ranges, )?) } @@ -484,7 +485,7 @@ fn virtio_mmio_config( let (start, irq, features) = match id { "virtio-net" => ( openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, - openvmm_defs::microvm::microvm_virtio_net_irq(None)?, + openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ, openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES, ), // The configuration orders virtio-fs devices by slot. @@ -805,8 +806,7 @@ pub(super) fn x86_topology_builder( } } -// The microVM runs on Linux and Windows hosts alone, where its NIC has an -// interrupt. +// The microVM runs on Linux and Windows hosts alone. #[cfg(all(test, any(target_os = "linux", windows)))] mod tests { use super::virtio_mmio_config; @@ -916,24 +916,25 @@ mod tests { ]; let mut sandbox_block_index = 0; let mut filesystem_index = 0; - // Each device in the order of the slots of its kind, with its window - // and the features that the guest sees: never access-platform, which - // the transport offers every device, nor the console size, which the - // virtio-console device offers. - let devices: [(&str, u64, &[u32]); 6] = [ - ("virtio-net", 0xd000_0000, &[5, 32]), - ("virtiofs", 0xd000_1000, &[28, 29, 32]), - ("virtio-console", 0xd000_2000, &[28, 29, 32]), - ("virtio-blk", 0xd000_3000, &[2, 5, 6, 9, 10, 28, 29, 32]), - ("virtio-blk", 0xd000_6000, &[2, 6, 9, 10, 28, 29, 32]), + // Each device in the order of the slots of its kind, with its window, + // its IOAPIC input, and the features that the guest sees: never + // access-platform, which the transport offers every device, nor the + // console size, which the virtio-console device offers. + let devices: [(&str, u64, u32, &[u32]); 6] = [ + ("virtio-net", 0xd000_0000, 16, &[5, 32]), + ("virtiofs", 0xd000_1000, 17, &[28, 29, 32]), + ("virtio-console", 0xd000_2000, 18, &[28, 29, 32]), + ("virtio-blk", 0xd000_3000, 19, &[2, 5, 6, 9, 10, 28, 29, 32]), + ("virtio-blk", 0xd000_6000, 22, &[2, 6, 9, 10, 28, 29, 32]), ( openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID, 0xd000_7000, + 23, &[28, 29, 32], ), ]; - for (id, mmio_base, bits) in devices { - let (start, len, _irq, features, interrupt_mode) = virtio_mmio_config( + for (id, mmio_base, input, bits) in devices { + let (start, len, irq, features, interrupt_mode) = virtio_mmio_config( id, &blocks, &mut sandbox_block_index, @@ -942,6 +943,7 @@ mod tests { ) .unwrap(); assert_eq!(start, mmio_base, "{id}"); + assert_eq!(irq, input, "{id}"); let mut device = VirtioMmioDevice::new_with_disabled_features_and_interrupt_mode( Box::new(EveryFeatureDevice), &driver_source.simple(), diff --git a/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs b/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs index d850a721792..21497306908 100644 --- a/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs +++ b/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs @@ -22,6 +22,7 @@ pub fn load_linux_x86_mptable( gm: &GuestMemory, apic_ids: &[u32], level_triggered_irqs: &[u32], + ioapic_inputs: u8, reserved_memory_ranges: &[MemoryRange], ) -> Result, LinuxError> { if !matches!(cfg.isolation, KernelIsolationConfig::None) { @@ -54,6 +55,7 @@ pub fn load_linux_x86_mptable( &loader::mptable::MpTableConfig { apic_ids, level_triggered_irqs, + ioapic_inputs, }, reserved_memory_ranges, ) diff --git a/openvmm/openvmm_defs/src/microvm.rs b/openvmm/openvmm_defs/src/microvm.rs index 9a7980ba685..e18afed1266 100644 --- a/openvmm/openvmm_defs/src/microvm.rs +++ b/openvmm/openvmm_defs/src/microvm.rs @@ -25,8 +25,12 @@ use vmotherboard::options::BaseChipsetManifest; /// /// Version 3 offers no device `VIRTIO_F_ACCESS_PLATFORM`, since the microVM /// has no IOMMU, and no console `VIRTIO_CONSOLE_F_SIZE`, both of which -/// version 2 offered. A snapshot of version 2 records those feature masks, so -/// restore refuses it by its version, before it compares the devices. +/// version 2 offered. It also routes the IOAPIC's inputs past the 16 ISA IRQs +/// in the MP table and gives each virtio-mmio slot one of them, the same on +/// every backend, where version 2 put each slot on an ISA IRQ and the NIC on +/// another one on WHP. A snapshot of version 2 records those feature masks +/// and interrupts, so restore refuses it by its version, before it compares +/// the devices. pub const MICROVM_ABI_VERSION_3: u32 = 3; /// Linux memory-block granularity used for microVM restore-time expansion. pub const MICROVM_MEMORY_BLOCK_SIZE_BYTES: u64 = 128 * 1024 * 1024; @@ -60,20 +64,23 @@ pub const MICROVM_VIRTIO_CONSOLE_MMIO_BASE: u64 = 0xd000_2000; pub const MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE: u64 = 0xd000_7000; /// Fixed microVM virtio transport window length. pub const MICROVM_VIRTIO_MMIO_LEN: u64 = 0x1000; -/// Fixed distro virtio-blk interrupt. -pub const MICROVM_VIRTIO_BLK_IRQ: u32 = 4; -/// Fixed virtio-blk interrupt for the writable scratch layer. -pub const MICROVM_VIRTIO_SCRATCH_BLK_IRQ: u32 = 11; -/// Fixed microVM virtio-console interrupt. -pub const MICROVM_VIRTIO_CONSOLE_IRQ: u32 = 7; -/// Fixed microVM control virtio-console interrupt. -pub const MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ: u32 = 3; -/// Fixed interrupt of the microVM virtio-fs slot. -pub const MICROVM_VIRTIO_FS_IRQ: u32 = 6; -/// Fixed microVM virtio-net interrupt on KVM. -pub const MICROVM_VIRTIO_NET_KVM_IRQ: u32 = 10; -/// Fixed microVM virtio-net interrupt on WHP. -pub const MICROVM_VIRTIO_NET_WHP_IRQ: u32 = 5; +/// The IOAPIC input of the first virtio-mmio window, the first input past the +/// 16 ISA IRQs, which the PIC does not see: the window at +/// `0xd0000000 + n * 0x1000` interrupts on input `16 + n`, on every backend. +pub const MICROVM_VIRTIO_IRQ_BASE: u32 = 16; +/// Fixed distro virtio-blk interrupt: the input of window 3. +pub const MICROVM_VIRTIO_BLK_IRQ: u32 = 19; +/// Fixed virtio-blk interrupt for the writable scratch layer: the input of +/// window 6. +pub const MICROVM_VIRTIO_SCRATCH_BLK_IRQ: u32 = 22; +/// Fixed microVM virtio-console interrupt: the input of window 2. +pub const MICROVM_VIRTIO_CONSOLE_IRQ: u32 = 18; +/// Fixed microVM control virtio-console interrupt: the input of window 7. +pub const MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ: u32 = 23; +/// Fixed interrupt of the microVM virtio-fs slot: the input of window 1. +pub const MICROVM_VIRTIO_FS_IRQ: u32 = 17; +/// Fixed microVM virtio-net interrupt: the input of window 0. +pub const MICROVM_VIRTIO_NET_IRQ: u32 = 16; // The virtio 1.2 feature bits that the exact microVM feature masks name. The // masks leave out `VIRTIO_F_ACCESS_PLATFORM` (bit 33), which the virtio // transport offers every device, since the microVM has no IOMMU to translate a @@ -141,9 +148,9 @@ pub const MICROVM_HOST_LOOPBACK_INGRESS_COMMAND_LINE_TOKEN: &str = "virtnet_ingr /// /// Each slot keeps its place whether or not the slots before it are present. /// The windows at `0xd0004000` and `0xd0005000`, which held the retired -/// `runtime` and `custom` sandbox layers, stay unassigned, as do IRQs 12 and 9 -/// and the shared-status words at offsets `0x10` and `0x14`: the slots after -/// them kept their places. +/// `runtime` and `custom` sandbox layers, stay unassigned, as do their IOAPIC +/// inputs 20 and 21 and the shared-status words at offsets `0x10` and `0x14`: +/// the slots after them kept their places. pub const MICROVM_VIRTIO_MMIO_BASES: [u64; 6] = [ MICROVM_VIRTIO_NET_MMIO_BASE, MICROVM_VIRTIO_FS_MMIO_BASE, @@ -220,8 +227,27 @@ pub const fn microvm_virtio_status_gpa(mmio_base: u64) -> Option { }; Some(MICROVM_SHARED_STATUS_PAGE_GPA + offset) } + +/// Returns the edge-triggered interrupt, the IOAPIC input, of a fixed +/// virtio-mmio slot. +pub const fn microvm_virtio_irq(mmio_base: u64) -> Option { + Some(match mmio_base { + MICROVM_VIRTIO_NET_MMIO_BASE => MICROVM_VIRTIO_NET_IRQ, + MICROVM_VIRTIO_FS_MMIO_BASE => MICROVM_VIRTIO_FS_IRQ, + MICROVM_VIRTIO_CONSOLE_MMIO_BASE => MICROVM_VIRTIO_CONSOLE_IRQ, + MICROVM_VIRTIO_BLK_MMIO_BASE => MICROVM_VIRTIO_BLK_IRQ, + MICROVM_VIRTIO_SCRATCH_BLK_MMIO_BASE => MICROVM_VIRTIO_SCRATCH_BLK_IRQ, + MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE => MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ, + _ => return None, + }) +} /// The microVM publishes no level-triggered virtio IRQs in its MP table. pub const MICROVM_LEVEL_TRIGGERED_IRQS: [u32; 0] = []; +/// The inputs of the microVM's IOAPIC, the generic IOAPIC of its chipset, +/// every one of which its MP table routes: the first 16 carry the ISA IRQs, +/// which the PIC also sees, and the IOAPIC alone sees inputs 16 through 23, +/// which the virtio-mmio slots use. +pub const MICROVM_IOAPIC_INPUTS: u8 = chipset_resources::ioapic::IOAPIC_NUM_ENTRIES; /// The stable role of a microVM sandbox block device. #[derive(MeshPayload, Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] @@ -1143,20 +1169,6 @@ impl std::str::FromStr for MicrovmNetworkConfig { } } -/// Returns the pinned virtio-net IRQ for the selected microVM backend. -pub fn microvm_virtio_net_irq(hypervisor_id: Option<&str>) -> anyhow::Result { - match hypervisor_id { - Some("kvm" | "mshv") => Ok(MICROVM_VIRTIO_NET_KVM_IRQ), - Some("whp") => Ok(MICROVM_VIRTIO_NET_WHP_IRQ), - Some(other) => anyhow::bail!("microVM virtio-net does not support hypervisor '{other}'"), - None if cfg!(target_os = "linux") => Ok(MICROVM_VIRTIO_NET_KVM_IRQ), - None if cfg!(windows) => Ok(MICROVM_VIRTIO_NET_WHP_IRQ), - None => { - anyhow::bail!("microVM virtio-net requires an explicit KVM, MSHV, or WHP hypervisor") - } - } -} - fn validate_microvm_virtio_reservations() -> anyhow::Result<()> { let bases = &MICROVM_VIRTIO_MMIO_BASES; for (index, base) in bases.iter().copied().enumerate() { @@ -1189,6 +1201,34 @@ fn validate_microvm_virtio_reservations() -> anyhow::Result<()> { ); previous_status_gpa = Some(status_gpa); } + validate_microvm_virtio_irqs( + bases + .iter() + .map(|&mmio_base| (mmio_base, microvm_virtio_irq(mmio_base))), + ) +} + +/// Checks that each slot of `slots`, a window base and its interrupt in +/// window order, interrupts on an IOAPIC input of its own past the ISA IRQs, +/// which neither the PIC nor a chipset device shares. +fn validate_microvm_virtio_irqs( + slots: impl IntoIterator)>, +) -> anyhow::Result<()> { + let mut previous_irq = None; + for (mmio_base, irq) in slots { + let irq = irq.ok_or_else(|| { + anyhow::anyhow!("microVM virtio MMIO slot {mmio_base:#x} has no interrupt") + })?; + anyhow::ensure!( + (MICROVM_VIRTIO_IRQ_BASE..u32::from(MICROVM_IOAPIC_INPUTS)).contains(&irq), + "microVM interrupt {irq} of slot {mmio_base:#x} is not an IOAPIC input past the ISA IRQs" + ); + anyhow::ensure!( + previous_irq.is_none_or(|previous| previous < irq), + "microVM virtio interrupts are duplicated or out of order" + ); + previous_irq = Some(irq); + } Ok(()) } @@ -1268,14 +1308,13 @@ pub fn microvm_filesystem_slot_count(first_slot: bool, attached: usize) -> anyho /// Appends sandbox virtio devices in fixed-address order. /// /// `filesystem_slot` reserves the virtio-fs slot, and `filesystems`, at most -/// one, occupies it. `network` carries the NIC's identity, its interrupt, the -/// gateway address, if any, that the guest names as its DNS server, and the -/// ingress, if any, for which the guest announces its listeners. +/// one, occupies it. `network` carries the NIC's identity, the gateway +/// address, if any, that the guest names as its DNS server, and the ingress, +/// if any, for which the guest announces its listeners. pub fn append_microvm_virtio_discovery( cmdline: &mut String, network: Option<( &MicrovmNetworkConfig, - u32, Option, Option, )>, @@ -1300,14 +1339,10 @@ pub fn append_microvm_virtio_discovery( validate_microvm_filesystems(filesystems)?; use std::fmt::Write as _; - if let Some((_, irq, _, _)) = network { - anyhow::ensure!( - matches!(irq, MICROVM_VIRTIO_NET_KVM_IRQ | MICROVM_VIRTIO_NET_WHP_IRQ), - "microVM virtio-net IRQ {irq} is not part of the fixed machine contract" - ); + if network.is_some() { write!( cmdline, - " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{irq}" + " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{MICROVM_VIRTIO_NET_IRQ}" )?; } if filesystem_slots > 0 { @@ -1341,7 +1376,7 @@ pub fn append_microvm_virtio_discovery( " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE:#x}:{MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ} {MICROVM_CONTROL_TTY_COMMAND_LINE}" )?; } - if let Some((network, _, gateway_dns, ingress)) = network { + if let Some((network, gateway_dns, ingress)) = network { write!( cmdline, " {}", @@ -1443,10 +1478,7 @@ fn microvm_command_line_ingress( } } -fn validate_microvm_command_line( - config: &Config, - hypervisor_id: Option<&str>, -) -> anyhow::Result<()> { +fn validate_microvm_command_line(config: &Config) -> anyhow::Result<()> { let MachineProfile::Microvm = config.machine_profile else { unreachable!("microVM command-line validation requires the microVM profile"); }; @@ -1563,9 +1595,8 @@ fn validate_microvm_command_line( let ingress = microvm_command_line_ingress(config.microvm.network.as_ref(), &tokens)?; let mut expected_discovery = Vec::new(); if has_network { - let irq = microvm_virtio_net_irq(hypervisor_id)?; expected_discovery.push(format!( - "virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{irq}" + "virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{MICROVM_VIRTIO_NET_IRQ}" )); } if filesystem_count > 0 { @@ -1612,15 +1643,27 @@ fn validate_microvm_command_line( ); } } - if !expected_discovery.is_empty() { - anyhow::ensure!( - tokens.ends_with( - &expected_discovery - .iter() - .map(String::as_str) - .collect::>() - ), - "microVM virtio discovery tokens are not in fixed-address order" + check_microvm_discovery_tokens(&tokens, &expected_discovery) +} + +/// Checks that the command line's `tokens` end with `expected`, the discovery +/// tokens of the machine's devices in fixed-address order, and names the +/// first token that differs, whether it is out of order or places its device +/// elsewhere, such as on the interrupt of another contract. +fn check_microvm_discovery_tokens(tokens: &[&str], expected: &[String]) -> anyhow::Result<()> { + let tail = &tokens[tokens.len().saturating_sub(expected.len())..]; + anyhow::ensure!( + tail.len() == expected.len(), + "microVM command line lacks virtio discovery tokens" + ); + if let Some((found, expected)) = tail + .iter() + .zip(expected) + .find(|(found, expected)| **found != expected.as_str()) + { + anyhow::bail!( + "microVM virtio discovery tokens are not this machine's in fixed-address order: \ + found `{found}` where `{expected}` belongs" ); } Ok(()) @@ -1937,7 +1980,7 @@ pub fn validate_machine_config(config: &Config, hypervisor_id: Option<&str>) -> }; validate_microvm_virtio_reservations()?; - validate_microvm_command_line(config, hypervisor_id)?; + validate_microvm_command_line(config)?; validate_machine_load_mode(config.machine_profile, &config.load_mode)?; anyhow::ensure!( config.microvm.time_abi.is_some(), @@ -2384,7 +2427,7 @@ mod tests { let discover = |cmdline: &mut String| { append_microvm_virtio_discovery( cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, None, None)), + Some((&network, None, None)), false, &[], false, @@ -2563,7 +2606,7 @@ mod tests { let mut cmdline = MICROVM_BASE_COMMAND_LINE.to_owned(); append_microvm_virtio_discovery( &mut cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, dns, Some(ingress))), + Some((&network, dns, Some(ingress))), false, &[], false, @@ -2633,8 +2676,8 @@ mod tests { cmdline, format!( "{MICROVM_BASE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0003000:4 \ - virtio_mmio.device=0x1000@0xd0006000:11" + virtio_mmio.device=0x1000@0xd0003000:19 \ + virtio_mmio.device=0x1000@0xd0006000:22" ) ); } @@ -2647,33 +2690,139 @@ mod tests { cmdline, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {MICROVM_CONTROL_TTY_COMMAND_LINE}" ) ); } #[test] - fn microvm_virtio_irqs_avoid_unused_lines_and_are_edge_triggered() { + fn microvm_virtio_slots_have_their_own_ioapic_inputs() { let irqs = [ - MICROVM_VIRTIO_NET_KVM_IRQ, - MICROVM_VIRTIO_NET_WHP_IRQ, + MICROVM_VIRTIO_NET_IRQ, MICROVM_VIRTIO_FS_IRQ, MICROVM_VIRTIO_CONSOLE_IRQ, MicrovmSandboxBlockRole::Distro.irq(), MicrovmSandboxBlockRole::Scratch.irq(), MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ, ]; - assert_eq!(irqs, [10, 5, 6, 7, 4, 11, 3]); - // The RTC owns IRQ 8. IRQs 12, 9, and 13 of the retired runtime and - // custom layers and second virtio-fs slot stay unused. - for unused in [8, 9, 12, 13] { - assert!(!irqs.contains(&unused), "IRQ {unused} is in use"); + assert_eq!(irqs, [16, 17, 18, 19, 22, 23]); + assert_eq!( + MICROVM_VIRTIO_MMIO_BASES.map(|mmio_base| microvm_virtio_irq(mmio_base).unwrap()), + irqs + ); + // The window at 0xd0000000 + n * 0x1000 interrupts on input 16 + n, + // past the ISA IRQs, which the PIC also sees and the PIT and RTC use. + for mmio_base in MICROVM_VIRTIO_MMIO_BASES { + let window = (mmio_base - MICROVM_VIRTIO_NET_MMIO_BASE) / MICROVM_VIRTIO_MMIO_LEN; + assert_eq!( + microvm_virtio_irq(mmio_base), + Some(MICROVM_VIRTIO_IRQ_BASE + window as u32) + ); + } + for (role, irq) in [ + (MicrovmSandboxBlockRole::Distro, 19), + (MicrovmSandboxBlockRole::Scratch, 22), + ] { + assert_eq!(microvm_virtio_irq(role.mmio_base()), Some(irq)); + } + // The windows of the retired runtime and custom layers keep inputs 20 + // and 21 free, and no window past them has an input. + for mmio_base in [0xd000_4000, 0xd000_5000, 0xd000_8000] { + assert_eq!(microvm_virtio_irq(mmio_base), None); } + assert_eq!( + u32::from(MICROVM_IOAPIC_INPUTS), + MICROVM_VIRTIO_IRQ_BASE + 8 + ); assert!(MICROVM_LEVEL_TRIGGERED_IRQS.is_empty()); } + #[test] + fn microvm_virtio_irq_validation_rejects_collisions() { + validate_microvm_virtio_reservations().unwrap(); + let slot = |window: u64, irq| { + ( + MICROVM_VIRTIO_NET_MMIO_BASE + window * MICROVM_VIRTIO_MMIO_LEN, + Some(irq), + ) + }; + validate_microvm_virtio_irqs([slot(0, 16), slot(1, 17), slot(7, 23)]).unwrap(); + for slots in [ + // Two slots on one input. + vec![slot(0, 16), slot(1, 16)], + // ISA IRQs, which the PIC also sees: the RTC's and a free one. + vec![slot(0, 8)], + vec![slot(0, 15)], + // Past the IOAPIC's last input. + vec![slot(0, 24)], + // No interrupt. + vec![(MICROVM_VIRTIO_NET_MMIO_BASE, None)], + ] { + assert!( + validate_microvm_virtio_irqs(slots.clone()).is_err(), + "{slots:?}" + ); + } + } + + #[test] + fn microvm_discovery_token_mismatch_names_the_token() { + let expected = [ + format!("virtio_mmio.device=0x1000@0xd0000000:{MICROVM_VIRTIO_NET_IRQ}"), + format!("virtio_mmio.device=0x1000@0xd0001000:{MICROVM_VIRTIO_FS_IRQ}"), + ]; + check_microvm_discovery_tokens( + &["quiet", expected[0].as_str(), expected[1].as_str()], + &expected, + ) + .unwrap(); + check_microvm_discovery_tokens(&["quiet"], &[]).unwrap(); + // A token that places its device on another interrupt, such as the + // NIC's ISA IRQ before each slot had an IOAPIC input of its own, is + // named, not only reported out of order. + let error = check_microvm_discovery_tokens( + &[ + "quiet", + "virtio_mmio.device=0x1000@0xd0000000:10", + expected[1].as_str(), + ], + &expected, + ) + .unwrap_err() + .to_string(); + assert!( + error.ends_with( + "found `virtio_mmio.device=0x1000@0xd0000000:10` where \ + `virtio_mmio.device=0x1000@0xd0000000:16` belongs" + ), + "{error}" + ); + let error = check_microvm_discovery_tokens( + &[expected[1].as_str(), expected[0].as_str()], + &expected, + ) + .unwrap_err() + .to_string(); + assert!( + error.ends_with(&format!( + "found `{}` where `{}` belongs", + expected[1], expected[0] + )), + "{error}" + ); + assert!(check_microvm_discovery_tokens(&[expected[1].as_str()], &expected).is_err()); + } + + #[test] + fn microvm_ioapic_inputs_have_backend_routes() { + // The MP table routes every input of the IOAPIC, and every backend + // keeps a route for each: KVM claims a GSI, and MSHV and WHP a route. + assert_eq!(MICROVM_IOAPIC_INPUTS, 24); + assert_eq!(usize::from(MICROVM_IOAPIC_INPUTS), virt::irqcon::IRQ_LINES); + } + #[test] fn microvm_shared_status_slots_are_stable() { let slots = [ @@ -2754,11 +2903,11 @@ mod tests { slot.mmio_base, slot.irq )), - [("fs:microvm0", "microvm", 0xd000_1000, 6)] + [("fs:microvm0", "microvm", 0xd000_1000, 17)] ); assert_eq!( MICROVM_FILESYSTEM_SLOTS[0].discovery_token(), - "virtio_mmio.device=0x1000@0xd0001000:6" + "virtio_mmio.device=0x1000@0xd0001000:17" ); validate_microvm_virtio_reservations().unwrap(); assert!(!MICROVM_VIRTIO_MMIO_BASES.contains(&0xd000_8000)); @@ -3207,7 +3356,7 @@ mod tests { let mut cmdline = MICROVM_CONSOLE_COMMAND_LINE.to_owned(); append_microvm_virtio_discovery( &mut cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, None, None)), + Some((&network, None, None)), true, std::slice::from_ref(&aggregate), true, @@ -3219,10 +3368,10 @@ mod tests { cmdline, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0000000:10 \ - virtio_mmio.device=0x1000@0xd0001000:6 \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0000000:16 \ + virtio_mmio.device=0x1000@0xd0001000:17 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {MICROVM_CONTROL_TTY_COMMAND_LINE} \ virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 \ virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1 \ @@ -3249,7 +3398,7 @@ mod tests { assert_eq!( cmdline, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:6 \ + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:17 \ virtfs_dir=/workspace virtfs_tag=microvm virtfs_mode=rw" ) ); @@ -3335,8 +3484,8 @@ mod tests { let mut cmdline = format!("{MICROVM_BASE_COMMAND_LINE} nvx_overlay_upper=ramfs"); append_microvm_virtio_discovery(&mut cmdline, None, false, &[], false, false, &[distro]) .unwrap(); - assert!(cmdline.contains("virtio_mmio.device=0x1000@0xd0003000:4")); - assert!(!cmdline.contains("virtio_mmio.device=0x1000@0xd0006000:11")); + assert!(cmdline.contains("virtio_mmio.device=0x1000@0xd0003000:19")); + assert!(!cmdline.contains("virtio_mmio.device=0x1000@0xd0006000:22")); let mut cmdline = format!("{MICROVM_BASE_COMMAND_LINE} nvx_overlay_upper"); assert!( diff --git a/openvmm/openvmm_entry/src/cli_args/microvm.rs b/openvmm/openvmm_entry/src/cli_args/microvm.rs index faf40529131..6ca3490d47d 100644 --- a/openvmm/openvmm_entry/src/cli_args/microvm.rs +++ b/openvmm/openvmm_entry/src/cli_args/microvm.rs @@ -2118,7 +2118,7 @@ mod tests { assert_eq!( with_devices, format!( - "{MICROVM_CONSOLE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0002000:7 virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11" + "{MICROVM_CONSOLE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0002000:18 virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22" ) ); let mut with_control_console = build_microvm_control_command_line(&[], true).unwrap(); @@ -2136,8 +2136,8 @@ mod tests { with_control_console, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {}", openvmm_defs::microvm::MICROVM_CONTROL_TTY_COMMAND_LINE ) @@ -2147,12 +2147,7 @@ mod tests { let mut with_network = build_microvm_command_line(&[], false).unwrap(); append_microvm_virtio_discovery( &mut with_network, - Some(( - &network, - openvmm_defs::microvm::MICROVM_VIRTIO_NET_KVM_IRQ, - None, - None, - )), + Some((&network, None, None)), false, &[], false, @@ -2163,20 +2158,15 @@ mod tests { assert_eq!( with_network, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0000000:10 virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0000000:16 virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" ) ); // The guest names either gateway as its DNS server. for (server, dns) in [("10.0.0.1", "10.0.0.1"), ("fd00::a00:1", "fd00::a00:1")] { - let mut with_whp_network = build_microvm_command_line(&[], false).unwrap(); + let mut with_dns = build_microvm_command_line(&[], false).unwrap(); append_microvm_virtio_discovery( - &mut with_whp_network, - Some(( - &network, - openvmm_defs::microvm::MICROVM_VIRTIO_NET_WHP_IRQ, - Some(server.parse().unwrap()), - None, - )), + &mut with_dns, + Some((&network, Some(server.parse().unwrap()), None)), false, &[], false, @@ -2185,10 +2175,10 @@ mod tests { ) .unwrap(); assert!( - with_whp_network.ends_with(&format!( + with_dns.ends_with(&format!( "virtnet_dns={dns} virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" )), - "{with_whp_network}" + "{with_dns}" ); } @@ -2211,7 +2201,7 @@ mod tests { assert_eq!( with_filesystem, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:6 virtfs_dir=/mnt/share virtfs_tag=microvm virtfs_mode=ro" + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:17 virtfs_dir=/mnt/share virtfs_tag=microvm virtfs_mode=ro" ) ); @@ -2236,7 +2226,7 @@ mod tests { "nvx-control-tty=hvc9", "driver_async_probe=virtio_console", "driver-async-probe=virtio_console", - "virtio-mmio.device=0x1000@0xd0007000:3", + "virtio-mmio.device=0x1000@0xd0007000:23", ] { assert!(build_microvm_control_command_line(&[reserved.into()], false).is_err()); assert!(build_microvm_command_line(&[reserved.into()], false).is_ok()); diff --git a/openvmm/openvmm_entry/src/microvm/config.rs b/openvmm/openvmm_entry/src/microvm/config.rs index dc3178f6ff8..3a3e4de7809 100644 --- a/openvmm/openvmm_entry/src/microvm/config.rs +++ b/openvmm/openvmm_entry/src/microvm/config.rs @@ -818,12 +818,6 @@ impl<'a> MicrovmConfigBuilder<'a> { let has_control_console = cfg.virtio_devices.iter().any(|(_, device)| { device.id() == openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID }); - let network_irq = cfg - .microvm - .network - .as_ref() - .map(|_| openvmm_defs::microvm::microvm_virtio_net_irq(requested_hypervisor)) - .transpose()?; let ingress = self .network .as_ref() @@ -832,8 +826,7 @@ impl<'a> MicrovmConfigBuilder<'a> { .microvm .network .as_ref() - .zip(network_irq) - .map(|(network, irq)| (network, irq, self.gateway_dns, ingress)); + .map(|network| (network, self.gateway_dns, ingress)); if let Some(snapshot_tier) = opt.microvm.snapshot_tier { anyhow::ensure!( !cmdline diff --git a/openvmm/openvmm_entry/src/microvm/network.rs b/openvmm/openvmm_entry/src/microvm/network.rs index 3106e9527a3..2c66cc24826 100644 --- a/openvmm/openvmm_entry/src/microvm/network.rs +++ b/openvmm/openvmm_entry/src/microvm/network.rs @@ -271,11 +271,10 @@ pub(crate) mod tests { policy: &net_backend_resources::egress::EgressPolicy, ) -> openvmm_helpers::snapshot::microvm::SnapshotMachineContract { let source_hypervisor = if cfg!(windows) { "whp" } else { "kvm" }; - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor)).unwrap(); let mut command_line = build_microvm_command_line(&[], false).unwrap(); openvmm_defs::microvm::append_microvm_virtio_discovery( &mut command_line, - Some((network, irq, policy.gateway_dns_server(), None)), + Some((network, policy.gateway_dns_server(), None)), false, &[], false, diff --git a/openvmm/openvmm_helpers/src/snapshot/microvm.rs b/openvmm/openvmm_helpers/src/snapshot/microvm.rs index 62af75d279d..b189479e825 100644 --- a/openvmm/openvmm_helpers/src/snapshot/microvm.rs +++ b/openvmm/openvmm_helpers/src/snapshot/microvm.rs @@ -1271,11 +1271,11 @@ pub fn microvm_machine_contract( && attachment.reconnect_timeout_ms == 0, "microVM network attachment has an unsupported endpoint policy" ); - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor))?; let discovery = format!( - "virtio_mmio.device={:#x}@{:#x}:{irq}", + "virtio_mmio.device={:#x}@{:#x}:{}", openvmm_defs::microvm::MICROVM_VIRTIO_MMIO_LEN, openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, + openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ, ); let tokens = effective_command_line .split_ascii_whitespace() @@ -1300,7 +1300,7 @@ pub fn microvm_machine_contract( openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, openvmm_defs::microvm::MICROVM_VIRTIO_MMIO_LEN, )], - irq: Some(irq), + irq: Some(openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ), transport: "virtio-mmio".to_owned(), feature_banks: feature_banks(openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES), queue_count: 2, @@ -2702,9 +2702,8 @@ mod tests { mode: net_backend_resources::egress::EgressPolicyMode, ) -> SnapshotMachineContract { let egress_policy = network.bind_egress_policy(mode).unwrap(); - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor)).unwrap(); let command_line = format!( - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0000000:{irq} {}", + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0000000:16 {}", network.command_line_fragment_with_dns(egress_policy.gateway_dns_server()) ); microvm_machine_contract( @@ -2748,7 +2747,7 @@ mod tests { microvm_machine_contract( "whp", MICROVM_BOOT_LAYOUT_VERSION, - "earlycon=xe9 console=hvc1 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0002000:7" + "earlycon=xe9 console=hvc1 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0002000:18" .to_owned(), None, false, @@ -2903,7 +2902,7 @@ mod tests { filesystem: &openvmm_defs::microvm::MicrovmFilesystemConfig, ) -> SnapshotMachineContract { let command_line = format!( - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:6 {}", + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:17 {}", filesystem.command_line_fragment(&openvmm_defs::microvm::MICROVM_FILESYSTEM_SLOTS[0]) ); microvm_machine_contract( @@ -2952,7 +2951,7 @@ mod tests { microvm_machine_contract( "whp", MICROVM_BOOT_LAYOUT_VERSION, - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:6" + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:17" .to_owned(), None, true, @@ -3133,7 +3132,7 @@ mod tests { ); // The contract requires the slot's discovery token once and the // bootstrap tokens exactly. - let discovery = " virtio_mmio.device=0x1000@0xd0001000:6"; + let discovery = " virtio_mmio.device=0x1000@0xd0001000:17"; for invalid in [ command_line.replace(discovery, ""), command_line.replace(discovery, &format!("{discovery}{discovery}")), @@ -3402,7 +3401,7 @@ mod tests { assert_eq!(console.state_unit_name, "virtio-console-3489669120"); assert_eq!(console.ranges[0].start, 0xd000_2000); assert_eq!(console.ranges[0].length, 0x1000); - assert_eq!(console.irq, Some(7)); + assert_eq!(console.irq, Some(18)); assert_eq!(console.transport, "virtio-mmio"); // Indirect descriptors, event index, and virtio version 1: no console // size and no access-platform. @@ -3429,7 +3428,7 @@ mod tests { assert_eq!(control.state_unit_name, "virtio-control-console-3489689600"); assert_eq!(control.ranges[0].start, 0xd000_7000); assert_eq!(control.ranges[0].length, 0x1000); - assert_eq!(control.irq, Some(3)); + assert_eq!(control.irq, Some(23)); assert_eq!(control.transport, "virtio-mmio"); assert_eq!(control.feature_banks, boot.feature_banks); assert_eq!(control.queue_max_sizes, [256, 256]); @@ -3467,15 +3466,15 @@ mod tests { } #[test] - fn generated_microvm_network_contract_has_backend_specific_fixed_abi() { - for (source_hypervisor, irq) in [("kvm", 10), ("mshv", 10), ("whp", 5)] { + fn generated_microvm_network_contract_has_one_fixed_abi_on_every_backend() { + for source_hypervisor in ["kvm", "mshv", "whp"] { let contract = generated_network_contract(source_hypervisor); let network_device = contract.devices.last().unwrap(); assert_eq!(network_device.stable_id, "net:microvm0"); assert_eq!(network_device.state_unit_name, "virtio-net-3489660928"); assert_eq!(network_device.ranges[0].start, 0xd000_0000); assert_eq!(network_device.ranges[0].length, 0x1000); - assert_eq!(network_device.irq, Some(irq)); + assert_eq!(network_device.irq, Some(16)); assert_eq!(network_device.transport, "virtio-mmio"); assert_eq!(network_device.feature_banks, [0x20, 0x1]); assert_eq!(network_device.queue_count, 2); @@ -3783,7 +3782,7 @@ mod tests { assert_eq!(filesystem_device.state_unit_name, "virtiofs-3489665024"); assert_eq!(filesystem_device.ranges[0].start, 0xd000_1000); assert_eq!(filesystem_device.ranges[0].length, 0x1000); - assert_eq!(filesystem_device.irq, Some(6)); + assert_eq!(filesystem_device.irq, Some(17)); assert_eq!(filesystem_device.transport, "virtio-mmio"); assert_eq!(filesystem_device.feature_banks, [0x3000_0000, 0x0000_0001]); assert_eq!(filesystem_device.queue_count, 2); @@ -3828,7 +3827,7 @@ mod tests { assert_eq!(filesystem.stable_id, "fs:microvm0"); assert_eq!(filesystem.state_unit_name, "virtiofs-3489665024"); assert_eq!(filesystem.ranges[0].start, 0xd000_1000); - assert_eq!(filesystem.irq, Some(6)); + assert_eq!(filesystem.irq, Some(17)); } #[test] @@ -4046,14 +4045,17 @@ mod tests { #[test] fn abi_v2_contracts_are_refused_by_their_version() { // Version 2 offered access-platform on every device but the NIC, and - // the console size, so a console of version 2 records other feature - // banks than one of version 3. Restore refuses such a contract by its - // version, before it compares the devices. + // the console size, and put the console on IRQ 7, so a console of + // version 2 records other feature banks and another interrupt than one + // of version 3. Restore refuses such a contract by its version, before + // it compares the devices. const REFUSAL: &str = "snapshot microVM ABI version 2 is unsupported; this OpenVMM supports version 3"; let mut contract = generated_console_contract(); contract.microvm_abi_version = 2; - contract.devices.last_mut().unwrap().feature_banks = vec![0x3000_0001, 0x0000_0003]; + let console = contract.devices.last_mut().unwrap(); + console.feature_banks = vec![0x3000_0001, 0x0000_0003]; + console.irq = Some(7); let err = validate_supported_microvm_contract(&contract).unwrap_err(); assert_eq!(err.to_string(), REFUSAL); @@ -4747,10 +4749,10 @@ mod tests { "earlycon=xe9 console=hvc1 reboot=t panic=-1 \ nvx_sandbox=1 nvx_config=0xd0010000,65536 nr_cpus=2 \ nvx_snapshot_tier=platform \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0003000:4 \ - virtio_mmio.device=0x1000@0xd0006000:11 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0003000:19 \ + virtio_mmio.device=0x1000@0xd0006000:22 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ nvx_control_tty=hvc2" ); validate_manifest_contents(&manifest).unwrap(); diff --git a/vm/devices/chipset/src/ioapic/mod.rs b/vm/devices/chipset/src/ioapic/mod.rs index b6c4e92a432..1da2bbd2e61 100644 --- a/vm/devices/chipset/src/ioapic/mod.rs +++ b/vm/devices/chipset/src/ioapic/mod.rs @@ -500,8 +500,10 @@ mod save_restore { use super::*; use chipset_device::interrupt::HandleEoi; use chipset_device::interrupt::LineInterruptTarget; + use chipset_resources::ioapic::IOAPIC_NUM_ENTRIES; use chipset_resources::ioapic::IoApicRouting; use parking_lot::Mutex; + use std::collections::BTreeMap; use std::sync::Arc; use test_with_tracing::test; @@ -512,6 +514,7 @@ mod save_restore { #[derive(Clone, Default)] struct RecordingRouting { assertions: Arc>>, + routes: Arc>>>, } impl IoApicRouting for RecordingRouting { @@ -519,7 +522,9 @@ mod save_restore { self.assertions.lock().push(irq); } - fn set_route(&self, _irq: u8, _request: Option<(u64, u32)>) {} + fn set_route(&self, irq: u8, request: Option<(u64, u32)>) { + self.routes.lock().insert(irq, request); + } } fn device() -> (IoApicDevice, RecordingRouting) { @@ -657,6 +662,47 @@ mod save_restore { assert_eq!(after.line_levels, before.line_levels); assert!(routing.assertions.lock().is_empty()); } + + #[test] + fn restore_routes_the_inputs_past_the_isa_irqs() { + // The generic IOAPIC's inputs past the 16 ISA IRQs, which the PIC + // does not see, deliver edges and keep their routes across a + // restore like the others. + let inputs = 16..IOAPIC_NUM_ENTRIES; + let source_routing = RecordingRouting::default(); + let mut source = + IoApicDevice::new(IOAPIC_NUM_ENTRIES, Box::new(source_routing.clone())); + for input in inputs.clone() { + let entry = RedirectionEntry::new().with_vector(VECTOR + input); + source.write_redirection_register(input * 2, u64::from(entry) as u32); + source.set_irq(input.into(), true); + source.set_irq(input.into(), false); + } + assert_eq!( + *source_routing.assertions.lock(), + inputs.clone().collect::>() + ); + let state = source.save().expect("save succeeds"); + assert_eq!( + state.redirection_entries.len(), + usize::from(IOAPIC_NUM_ENTRIES) + ); + + let routing = RecordingRouting::default(); + let mut restored = IoApicDevice::new(IOAPIC_NUM_ENTRIES, Box::new(routing.clone())); + restored.restore(state).expect("restore succeeds"); + for input in inputs.clone() { + let route = routing.routes.lock().get(&input).copied().flatten(); + assert!(route.is_some(), "input {input} has no route"); + assert_eq!( + route, + source_routing.routes.lock().get(&input).copied().flatten() + ); + restored.set_irq(input.into(), true); + restored.set_irq(input.into(), false); + } + assert_eq!(*routing.assertions.lock(), inputs.collect::>()); + } } } diff --git a/vm/loader/src/linux/microvm.rs b/vm/loader/src/linux/microvm.rs index 9aa8dcd2ace..a87d01fc317 100644 --- a/vm/loader/src/linux/microvm.rs +++ b/vm/loader/src/linux/microvm.rs @@ -643,6 +643,7 @@ mod tests { mptable::MpTableConfig { apic_ids, level_triggered_irqs: &[4, 5, 6, 7], + ioapic_inputs: 24, } } diff --git a/vm/loader/src/mptable.rs b/vm/loader/src/mptable.rs index 206b23a5c28..38fe9f4b693 100644 --- a/vm/loader/src/mptable.rs +++ b/vm/loader/src/mptable.rs @@ -11,9 +11,14 @@ pub const MP_FLOATING_POINTER_ADDR: usize = 0; pub const MP_CONFIG_TABLE_ADDR: usize = 0x400; const MP_IRQ_FLAGS_LEVEL_HIGH: u16 = 0x000d; +/// Active-high polarity (bits 1:0 = 1) and edge trigger (bits 3:2 = 1). +const MP_IRQ_FLAGS_EDGE_HIGH: u16 = 0x0005; const MP_CONFIG_HEADER_SIZE: usize = 44; const MP_PROCESSOR_SIZE: usize = 20; -const MP_NON_PROCESSOR_ENTRY_COUNT: usize = 17; +/// The size of a bus, IOAPIC, or interrupt-source entry. +const MP_ENTRY_SIZE: usize = 8; +/// The ISA IRQs, which the first IOAPIC inputs carry. +const ISA_IRQ_COUNT: u8 = 16; /// Guest-visible processor and interrupt data represented by MP tables. #[derive(Debug, Clone, Copy)] @@ -22,6 +27,10 @@ pub struct MpTableConfig<'a> { pub apic_ids: &'a [u32], /// ISA IRQs described as active-high, level-triggered. pub level_triggered_irqs: &'a [u32], + /// The number of inputs of the IOAPIC: at least the 16 that carry the ISA + /// IRQs. The table routes each input past those as an active-high, + /// edge-triggered interrupt of its own number, which the PIC does not see. + pub ioapic_inputs: u8, } /// Fully constructed MP 1.4 data. @@ -63,6 +72,9 @@ pub enum Error { /// ISA IRQs are limited to 0 through 15. #[error("MP table IRQ {0} is outside the ISA range")] InvalidIrq(u32), + /// The IOAPIC's first 16 inputs carry the ISA IRQs. + #[error("MP table IOAPIC has {0} inputs; the ISA IRQs need 16")] + TooFewIoApicInputs(u8), } /// Builds the MP floating pointer and configuration table. @@ -85,14 +97,25 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> if config.apic_ids.is_empty() { return Err(Error::NoProcessors); } - if let Some(&irq) = config.level_triggered_irqs.iter().find(|irq| **irq >= 16) { + if let Some(&irq) = config + .level_triggered_irqs + .iter() + .find(|irq| **irq >= u32::from(ISA_IRQ_COUNT)) + { return Err(Error::InvalidIrq(irq)); } + if config.ioapic_inputs < ISA_IRQ_COUNT { + return Err(Error::TooFewIoApicInputs(config.ioapic_inputs)); + } + // One bus, one IOAPIC, and an interrupt source for every ISA IRQ but the + // PIC cascade (IRQ 2) and for every IOAPIC input past the ISA IRQs. + let non_processor_entry_count = + 2 + usize::from(ISA_IRQ_COUNT - 1) + usize::from(config.ioapic_inputs - ISA_IRQ_COUNT); let entry_count = u16::try_from( config .apic_ids .len() - .checked_add(MP_NON_PROCESSOR_ENTRY_COUNT) + .checked_add(non_processor_entry_count) .ok_or(Error::TooManyEntries)?, ) .map_err(|_| Error::TooManyEntries)?; @@ -103,7 +126,7 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> .checked_mul(config.apic_ids.len()) .ok_or(Error::TooManyEntries)?, ) - .and_then(|size| size.checked_add(136)) + .and_then(|size| size.checked_add(MP_ENTRY_SIZE * non_processor_entry_count)) .ok_or(Error::TooManyEntries)?; let mut table = Vec::with_capacity(capacity); table.extend_from_slice(b"PCMP"); @@ -145,7 +168,7 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> table.extend_from_slice(&[2, 0, 0x11, 1]); table.extend_from_slice(&0xfec0_0000u32.to_le_bytes()); - for irq in (0u8..16).filter(|irq| *irq != 2) { + for irq in (0..ISA_IRQ_COUNT).filter(|irq| *irq != 2) { let pin = if irq == 0 { 2 } else { irq }; let flags = if config.level_triggered_irqs.contains(&u32::from(irq)) { MP_IRQ_FLAGS_LEVEL_HIGH @@ -156,6 +179,15 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> table.extend_from_slice(&flags.to_le_bytes()); table.extend_from_slice(&[0, irq, 0, pin]); } + // An input past the ISA IRQs has no bus convention to conform to, so its + // entry states its polarity and trigger. Linux takes the source IRQ of a + // non-PCI bus, here the input's own number, as the interrupt's number. + for pin in ISA_IRQ_COUNT..config.ioapic_inputs { + table.extend_from_slice(&[3, 0]); + table.extend_from_slice(&MP_IRQ_FLAGS_EDGE_HIGH.to_le_bytes()); + table.extend_from_slice(&[0, pin, 0, pin]); + } + assert_eq!(table.len(), capacity); let table_len = u16::try_from(table.len()).map_err(|_| Error::TooManyEntries)?; table[4..6].copy_from_slice(&table_len.to_le_bytes()); @@ -177,6 +209,19 @@ mod tests { use super::*; use test_with_tracing::test; + /// The inputs of the microVM's IOAPIC. + const IOAPIC_INPUTS: u8 = 24; + + /// Returns the interrupt-source entries of `table`, which describes + /// `processor_count` processors, in table order. + fn interrupt_sources(table: &[u8], processor_count: usize) -> Vec<[u8; MP_ENTRY_SIZE]> { + table[MP_CONFIG_HEADER_SIZE + MP_PROCESSOR_SIZE * processor_count..] + .chunks_exact(MP_ENTRY_SIZE) + .map(|entry| entry.try_into().unwrap()) + .filter(|entry: &[u8; MP_ENTRY_SIZE]| entry[0] == 3) + .collect() + } + #[test] fn builds_valid_checksums_and_processor_entries() { for processor_count in [1usize, 2, 4, 8] { @@ -184,6 +229,7 @@ mod tests { let tables = build(&MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[4, 5, 6, 7], + ioapic_inputs: IOAPIC_INPUTS, }) .unwrap(); @@ -208,9 +254,15 @@ mod tests { .fold(0u8, |sum, byte| sum.wrapping_add(byte)), 0 ); + // The header, the processors, the bus, the IOAPIC, and 23 + // interrupt sources: 15 ISA IRQs and inputs 16 through 23. assert_eq!( tables.configuration_table.len(), - 180 + MP_PROCESSOR_SIZE * processor_count + 244 + MP_PROCESSOR_SIZE * processor_count + ); + assert_eq!( + u16::from_le_bytes(tables.configuration_table[34..36].try_into().unwrap()), + processor_count as u16 + 25 ); for (index, entry) in tables.configuration_table [MP_CONFIG_HEADER_SIZE..MP_CONFIG_HEADER_SIZE + MP_PROCESSOR_SIZE * processor_count] @@ -224,12 +276,56 @@ mod tests { } } + #[test] + fn routes_every_ioapic_input() { + let table = build_config_table(&MpTableConfig { + apic_ids: &[0, 1], + level_triggered_irqs: &[5], + ioapic_inputs: IOAPIC_INPUTS, + }) + .unwrap(); + // Each entry: type, interrupt type, flags, source bus, source IRQ, + // destination IOAPIC, and destination input. The PIT's IRQ 0 reaches + // input 2, and IRQ 2, the PIC cascade, has no entry. + let mut expected = vec![[3, 0, 0, 0, 0, 0, 0, 2]]; + for irq in 1..16 { + if irq != 2 { + let flags = if irq == 5 { 0x0d } else { 0 }; + expected.push([3, 0, flags, 0, 0, irq, 0, irq]); + } + } + // The inputs past the ISA IRQs are active high and edge triggered. + for input in 16..24 { + expected.push([3, 0, 0x05, 0, 0, input, 0, input]); + } + assert_eq!(interrupt_sources(&table, 2), expected); + } + + #[test] + fn routes_the_isa_irqs_alone_without_further_inputs() { + let table = build_config_table(&MpTableConfig { + apic_ids: &[0], + level_triggered_irqs: &[], + ioapic_inputs: 16, + }) + .unwrap(); + assert_eq!(table.len(), 180 + MP_PROCESSOR_SIZE); + let sources = interrupt_sources(&table, 1); + assert_eq!(sources.len(), 15); + assert!( + sources + .iter() + .all(|source| source[5] < 16 && source[7] < 16) + ); + } + #[test] fn rejects_invalid_inputs() { assert_eq!( build(&MpTableConfig { apic_ids: &[], level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::NoProcessors) ); @@ -237,21 +333,33 @@ mod tests { build(&MpTableConfig { apic_ids: &[0, 2], level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::InvalidApicId { .. }) )); + // Only the ISA IRQs may be level triggered. assert_eq!( build(&MpTableConfig { apic_ids: &[0], level_triggered_irqs: &[16], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::InvalidIrq(16)) ); + assert_eq!( + build(&MpTableConfig { + apic_ids: &[0], + level_triggered_irqs: &[], + ioapic_inputs: 15, + }), + Err(Error::TooFewIoApicInputs(15)) + ); let apic_ids = (0..=256).collect::>(); assert!(matches!( build(&MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::ApicIdTooLarge { index: 256, diff --git a/vmm_core/src/acpi_builder/test/microvm.rs b/vmm_core/src/acpi_builder/test/microvm.rs index aee23d4e290..859b4d0da7f 100644 --- a/vmm_core/src/acpi_builder/test/microvm.rs +++ b/vmm_core/src/acpi_builder/test/microvm.rs @@ -33,6 +33,7 @@ fn test_microvm_smp_madt_matches_shared_mp_table() { let mp_table = loader::mptable::build_config_table(&loader::mptable::MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[], + ioapic_inputs: 24, }) .unwrap(); let mp_ids = mp_table[44..44 + apic_ids.len() * 20]