From e13c1a644fb1cc32cbf7e0385d7b6fcd75622503 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sun, 1 Nov 2026 15:58:18 -0800 Subject: [PATCH 1/2] microVM: route every IOAPIC input in the MP table Part of microsoft/nvx#759 and microsoft/nvx#1178. The microVM's IOAPIC, the generic IOAPIC of its chipset, has 24 inputs, and every backend routes all of them: KVM reserves 24 IOAPIC GSIs in its split irqchip, and MSHV and WHP keep 24 routes. The IRQ line set reaches the PIC only on the 16 ISA IRQs, so inputs 16 through 23 reach the IOAPIC alone. But the MP table described only the ISA IRQs, and Linux sets up an interrupt for an IOAPIC input only when an MP interrupt-source entry names it, so a device could not use an input past the ISA IRQs: a larger IRQ number on the kernel command line was not enough. MpTableConfig gets ioapic_inputs, the number of the IOAPIC's inputs, at least the 16 that carry the ISA IRQs. The table routes the ISA IRQs as before and then each input from 16 up with an interrupt-source entry on the ISA bus whose source IRQ is the input's own number, with an explicit active-high polarity and edge trigger (flags 0x0005), since no bus convention applies past the ISA IRQs. Linux's MP-table parser treats every bus but PCI as ISA-like and gives a source IRQ of 16 or more the interrupt number of the input's GSI, so a "virtio_mmio.device=...:16" token names input 16. Firecracker describes its IOAPIC's 24 inputs the same way. The builder refuses an IOAPIC of fewer than 16 inputs, and level triggering stays limited to the ISA IRQs. openvmm_defs::microvm::MICROVM_IOAPIC_INPUTS names the microVM's 24 inputs, from IOAPIC_NUM_ENTRIES, and the microVM loader passes it. The configuration table grows by eight 8-byte entries to 23 interrupt sources, well within its page before the GDT at 0x1000. Linux now sets up IRQs 16 through 23 at boot, masked until a driver requests one; no device uses them yet. The machine's interrupts belong to the versioned machine contract, so this extends microVM ABI version 3, which the layer below introduced and which no release has shipped, rather than bumping it again. The boot layout keeps version 2: the MP tables keep their placement and MP 1.4 format, and their entries describe the machine, as their processor entries follow the topology. Tests: - loader: routes_every_ioapic_input decodes every interrupt source of a 24-input IOAPIC, from the PIT's IRQ 0 on input 2 to the edge entries of inputs 16 through 23; a 16-input IOAPIC keeps the previous table (routes_the_isa_irqs_alone_without_further_inputs); rejects_invalid_inputs refuses 15 inputs; and the checksum test checks the new length and entry count. - chipset: restore_routes_the_inputs_past_the_isa_irqs shows that inputs 16 through 23 of the 24-input IOAPIC deliver edges and keep their routes across a save and restore, which already covered every input. - openvmm_defs: microvm_ioapic_inputs_have_backend_routes requires the IOAPIC's inputs to equal the routes that every backend keeps. Update the Guide's Linux direct boot and CLI pages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../devices/firmware/linux_direct.md | 4 +- Guide/src/reference/openvmm/management/cli.md | 5 +- .../src/worker/dispatch/microvm.rs | 1 + .../src/worker/vm_loaders/microvm.rs | 2 + openvmm/openvmm_defs/src/microvm.rs | 12 ++ vm/devices/chipset/src/ioapic/mod.rs | 48 ++++++- vm/loader/src/linux/microvm.rs | 1 + vm/loader/src/mptable.rs | 120 +++++++++++++++++- vmm_core/src/acpi_builder/test/microvm.rs | 1 + 9 files changed, 184 insertions(+), 10 deletions(-) diff --git a/Guide/src/reference/devices/firmware/linux_direct.md b/Guide/src/reference/devices/firmware/linux_direct.md index 06c34ba94e6..e8bd5196c28 100644 --- a/Guide/src/reference/devices/firmware/linux_direct.md +++ b/Guide/src/reference/devices/firmware/linux_direct.md @@ -27,7 +27,9 @@ uncompressed ELF kernel, but selects an ACPI-free platform mode. OpenVMM places an Intel MP 1.4 floating pointer at `0x0`, its configuration table at `0x400`, the bootstrap GDT at `0x1000`, `boot_params` at `0x2000`, and the command line in the 64-KiB region starting at `0x20000`. The MP table describes contiguous -APIC IDs, the ISA bus, the IOAPIC, and legacy IRQ routing. No RSDP, ACPI table, +APIC IDs, the ISA bus, the IOAPIC, and the routing of each of the IOAPIC's 24 +inputs: the ISA IRQs, and inputs 16 through 23, which the PIC does not see, as +active-high, edge-triggered interrupts of their own numbers. No RSDP, ACPI table, SMBIOS anchor, or SMBIOS structure table is imported. On AArch64, pass the uncompressed `Image` file (not `Image.gz`). diff --git a/Guide/src/reference/openvmm/management/cli.md b/Guide/src/reference/openvmm/management/cli.md index c995220c21e..7d08de6f931 100644 --- a/Guide/src/reference/openvmm/management/cli.md +++ b/Guide/src/reference/openvmm/management/cli.md @@ -71,8 +71,9 @@ describes the source definitions. notification suppression. No device offers `VIRTIO_F_ACCESS_PLATFORM`, since the microVM has no IOMMU, and no console offers its size. Firmware, ACPI, SMBIOS, PCI, VMBus, UARTs, graphics, isolation, nested virtualization, and - other devices are rejected. Linux discovers contiguous APIC IDs and the - IOAPIC from Intel MP 1.4 tables at `0x0` and `0x400`; `boot_params` is at + other devices are rejected. Linux discovers contiguous APIC IDs, the IOAPIC, + and the routing of its 24 inputs from Intel MP 1.4 tables at `0x0` and + `0x400`; `boot_params` is at `0x2000`, the command line starts at `0x20000`, and no ACPI or SMBIOS data is exposed. Host-driven save/restore, pulse-save/restore, and worker restart remain unavailable. diff --git a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs index 11c680f22bd..978378f82fe 100644 --- a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs +++ b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs @@ -187,6 +187,7 @@ pub(super) fn load_linux_x86_mptable( &vm.gm, &apic_ids, &openvmm_defs::microvm::MICROVM_LEVEL_TRIGGERED_IRQS, + openvmm_defs::microvm::MICROVM_IOAPIC_INPUTS, &reserved_memory_ranges, )?) } diff --git a/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs b/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs index d850a721792..21497306908 100644 --- a/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs +++ b/openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs @@ -22,6 +22,7 @@ pub fn load_linux_x86_mptable( gm: &GuestMemory, apic_ids: &[u32], level_triggered_irqs: &[u32], + ioapic_inputs: u8, reserved_memory_ranges: &[MemoryRange], ) -> Result, LinuxError> { if !matches!(cfg.isolation, KernelIsolationConfig::None) { @@ -54,6 +55,7 @@ pub fn load_linux_x86_mptable( &loader::mptable::MpTableConfig { apic_ids, level_triggered_irqs, + ioapic_inputs, }, reserved_memory_ranges, ) diff --git a/openvmm/openvmm_defs/src/microvm.rs b/openvmm/openvmm_defs/src/microvm.rs index 9a7980ba685..a72b4b6274e 100644 --- a/openvmm/openvmm_defs/src/microvm.rs +++ b/openvmm/openvmm_defs/src/microvm.rs @@ -222,6 +222,10 @@ pub const fn microvm_virtio_status_gpa(mmio_base: u64) -> Option { } /// The microVM publishes no level-triggered virtio IRQs in its MP table. pub const MICROVM_LEVEL_TRIGGERED_IRQS: [u32; 0] = []; +/// The inputs of the microVM's IOAPIC, the generic IOAPIC of its chipset, +/// every one of which its MP table routes: the first 16 carry the ISA IRQs, +/// which the PIC also sees, and the IOAPIC alone sees inputs 16 through 23. +pub const MICROVM_IOAPIC_INPUTS: u8 = chipset_resources::ioapic::IOAPIC_NUM_ENTRIES; /// The stable role of a microVM sandbox block device. #[derive(MeshPayload, Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] @@ -2674,6 +2678,14 @@ mod tests { assert!(MICROVM_LEVEL_TRIGGERED_IRQS.is_empty()); } + #[test] + fn microvm_ioapic_inputs_have_backend_routes() { + // The MP table routes every input of the IOAPIC, and every backend + // keeps a route for each: KVM claims a GSI, and MSHV and WHP a route. + assert_eq!(MICROVM_IOAPIC_INPUTS, 24); + assert_eq!(usize::from(MICROVM_IOAPIC_INPUTS), virt::irqcon::IRQ_LINES); + } + #[test] fn microvm_shared_status_slots_are_stable() { let slots = [ diff --git a/vm/devices/chipset/src/ioapic/mod.rs b/vm/devices/chipset/src/ioapic/mod.rs index b6c4e92a432..1da2bbd2e61 100644 --- a/vm/devices/chipset/src/ioapic/mod.rs +++ b/vm/devices/chipset/src/ioapic/mod.rs @@ -500,8 +500,10 @@ mod save_restore { use super::*; use chipset_device::interrupt::HandleEoi; use chipset_device::interrupt::LineInterruptTarget; + use chipset_resources::ioapic::IOAPIC_NUM_ENTRIES; use chipset_resources::ioapic::IoApicRouting; use parking_lot::Mutex; + use std::collections::BTreeMap; use std::sync::Arc; use test_with_tracing::test; @@ -512,6 +514,7 @@ mod save_restore { #[derive(Clone, Default)] struct RecordingRouting { assertions: Arc>>, + routes: Arc>>>, } impl IoApicRouting for RecordingRouting { @@ -519,7 +522,9 @@ mod save_restore { self.assertions.lock().push(irq); } - fn set_route(&self, _irq: u8, _request: Option<(u64, u32)>) {} + fn set_route(&self, irq: u8, request: Option<(u64, u32)>) { + self.routes.lock().insert(irq, request); + } } fn device() -> (IoApicDevice, RecordingRouting) { @@ -657,6 +662,47 @@ mod save_restore { assert_eq!(after.line_levels, before.line_levels); assert!(routing.assertions.lock().is_empty()); } + + #[test] + fn restore_routes_the_inputs_past_the_isa_irqs() { + // The generic IOAPIC's inputs past the 16 ISA IRQs, which the PIC + // does not see, deliver edges and keep their routes across a + // restore like the others. + let inputs = 16..IOAPIC_NUM_ENTRIES; + let source_routing = RecordingRouting::default(); + let mut source = + IoApicDevice::new(IOAPIC_NUM_ENTRIES, Box::new(source_routing.clone())); + for input in inputs.clone() { + let entry = RedirectionEntry::new().with_vector(VECTOR + input); + source.write_redirection_register(input * 2, u64::from(entry) as u32); + source.set_irq(input.into(), true); + source.set_irq(input.into(), false); + } + assert_eq!( + *source_routing.assertions.lock(), + inputs.clone().collect::>() + ); + let state = source.save().expect("save succeeds"); + assert_eq!( + state.redirection_entries.len(), + usize::from(IOAPIC_NUM_ENTRIES) + ); + + let routing = RecordingRouting::default(); + let mut restored = IoApicDevice::new(IOAPIC_NUM_ENTRIES, Box::new(routing.clone())); + restored.restore(state).expect("restore succeeds"); + for input in inputs.clone() { + let route = routing.routes.lock().get(&input).copied().flatten(); + assert!(route.is_some(), "input {input} has no route"); + assert_eq!( + route, + source_routing.routes.lock().get(&input).copied().flatten() + ); + restored.set_irq(input.into(), true); + restored.set_irq(input.into(), false); + } + assert_eq!(*routing.assertions.lock(), inputs.collect::>()); + } } } diff --git a/vm/loader/src/linux/microvm.rs b/vm/loader/src/linux/microvm.rs index 9aa8dcd2ace..a87d01fc317 100644 --- a/vm/loader/src/linux/microvm.rs +++ b/vm/loader/src/linux/microvm.rs @@ -643,6 +643,7 @@ mod tests { mptable::MpTableConfig { apic_ids, level_triggered_irqs: &[4, 5, 6, 7], + ioapic_inputs: 24, } } diff --git a/vm/loader/src/mptable.rs b/vm/loader/src/mptable.rs index 206b23a5c28..38fe9f4b693 100644 --- a/vm/loader/src/mptable.rs +++ b/vm/loader/src/mptable.rs @@ -11,9 +11,14 @@ pub const MP_FLOATING_POINTER_ADDR: usize = 0; pub const MP_CONFIG_TABLE_ADDR: usize = 0x400; const MP_IRQ_FLAGS_LEVEL_HIGH: u16 = 0x000d; +/// Active-high polarity (bits 1:0 = 1) and edge trigger (bits 3:2 = 1). +const MP_IRQ_FLAGS_EDGE_HIGH: u16 = 0x0005; const MP_CONFIG_HEADER_SIZE: usize = 44; const MP_PROCESSOR_SIZE: usize = 20; -const MP_NON_PROCESSOR_ENTRY_COUNT: usize = 17; +/// The size of a bus, IOAPIC, or interrupt-source entry. +const MP_ENTRY_SIZE: usize = 8; +/// The ISA IRQs, which the first IOAPIC inputs carry. +const ISA_IRQ_COUNT: u8 = 16; /// Guest-visible processor and interrupt data represented by MP tables. #[derive(Debug, Clone, Copy)] @@ -22,6 +27,10 @@ pub struct MpTableConfig<'a> { pub apic_ids: &'a [u32], /// ISA IRQs described as active-high, level-triggered. pub level_triggered_irqs: &'a [u32], + /// The number of inputs of the IOAPIC: at least the 16 that carry the ISA + /// IRQs. The table routes each input past those as an active-high, + /// edge-triggered interrupt of its own number, which the PIC does not see. + pub ioapic_inputs: u8, } /// Fully constructed MP 1.4 data. @@ -63,6 +72,9 @@ pub enum Error { /// ISA IRQs are limited to 0 through 15. #[error("MP table IRQ {0} is outside the ISA range")] InvalidIrq(u32), + /// The IOAPIC's first 16 inputs carry the ISA IRQs. + #[error("MP table IOAPIC has {0} inputs; the ISA IRQs need 16")] + TooFewIoApicInputs(u8), } /// Builds the MP floating pointer and configuration table. @@ -85,14 +97,25 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> if config.apic_ids.is_empty() { return Err(Error::NoProcessors); } - if let Some(&irq) = config.level_triggered_irqs.iter().find(|irq| **irq >= 16) { + if let Some(&irq) = config + .level_triggered_irqs + .iter() + .find(|irq| **irq >= u32::from(ISA_IRQ_COUNT)) + { return Err(Error::InvalidIrq(irq)); } + if config.ioapic_inputs < ISA_IRQ_COUNT { + return Err(Error::TooFewIoApicInputs(config.ioapic_inputs)); + } + // One bus, one IOAPIC, and an interrupt source for every ISA IRQ but the + // PIC cascade (IRQ 2) and for every IOAPIC input past the ISA IRQs. + let non_processor_entry_count = + 2 + usize::from(ISA_IRQ_COUNT - 1) + usize::from(config.ioapic_inputs - ISA_IRQ_COUNT); let entry_count = u16::try_from( config .apic_ids .len() - .checked_add(MP_NON_PROCESSOR_ENTRY_COUNT) + .checked_add(non_processor_entry_count) .ok_or(Error::TooManyEntries)?, ) .map_err(|_| Error::TooManyEntries)?; @@ -103,7 +126,7 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> .checked_mul(config.apic_ids.len()) .ok_or(Error::TooManyEntries)?, ) - .and_then(|size| size.checked_add(136)) + .and_then(|size| size.checked_add(MP_ENTRY_SIZE * non_processor_entry_count)) .ok_or(Error::TooManyEntries)?; let mut table = Vec::with_capacity(capacity); table.extend_from_slice(b"PCMP"); @@ -145,7 +168,7 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> table.extend_from_slice(&[2, 0, 0x11, 1]); table.extend_from_slice(&0xfec0_0000u32.to_le_bytes()); - for irq in (0u8..16).filter(|irq| *irq != 2) { + for irq in (0..ISA_IRQ_COUNT).filter(|irq| *irq != 2) { let pin = if irq == 0 { 2 } else { irq }; let flags = if config.level_triggered_irqs.contains(&u32::from(irq)) { MP_IRQ_FLAGS_LEVEL_HIGH @@ -156,6 +179,15 @@ pub fn build_config_table(config: &MpTableConfig<'_>) -> Result, Error> table.extend_from_slice(&flags.to_le_bytes()); table.extend_from_slice(&[0, irq, 0, pin]); } + // An input past the ISA IRQs has no bus convention to conform to, so its + // entry states its polarity and trigger. Linux takes the source IRQ of a + // non-PCI bus, here the input's own number, as the interrupt's number. + for pin in ISA_IRQ_COUNT..config.ioapic_inputs { + table.extend_from_slice(&[3, 0]); + table.extend_from_slice(&MP_IRQ_FLAGS_EDGE_HIGH.to_le_bytes()); + table.extend_from_slice(&[0, pin, 0, pin]); + } + assert_eq!(table.len(), capacity); let table_len = u16::try_from(table.len()).map_err(|_| Error::TooManyEntries)?; table[4..6].copy_from_slice(&table_len.to_le_bytes()); @@ -177,6 +209,19 @@ mod tests { use super::*; use test_with_tracing::test; + /// The inputs of the microVM's IOAPIC. + const IOAPIC_INPUTS: u8 = 24; + + /// Returns the interrupt-source entries of `table`, which describes + /// `processor_count` processors, in table order. + fn interrupt_sources(table: &[u8], processor_count: usize) -> Vec<[u8; MP_ENTRY_SIZE]> { + table[MP_CONFIG_HEADER_SIZE + MP_PROCESSOR_SIZE * processor_count..] + .chunks_exact(MP_ENTRY_SIZE) + .map(|entry| entry.try_into().unwrap()) + .filter(|entry: &[u8; MP_ENTRY_SIZE]| entry[0] == 3) + .collect() + } + #[test] fn builds_valid_checksums_and_processor_entries() { for processor_count in [1usize, 2, 4, 8] { @@ -184,6 +229,7 @@ mod tests { let tables = build(&MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[4, 5, 6, 7], + ioapic_inputs: IOAPIC_INPUTS, }) .unwrap(); @@ -208,9 +254,15 @@ mod tests { .fold(0u8, |sum, byte| sum.wrapping_add(byte)), 0 ); + // The header, the processors, the bus, the IOAPIC, and 23 + // interrupt sources: 15 ISA IRQs and inputs 16 through 23. assert_eq!( tables.configuration_table.len(), - 180 + MP_PROCESSOR_SIZE * processor_count + 244 + MP_PROCESSOR_SIZE * processor_count + ); + assert_eq!( + u16::from_le_bytes(tables.configuration_table[34..36].try_into().unwrap()), + processor_count as u16 + 25 ); for (index, entry) in tables.configuration_table [MP_CONFIG_HEADER_SIZE..MP_CONFIG_HEADER_SIZE + MP_PROCESSOR_SIZE * processor_count] @@ -224,12 +276,56 @@ mod tests { } } + #[test] + fn routes_every_ioapic_input() { + let table = build_config_table(&MpTableConfig { + apic_ids: &[0, 1], + level_triggered_irqs: &[5], + ioapic_inputs: IOAPIC_INPUTS, + }) + .unwrap(); + // Each entry: type, interrupt type, flags, source bus, source IRQ, + // destination IOAPIC, and destination input. The PIT's IRQ 0 reaches + // input 2, and IRQ 2, the PIC cascade, has no entry. + let mut expected = vec![[3, 0, 0, 0, 0, 0, 0, 2]]; + for irq in 1..16 { + if irq != 2 { + let flags = if irq == 5 { 0x0d } else { 0 }; + expected.push([3, 0, flags, 0, 0, irq, 0, irq]); + } + } + // The inputs past the ISA IRQs are active high and edge triggered. + for input in 16..24 { + expected.push([3, 0, 0x05, 0, 0, input, 0, input]); + } + assert_eq!(interrupt_sources(&table, 2), expected); + } + + #[test] + fn routes_the_isa_irqs_alone_without_further_inputs() { + let table = build_config_table(&MpTableConfig { + apic_ids: &[0], + level_triggered_irqs: &[], + ioapic_inputs: 16, + }) + .unwrap(); + assert_eq!(table.len(), 180 + MP_PROCESSOR_SIZE); + let sources = interrupt_sources(&table, 1); + assert_eq!(sources.len(), 15); + assert!( + sources + .iter() + .all(|source| source[5] < 16 && source[7] < 16) + ); + } + #[test] fn rejects_invalid_inputs() { assert_eq!( build(&MpTableConfig { apic_ids: &[], level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::NoProcessors) ); @@ -237,21 +333,33 @@ mod tests { build(&MpTableConfig { apic_ids: &[0, 2], level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::InvalidApicId { .. }) )); + // Only the ISA IRQs may be level triggered. assert_eq!( build(&MpTableConfig { apic_ids: &[0], level_triggered_irqs: &[16], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::InvalidIrq(16)) ); + assert_eq!( + build(&MpTableConfig { + apic_ids: &[0], + level_triggered_irqs: &[], + ioapic_inputs: 15, + }), + Err(Error::TooFewIoApicInputs(15)) + ); let apic_ids = (0..=256).collect::>(); assert!(matches!( build(&MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[], + ioapic_inputs: IOAPIC_INPUTS, }), Err(Error::ApicIdTooLarge { index: 256, diff --git a/vmm_core/src/acpi_builder/test/microvm.rs b/vmm_core/src/acpi_builder/test/microvm.rs index aee23d4e290..859b4d0da7f 100644 --- a/vmm_core/src/acpi_builder/test/microvm.rs +++ b/vmm_core/src/acpi_builder/test/microvm.rs @@ -33,6 +33,7 @@ fn test_microvm_smp_madt_matches_shared_mp_table() { let mp_table = loader::mptable::build_config_table(&loader::mptable::MpTableConfig { apic_ids: &apic_ids, level_triggered_irqs: &[], + ioapic_inputs: 24, }) .unwrap(); let mp_ids = mp_table[44..44 + apic_ids.len() * 20] From 35e76265fa8b63922edead112fcd5a38c59445c8 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sun, 1 Nov 2026 16:07:45 -0800 Subject: [PATCH 2/2] microVM: give each virtio slot its own IOAPIC input in ABI 3 Part of microsoft/nvx#759 and microsoft/nvx#1178. Every microVM virtio device used an ISA IRQ: the NIC IRQ 10 on KVM and MSHV but IRQ 5 on WHP, the host filesystem 6, the boot console 7, the distro block 4, the scratch block 11, and the control console 3. With the PIT on IRQ 0, the PIC cascade on 2, and the RTC on 8, only IRQs 1, 9, and 12 through 15 were left for a further fixed device. Each of those lines also reached the PIC, which Linux masks once it routes interrupts through the IOAPIC, so every virtio interrupt also toggled a masked PIC input. And the NIC's IRQ was the machine contract's one backend-specific value: nothing in the code needs it, since every backend delivers every IOAPIC input the same way, through the userspace IOAPIC and an MSI route of the backend, and the history records no reason for it. The previous commit routes IOAPIC inputs 16 through 23 in the MP table. Move every virtio slot to one of them, the same on every backend: the slot whose window is n windows past 0xd0000000 interrupts on input 16 + n, as its shared-status word is word n of the status page: - the NIC, window 0: IRQ 16 (was 10, or 5 on WHP); - the host filesystem, window 1: IRQ 17 (was 6); - the boot console, window 2: IRQ 18 (was 7); - the distro block, window 3: IRQ 19 (was 4); - the scratch block, window 6: IRQ 22 (was 11); - the control console, window 7: IRQ 23 (was 3). Inputs 20 and 21 stay free with the windows of the retired runtime and custom layers. The ISA IRQs now carry only the PIT and the RTC, so IRQs 1, 3 through 7, and 9 through 15 are free as well, and the PIC sees no virtio interrupt. Each slot's interrupt stays active-high and edge triggered, and the guest discovers it from the same virtio_mmio.device= token, with the new number. In openvmm_defs::microvm: - MICROVM_VIRTIO_NET_IRQ replaces MICROVM_VIRTIO_NET_KVM_IRQ, MICROVM_VIRTIO_NET_WHP_IRQ, and microvm_virtio_net_irq(), so the network tuple of append_microvm_virtio_discovery no longer carries an interrupt, and the command-line validation no longer needs the hypervisor. - MICROVM_VIRTIO_IRQ_BASE names input 16, and microvm_virtio_irq() returns a slot's interrupt by its window, as microvm_virtio_status_gpa() returns its status word. - validate_machine_config() now also refuses a slot without an interrupt, one on an ISA IRQ or past the IOAPIC's last input, and two slots on one input. Snapshots record each device's interrupt and the effective command line, whose discovery tokens carry it, so a snapshot of the old interrupts no longer matches the machine: its restore stops at the command-line check. That check now names the first discovery token that differs, such as virtio_mmio.device=0x1000@0xd0000000:10 where virtio_mmio.device=0x1000@0xd0000000:16 belongs, rather than reporting only that the tokens are not in fixed-address order. This extends microVM ABI version 3, which the layer below introduced to drop access-platform and the console size and which no release has shipped, rather than bumping it again: a snapshot of version 2 is still refused by its version before anything is compared. The boot layout keeps version 2. Tests: - openvmm_defs: microvm_virtio_slots_have_their_own_ioapic_inputs pins the six interrupts, the window rule, and the free inputs; microvm_virtio_irq_validation_rejects_collisions refuses a shared input, an ISA IRQ, an input past the IOAPIC, and a missing interrupt; microvm_discovery_token_mismatch_names_the_token names a token on another interrupt and one out of order; and the discovery tests carry the new tokens. - openvmm_core: microvm_devices_offer_exactly_their_feature_masks also checks the interrupt that virtio_mmio_config gives each of the six devices. - openvmm_helpers: the generated contracts record the new interrupts, the network contract the same one on KVM, MSHV, and WHP, and the version 2 console of abi_v2_contracts_are_refused_by_their_version also records its old IRQ. - openvmm_entry: the CLI command-line tests carry the new tokens. Update the Guide's CLI, gRPC, run, and snapshot pages and its virtio-fs device reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/reference/devices/virtio/virtio-fs.md | 2 +- Guide/src/reference/openvmm/management/cli.md | 33 +- .../src/reference/openvmm/management/grpc.md | 7 +- Guide/src/user_guide/openvmm/run.md | 15 +- Guide/src/user_guide/openvmm/snapshots.md | 2 +- .../src/worker/dispatch/microvm.rs | 31 +- openvmm/openvmm_defs/src/microvm.rs | 307 +++++++++++++----- openvmm/openvmm_entry/src/cli_args/microvm.rs | 34 +- openvmm/openvmm_entry/src/microvm/config.rs | 9 +- openvmm/openvmm_entry/src/microvm/network.rs | 3 +- .../openvmm_helpers/src/snapshot/microvm.rs | 50 +-- 11 files changed, 312 insertions(+), 181 deletions(-) diff --git a/Guide/src/reference/devices/virtio/virtio-fs.md b/Guide/src/reference/devices/virtio/virtio-fs.md index 95113e1a992..755e15de214 100644 --- a/Guide/src/reference/devices/virtio/virtio-fs.md +++ b/Guide/src/reference/devices/virtio/virtio-fs.md @@ -45,7 +45,7 @@ fixes the remaining guest-visible configuration: | Stable ID | `fs:microvm0` | | Tag | `microvm` | | Transport | virtio-mmio at `0xd0001000` | -| Interrupt | IRQ 6 | +| Interrupt | IRQ 17, an IOAPIC input past the ISA IRQs | | Queues | One high-priority and one request queue | | Features | Indirect descriptors, event index, version 1 | | DAX window | None | diff --git a/Guide/src/reference/openvmm/management/cli.md b/Guide/src/reference/openvmm/management/cli.md index 7d08de6f931..8da4addf608 100644 --- a/Guide/src/reference/openvmm/management/cli.md +++ b/Guide/src/reference/openvmm/management/cli.md @@ -52,17 +52,20 @@ describes the source definitions. rejects a longer command line, or one with an embedded NUL, rather than let the guest truncate it. - One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 6 and remains + One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 17 and remains dormant when neither `--mount` nor `--mount-aggregate` is given; either one binds HostFs to it; one optional `--virtio-console ` is exposed at MMIO `0xd0002000`, - IRQ 7 as the boot/log console (`hvc1`); and `--microvm-sandbox-block` - exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 4 and the fixed - `scratch` slot at MMIO `0xd0006000`, IRQ 11. The windows between them and - IRQs 9 and 12 stay unassigned. The profile also reserves MMIO `0xd0007000`, - IRQ 3 for the dedicated control console (`hvc2`), selected by the host-owned - `nvx_control_tty=hvc2` token. The reserved slot and resource identity do not - expose a live control endpoint. Both consoles + IRQ 18 as the boot/log console (`hvc1`); and `--microvm-sandbox-block` + exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 19 and the fixed + `scratch` slot at MMIO `0xd0006000`, IRQ 22. The windows between them and + IRQs 20 and 21 stay unassigned. The profile also reserves MMIO `0xd0007000`, + IRQ 23 for the dedicated control console (`hvc2`), selected by the host-owned + `nvx_control_tty=hvc2` token. Each virtio-mmio window, `0xd0000000 + n * + 0x1000`, interrupts on its own IOAPIC input, `16 + n`, with active-high edges + on every backend; the PIC sees none of these inputs, and the ISA IRQs carry + only the PIT (IRQ 0) and the RTC (IRQ 8). The reserved slot and resource + identity do not expose a live control endpoint. Both consoles remain virtio-console devices from the guest's perspective. Ordinary `--virtio-blk` is rejected. All use split rings, and each offers an exact feature mask of virtio 1.0 features: the NIC its MAC address; the block @@ -79,7 +82,7 @@ describes the source definitions. remain unavailable. `microvm` may also expose a dedicated control virtio-console at MMIO - `0xd0007000`, IRQ 3. It requires the boot virtio-console, preserves + `0xd0007000`, IRQ 23. It requires the boot virtio-console, preserves `console=hvc1`, and publishes `nvx_control_tty=hvc2`. The profile fixes boot-before-control discovery order and rejects user overrides that could change it. @@ -103,11 +106,13 @@ describes the source definitions. as `microvm-v2`; the `microvm-v2` selector and the former ABI-v1 behavior are removed. Snapshots record microVM ABI value 3 and boot layout value 2. ABI value 3 offers no device `VIRTIO_F_ACCESS_PLATFORM` and no console - `VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, so restore refuses a + `VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, and gives each virtio-mmio + slot its own IOAPIC input past the ISA IRQs on every backend, where value 2 + used ISA IRQs and put the NIC on another one on WHP, so restore refuses a snapshot of value 2, which must be recaptured. ``` * `--net `: With `--machine microvm`, attach one virtio-net NIC - at MMIO `0xd0000000`. KVM and MSHV use IRQ 10; WHP uses IRQ 5. Prefixes + at MMIO `0xd0000000`, IRQ 16, on every backend. Prefixes `/1` through `/30` are accepted. The first usable subnet address becomes the gateway; network, broadcast, and gateway addresses cannot be assigned to the guest. Guest and gateway MAC addresses are derived as @@ -267,7 +272,7 @@ describes the source definitions. protocol does not retain pre-capture endpoint completions; restored guest software must establish new host-side flows. * `--mount `: With `--machine microvm`, attach - one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 6, with tag `microvm`. + one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 17, with tag `microvm`. `--mount` may appear once. The default mode is read-only; `rw` must be explicit. The guest target ends at the first comma, and the mode follows the last comma, so a host path may contain commas when the mode is given. @@ -1007,7 +1012,7 @@ Serial devices can be configured to appear as different devices inside the guest behaves normally. * `--virtio-console `: Expose a virtio console device. It normally appears as `/dev/hvc0`. Under `--machine microvm`, it occupies fixed MMIO - `0xd0002000`, IRQ 7, and is selected as `/dev/hvc1`; the raw portb path + `0xd0002000`, IRQ 18, and is selected as `/dev/hvc1`; the raw portb path remains available as `hvc0` for early output and recovery. A microVM accepts these explicit attachment policies: @@ -1035,7 +1040,7 @@ Serial devices can be configured to appear as different devices inside the guest * `--microvm-control-console `: With `--machine microvm`, expose a second independent single-port virtio console at fixed MMIO `0xd0007000`, IRQ - 3. `--virtio-console` is required on a fresh boot and remains the only kernel + 23. `--virtio-console` is required on a fresh boot and remains the only kernel console. The control device normally appears as the profile-owned `nvx_control_tty=hvc2`. diff --git a/Guide/src/reference/openvmm/management/grpc.md b/Guide/src/reference/openvmm/management/grpc.md index caeb7d9bb5d..71b0c058761 100644 --- a/Guide/src/reference/openvmm/management/grpc.md +++ b/Guide/src/reference/openvmm/management/grpc.md @@ -219,7 +219,7 @@ On cold boot, `DevicesConfig.virtio_consoles` may hold one entry, which configures a microVM Unix socket or named-pipe endpoint. Listener mode recreates the path on restore. Client mode is required and uses a five-second connection timeout -before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 7, and selects +before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 18, and selects `hvc1`; its canonical path and policy become the stable restore attachment. `DevicesConfig.virtiofs_config` may bind one HostFs attachment to the fixed @@ -232,7 +232,7 @@ target, and access mode, and the live root must retain its saved identity; the snapshot's denied, allowed, and writable paths apply again. A snapshot captured with the slot dormant may omit the attachment or supply a new one; after `ResumeVM`, the guest explicitly -mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 6, one +mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 17, one request queue, and no DAX window. The management API binds only a single directory: it cannot attach an aggregate of several host directories, as the command line's `--mount-aggregate` does, and it cannot restore a snapshot of @@ -264,7 +264,8 @@ currently blockless; role-bearing sandbox blocks remain CLI-only. Snapshots record microVM ABI value 3 and boot layout value 2, which identifies the Linux-direct MP-table layout. Snapshots of ABI values 1 and 2 are unsupported: value 2 offered `VIRTIO_F_ACCESS_PLATFORM` on every device but the NIC, and the -console size on the consoles. +console size on the consoles, and put the virtio devices on ISA IRQs, where +value 3 gives each one its own IOAPIC input past them. `VMConfig.cpu_profile` selects the microVM's CPU profile as [`--cpu-profile`](cli.md) does, with the same values and checks: `auto`, the diff --git a/Guide/src/user_guide/openvmm/run.md b/Guide/src/user_guide/openvmm/run.md index 0da0ac7efde..84f8d99b9a6 100644 --- a/Guide/src/user_guide/openvmm/run.md +++ b/Guide/src/user_guide/openvmm/run.md @@ -44,11 +44,12 @@ locations: | Role | Access | MMIO address | IRQ | | --- | --- | ---: | ---: | -| `distro` | read-only | `0xd0003000` | 4 | -| `scratch` | writable | `0xd0006000` | 11 | +| `distro` | read-only | `0xd0003000` | 19 | +| `scratch` | writable | `0xd0006000` | 22 | -The windows `0xd0004000` and `0xd0005000` between them, and IRQs 9 and 12, -stay unassigned, so that no slot moves. +The windows `0xd0004000` and `0xd0005000` between them, and their IRQs 20 and +21, stay unassigned, so that no slot moves. Each IRQ is an IOAPIC input past +the ISA IRQs, which the PIC does not see. Use `--microvm-sandbox-block ROLE:DISK`, in the order shown. The `distro` role requires the normal disk `,ro` option and a non-empty topology must end @@ -94,8 +95,10 @@ Custom socket, SMT, APIC, x2APIC, and NUMA options are rejected. The microVM uses fixed virtio device slots and sandbox block roles. Its persisted ABI is value 3 and its boot layout value 2. Each fixed virtio device offers an exact feature mask, which never includes `VIRTIO_F_ACCESS_PLATFORM`, -since the microVM has no IOMMU, nor packed virtqueues. The -MP floating pointer begins at `0x0`, the MP configuration table at `0x400`, +since the microVM has no IOMMU, nor packed virtqueues, and interrupts on an +IOAPIC input of its own past the ISA IRQs, the same on every backend. The +MP floating pointer begins at `0x0`, the MP configuration table, which routes +all 24 IOAPIC inputs, at `0x400`, the boot GDT at `0x1000`, and the Linux zero page at `0x2000`. No ACPI MADT or SMBIOS data is exposed. diff --git a/Guide/src/user_guide/openvmm/snapshots.md b/Guide/src/user_guide/openvmm/snapshots.md index 0b2b07665bf..835b5c31afc 100644 --- a/Guide/src/user_guide/openvmm/snapshots.md +++ b/Guide/src/user_guide/openvmm/snapshots.md @@ -606,7 +606,7 @@ immediately with a clear error if any active device does not support it. `--processors`. MicroVM restore reads them authoritatively from the manifest and rejects overrides. Persisted microVM ABI value 3 and boot-layout value 2 are supported; a snapshot of ABI value 2, whose devices offered - `VIRTIO_F_ACCESS_PLATFORM`, must be recaptured. + `VIRTIO_F_ACCESS_PLATFORM` and used ISA IRQs, must be recaptured. - MicroVM restore requires the backend, CPU generation, and CPU profile of the capture, and an OpenVMM build that computes the same effective CPUID. A host profile's snapshot, including one of a cold boot whose `auto` fell diff --git a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs index 978378f82fe..7cee87dd756 100644 --- a/openvmm/openvmm_core/src/worker/dispatch/microvm.rs +++ b/openvmm/openvmm_core/src/worker/dispatch/microvm.rs @@ -485,7 +485,7 @@ fn virtio_mmio_config( let (start, irq, features) = match id { "virtio-net" => ( openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, - openvmm_defs::microvm::microvm_virtio_net_irq(None)?, + openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ, openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES, ), // The configuration orders virtio-fs devices by slot. @@ -806,8 +806,7 @@ pub(super) fn x86_topology_builder( } } -// The microVM runs on Linux and Windows hosts alone, where its NIC has an -// interrupt. +// The microVM runs on Linux and Windows hosts alone. #[cfg(all(test, any(target_os = "linux", windows)))] mod tests { use super::virtio_mmio_config; @@ -917,24 +916,25 @@ mod tests { ]; let mut sandbox_block_index = 0; let mut filesystem_index = 0; - // Each device in the order of the slots of its kind, with its window - // and the features that the guest sees: never access-platform, which - // the transport offers every device, nor the console size, which the - // virtio-console device offers. - let devices: [(&str, u64, &[u32]); 6] = [ - ("virtio-net", 0xd000_0000, &[5, 32]), - ("virtiofs", 0xd000_1000, &[28, 29, 32]), - ("virtio-console", 0xd000_2000, &[28, 29, 32]), - ("virtio-blk", 0xd000_3000, &[2, 5, 6, 9, 10, 28, 29, 32]), - ("virtio-blk", 0xd000_6000, &[2, 6, 9, 10, 28, 29, 32]), + // Each device in the order of the slots of its kind, with its window, + // its IOAPIC input, and the features that the guest sees: never + // access-platform, which the transport offers every device, nor the + // console size, which the virtio-console device offers. + let devices: [(&str, u64, u32, &[u32]); 6] = [ + ("virtio-net", 0xd000_0000, 16, &[5, 32]), + ("virtiofs", 0xd000_1000, 17, &[28, 29, 32]), + ("virtio-console", 0xd000_2000, 18, &[28, 29, 32]), + ("virtio-blk", 0xd000_3000, 19, &[2, 5, 6, 9, 10, 28, 29, 32]), + ("virtio-blk", 0xd000_6000, 22, &[2, 6, 9, 10, 28, 29, 32]), ( openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID, 0xd000_7000, + 23, &[28, 29, 32], ), ]; - for (id, mmio_base, bits) in devices { - let (start, len, _irq, features, interrupt_mode) = virtio_mmio_config( + for (id, mmio_base, input, bits) in devices { + let (start, len, irq, features, interrupt_mode) = virtio_mmio_config( id, &blocks, &mut sandbox_block_index, @@ -943,6 +943,7 @@ mod tests { ) .unwrap(); assert_eq!(start, mmio_base, "{id}"); + assert_eq!(irq, input, "{id}"); let mut device = VirtioMmioDevice::new_with_disabled_features_and_interrupt_mode( Box::new(EveryFeatureDevice), &driver_source.simple(), diff --git a/openvmm/openvmm_defs/src/microvm.rs b/openvmm/openvmm_defs/src/microvm.rs index a72b4b6274e..e18afed1266 100644 --- a/openvmm/openvmm_defs/src/microvm.rs +++ b/openvmm/openvmm_defs/src/microvm.rs @@ -25,8 +25,12 @@ use vmotherboard::options::BaseChipsetManifest; /// /// Version 3 offers no device `VIRTIO_F_ACCESS_PLATFORM`, since the microVM /// has no IOMMU, and no console `VIRTIO_CONSOLE_F_SIZE`, both of which -/// version 2 offered. A snapshot of version 2 records those feature masks, so -/// restore refuses it by its version, before it compares the devices. +/// version 2 offered. It also routes the IOAPIC's inputs past the 16 ISA IRQs +/// in the MP table and gives each virtio-mmio slot one of them, the same on +/// every backend, where version 2 put each slot on an ISA IRQ and the NIC on +/// another one on WHP. A snapshot of version 2 records those feature masks +/// and interrupts, so restore refuses it by its version, before it compares +/// the devices. pub const MICROVM_ABI_VERSION_3: u32 = 3; /// Linux memory-block granularity used for microVM restore-time expansion. pub const MICROVM_MEMORY_BLOCK_SIZE_BYTES: u64 = 128 * 1024 * 1024; @@ -60,20 +64,23 @@ pub const MICROVM_VIRTIO_CONSOLE_MMIO_BASE: u64 = 0xd000_2000; pub const MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE: u64 = 0xd000_7000; /// Fixed microVM virtio transport window length. pub const MICROVM_VIRTIO_MMIO_LEN: u64 = 0x1000; -/// Fixed distro virtio-blk interrupt. -pub const MICROVM_VIRTIO_BLK_IRQ: u32 = 4; -/// Fixed virtio-blk interrupt for the writable scratch layer. -pub const MICROVM_VIRTIO_SCRATCH_BLK_IRQ: u32 = 11; -/// Fixed microVM virtio-console interrupt. -pub const MICROVM_VIRTIO_CONSOLE_IRQ: u32 = 7; -/// Fixed microVM control virtio-console interrupt. -pub const MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ: u32 = 3; -/// Fixed interrupt of the microVM virtio-fs slot. -pub const MICROVM_VIRTIO_FS_IRQ: u32 = 6; -/// Fixed microVM virtio-net interrupt on KVM. -pub const MICROVM_VIRTIO_NET_KVM_IRQ: u32 = 10; -/// Fixed microVM virtio-net interrupt on WHP. -pub const MICROVM_VIRTIO_NET_WHP_IRQ: u32 = 5; +/// The IOAPIC input of the first virtio-mmio window, the first input past the +/// 16 ISA IRQs, which the PIC does not see: the window at +/// `0xd0000000 + n * 0x1000` interrupts on input `16 + n`, on every backend. +pub const MICROVM_VIRTIO_IRQ_BASE: u32 = 16; +/// Fixed distro virtio-blk interrupt: the input of window 3. +pub const MICROVM_VIRTIO_BLK_IRQ: u32 = 19; +/// Fixed virtio-blk interrupt for the writable scratch layer: the input of +/// window 6. +pub const MICROVM_VIRTIO_SCRATCH_BLK_IRQ: u32 = 22; +/// Fixed microVM virtio-console interrupt: the input of window 2. +pub const MICROVM_VIRTIO_CONSOLE_IRQ: u32 = 18; +/// Fixed microVM control virtio-console interrupt: the input of window 7. +pub const MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ: u32 = 23; +/// Fixed interrupt of the microVM virtio-fs slot: the input of window 1. +pub const MICROVM_VIRTIO_FS_IRQ: u32 = 17; +/// Fixed microVM virtio-net interrupt: the input of window 0. +pub const MICROVM_VIRTIO_NET_IRQ: u32 = 16; // The virtio 1.2 feature bits that the exact microVM feature masks name. The // masks leave out `VIRTIO_F_ACCESS_PLATFORM` (bit 33), which the virtio // transport offers every device, since the microVM has no IOMMU to translate a @@ -141,9 +148,9 @@ pub const MICROVM_HOST_LOOPBACK_INGRESS_COMMAND_LINE_TOKEN: &str = "virtnet_ingr /// /// Each slot keeps its place whether or not the slots before it are present. /// The windows at `0xd0004000` and `0xd0005000`, which held the retired -/// `runtime` and `custom` sandbox layers, stay unassigned, as do IRQs 12 and 9 -/// and the shared-status words at offsets `0x10` and `0x14`: the slots after -/// them kept their places. +/// `runtime` and `custom` sandbox layers, stay unassigned, as do their IOAPIC +/// inputs 20 and 21 and the shared-status words at offsets `0x10` and `0x14`: +/// the slots after them kept their places. pub const MICROVM_VIRTIO_MMIO_BASES: [u64; 6] = [ MICROVM_VIRTIO_NET_MMIO_BASE, MICROVM_VIRTIO_FS_MMIO_BASE, @@ -220,11 +227,26 @@ pub const fn microvm_virtio_status_gpa(mmio_base: u64) -> Option { }; Some(MICROVM_SHARED_STATUS_PAGE_GPA + offset) } + +/// Returns the edge-triggered interrupt, the IOAPIC input, of a fixed +/// virtio-mmio slot. +pub const fn microvm_virtio_irq(mmio_base: u64) -> Option { + Some(match mmio_base { + MICROVM_VIRTIO_NET_MMIO_BASE => MICROVM_VIRTIO_NET_IRQ, + MICROVM_VIRTIO_FS_MMIO_BASE => MICROVM_VIRTIO_FS_IRQ, + MICROVM_VIRTIO_CONSOLE_MMIO_BASE => MICROVM_VIRTIO_CONSOLE_IRQ, + MICROVM_VIRTIO_BLK_MMIO_BASE => MICROVM_VIRTIO_BLK_IRQ, + MICROVM_VIRTIO_SCRATCH_BLK_MMIO_BASE => MICROVM_VIRTIO_SCRATCH_BLK_IRQ, + MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE => MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ, + _ => return None, + }) +} /// The microVM publishes no level-triggered virtio IRQs in its MP table. pub const MICROVM_LEVEL_TRIGGERED_IRQS: [u32; 0] = []; /// The inputs of the microVM's IOAPIC, the generic IOAPIC of its chipset, /// every one of which its MP table routes: the first 16 carry the ISA IRQs, -/// which the PIC also sees, and the IOAPIC alone sees inputs 16 through 23. +/// which the PIC also sees, and the IOAPIC alone sees inputs 16 through 23, +/// which the virtio-mmio slots use. pub const MICROVM_IOAPIC_INPUTS: u8 = chipset_resources::ioapic::IOAPIC_NUM_ENTRIES; /// The stable role of a microVM sandbox block device. @@ -1147,20 +1169,6 @@ impl std::str::FromStr for MicrovmNetworkConfig { } } -/// Returns the pinned virtio-net IRQ for the selected microVM backend. -pub fn microvm_virtio_net_irq(hypervisor_id: Option<&str>) -> anyhow::Result { - match hypervisor_id { - Some("kvm" | "mshv") => Ok(MICROVM_VIRTIO_NET_KVM_IRQ), - Some("whp") => Ok(MICROVM_VIRTIO_NET_WHP_IRQ), - Some(other) => anyhow::bail!("microVM virtio-net does not support hypervisor '{other}'"), - None if cfg!(target_os = "linux") => Ok(MICROVM_VIRTIO_NET_KVM_IRQ), - None if cfg!(windows) => Ok(MICROVM_VIRTIO_NET_WHP_IRQ), - None => { - anyhow::bail!("microVM virtio-net requires an explicit KVM, MSHV, or WHP hypervisor") - } - } -} - fn validate_microvm_virtio_reservations() -> anyhow::Result<()> { let bases = &MICROVM_VIRTIO_MMIO_BASES; for (index, base) in bases.iter().copied().enumerate() { @@ -1193,6 +1201,34 @@ fn validate_microvm_virtio_reservations() -> anyhow::Result<()> { ); previous_status_gpa = Some(status_gpa); } + validate_microvm_virtio_irqs( + bases + .iter() + .map(|&mmio_base| (mmio_base, microvm_virtio_irq(mmio_base))), + ) +} + +/// Checks that each slot of `slots`, a window base and its interrupt in +/// window order, interrupts on an IOAPIC input of its own past the ISA IRQs, +/// which neither the PIC nor a chipset device shares. +fn validate_microvm_virtio_irqs( + slots: impl IntoIterator)>, +) -> anyhow::Result<()> { + let mut previous_irq = None; + for (mmio_base, irq) in slots { + let irq = irq.ok_or_else(|| { + anyhow::anyhow!("microVM virtio MMIO slot {mmio_base:#x} has no interrupt") + })?; + anyhow::ensure!( + (MICROVM_VIRTIO_IRQ_BASE..u32::from(MICROVM_IOAPIC_INPUTS)).contains(&irq), + "microVM interrupt {irq} of slot {mmio_base:#x} is not an IOAPIC input past the ISA IRQs" + ); + anyhow::ensure!( + previous_irq.is_none_or(|previous| previous < irq), + "microVM virtio interrupts are duplicated or out of order" + ); + previous_irq = Some(irq); + } Ok(()) } @@ -1272,14 +1308,13 @@ pub fn microvm_filesystem_slot_count(first_slot: bool, attached: usize) -> anyho /// Appends sandbox virtio devices in fixed-address order. /// /// `filesystem_slot` reserves the virtio-fs slot, and `filesystems`, at most -/// one, occupies it. `network` carries the NIC's identity, its interrupt, the -/// gateway address, if any, that the guest names as its DNS server, and the -/// ingress, if any, for which the guest announces its listeners. +/// one, occupies it. `network` carries the NIC's identity, the gateway +/// address, if any, that the guest names as its DNS server, and the ingress, +/// if any, for which the guest announces its listeners. pub fn append_microvm_virtio_discovery( cmdline: &mut String, network: Option<( &MicrovmNetworkConfig, - u32, Option, Option, )>, @@ -1304,14 +1339,10 @@ pub fn append_microvm_virtio_discovery( validate_microvm_filesystems(filesystems)?; use std::fmt::Write as _; - if let Some((_, irq, _, _)) = network { - anyhow::ensure!( - matches!(irq, MICROVM_VIRTIO_NET_KVM_IRQ | MICROVM_VIRTIO_NET_WHP_IRQ), - "microVM virtio-net IRQ {irq} is not part of the fixed machine contract" - ); + if network.is_some() { write!( cmdline, - " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{irq}" + " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{MICROVM_VIRTIO_NET_IRQ}" )?; } if filesystem_slots > 0 { @@ -1345,7 +1376,7 @@ pub fn append_microvm_virtio_discovery( " virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_CONTROL_CONSOLE_MMIO_BASE:#x}:{MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ} {MICROVM_CONTROL_TTY_COMMAND_LINE}" )?; } - if let Some((network, _, gateway_dns, ingress)) = network { + if let Some((network, gateway_dns, ingress)) = network { write!( cmdline, " {}", @@ -1447,10 +1478,7 @@ fn microvm_command_line_ingress( } } -fn validate_microvm_command_line( - config: &Config, - hypervisor_id: Option<&str>, -) -> anyhow::Result<()> { +fn validate_microvm_command_line(config: &Config) -> anyhow::Result<()> { let MachineProfile::Microvm = config.machine_profile else { unreachable!("microVM command-line validation requires the microVM profile"); }; @@ -1567,9 +1595,8 @@ fn validate_microvm_command_line( let ingress = microvm_command_line_ingress(config.microvm.network.as_ref(), &tokens)?; let mut expected_discovery = Vec::new(); if has_network { - let irq = microvm_virtio_net_irq(hypervisor_id)?; expected_discovery.push(format!( - "virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{irq}" + "virtio_mmio.device={MICROVM_VIRTIO_MMIO_LEN:#x}@{MICROVM_VIRTIO_NET_MMIO_BASE:#x}:{MICROVM_VIRTIO_NET_IRQ}" )); } if filesystem_count > 0 { @@ -1616,15 +1643,27 @@ fn validate_microvm_command_line( ); } } - if !expected_discovery.is_empty() { - anyhow::ensure!( - tokens.ends_with( - &expected_discovery - .iter() - .map(String::as_str) - .collect::>() - ), - "microVM virtio discovery tokens are not in fixed-address order" + check_microvm_discovery_tokens(&tokens, &expected_discovery) +} + +/// Checks that the command line's `tokens` end with `expected`, the discovery +/// tokens of the machine's devices in fixed-address order, and names the +/// first token that differs, whether it is out of order or places its device +/// elsewhere, such as on the interrupt of another contract. +fn check_microvm_discovery_tokens(tokens: &[&str], expected: &[String]) -> anyhow::Result<()> { + let tail = &tokens[tokens.len().saturating_sub(expected.len())..]; + anyhow::ensure!( + tail.len() == expected.len(), + "microVM command line lacks virtio discovery tokens" + ); + if let Some((found, expected)) = tail + .iter() + .zip(expected) + .find(|(found, expected)| **found != expected.as_str()) + { + anyhow::bail!( + "microVM virtio discovery tokens are not this machine's in fixed-address order: \ + found `{found}` where `{expected}` belongs" ); } Ok(()) @@ -1941,7 +1980,7 @@ pub fn validate_machine_config(config: &Config, hypervisor_id: Option<&str>) -> }; validate_microvm_virtio_reservations()?; - validate_microvm_command_line(config, hypervisor_id)?; + validate_microvm_command_line(config)?; validate_machine_load_mode(config.machine_profile, &config.load_mode)?; anyhow::ensure!( config.microvm.time_abi.is_some(), @@ -2388,7 +2427,7 @@ mod tests { let discover = |cmdline: &mut String| { append_microvm_virtio_discovery( cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, None, None)), + Some((&network, None, None)), false, &[], false, @@ -2567,7 +2606,7 @@ mod tests { let mut cmdline = MICROVM_BASE_COMMAND_LINE.to_owned(); append_microvm_virtio_discovery( &mut cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, dns, Some(ingress))), + Some((&network, dns, Some(ingress))), false, &[], false, @@ -2637,8 +2676,8 @@ mod tests { cmdline, format!( "{MICROVM_BASE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0003000:4 \ - virtio_mmio.device=0x1000@0xd0006000:11" + virtio_mmio.device=0x1000@0xd0003000:19 \ + virtio_mmio.device=0x1000@0xd0006000:22" ) ); } @@ -2651,33 +2690,131 @@ mod tests { cmdline, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {MICROVM_CONTROL_TTY_COMMAND_LINE}" ) ); } #[test] - fn microvm_virtio_irqs_avoid_unused_lines_and_are_edge_triggered() { + fn microvm_virtio_slots_have_their_own_ioapic_inputs() { let irqs = [ - MICROVM_VIRTIO_NET_KVM_IRQ, - MICROVM_VIRTIO_NET_WHP_IRQ, + MICROVM_VIRTIO_NET_IRQ, MICROVM_VIRTIO_FS_IRQ, MICROVM_VIRTIO_CONSOLE_IRQ, MicrovmSandboxBlockRole::Distro.irq(), MicrovmSandboxBlockRole::Scratch.irq(), MICROVM_VIRTIO_CONTROL_CONSOLE_IRQ, ]; - assert_eq!(irqs, [10, 5, 6, 7, 4, 11, 3]); - // The RTC owns IRQ 8. IRQs 12, 9, and 13 of the retired runtime and - // custom layers and second virtio-fs slot stay unused. - for unused in [8, 9, 12, 13] { - assert!(!irqs.contains(&unused), "IRQ {unused} is in use"); + assert_eq!(irqs, [16, 17, 18, 19, 22, 23]); + assert_eq!( + MICROVM_VIRTIO_MMIO_BASES.map(|mmio_base| microvm_virtio_irq(mmio_base).unwrap()), + irqs + ); + // The window at 0xd0000000 + n * 0x1000 interrupts on input 16 + n, + // past the ISA IRQs, which the PIC also sees and the PIT and RTC use. + for mmio_base in MICROVM_VIRTIO_MMIO_BASES { + let window = (mmio_base - MICROVM_VIRTIO_NET_MMIO_BASE) / MICROVM_VIRTIO_MMIO_LEN; + assert_eq!( + microvm_virtio_irq(mmio_base), + Some(MICROVM_VIRTIO_IRQ_BASE + window as u32) + ); } + for (role, irq) in [ + (MicrovmSandboxBlockRole::Distro, 19), + (MicrovmSandboxBlockRole::Scratch, 22), + ] { + assert_eq!(microvm_virtio_irq(role.mmio_base()), Some(irq)); + } + // The windows of the retired runtime and custom layers keep inputs 20 + // and 21 free, and no window past them has an input. + for mmio_base in [0xd000_4000, 0xd000_5000, 0xd000_8000] { + assert_eq!(microvm_virtio_irq(mmio_base), None); + } + assert_eq!( + u32::from(MICROVM_IOAPIC_INPUTS), + MICROVM_VIRTIO_IRQ_BASE + 8 + ); assert!(MICROVM_LEVEL_TRIGGERED_IRQS.is_empty()); } + #[test] + fn microvm_virtio_irq_validation_rejects_collisions() { + validate_microvm_virtio_reservations().unwrap(); + let slot = |window: u64, irq| { + ( + MICROVM_VIRTIO_NET_MMIO_BASE + window * MICROVM_VIRTIO_MMIO_LEN, + Some(irq), + ) + }; + validate_microvm_virtio_irqs([slot(0, 16), slot(1, 17), slot(7, 23)]).unwrap(); + for slots in [ + // Two slots on one input. + vec![slot(0, 16), slot(1, 16)], + // ISA IRQs, which the PIC also sees: the RTC's and a free one. + vec![slot(0, 8)], + vec![slot(0, 15)], + // Past the IOAPIC's last input. + vec![slot(0, 24)], + // No interrupt. + vec![(MICROVM_VIRTIO_NET_MMIO_BASE, None)], + ] { + assert!( + validate_microvm_virtio_irqs(slots.clone()).is_err(), + "{slots:?}" + ); + } + } + + #[test] + fn microvm_discovery_token_mismatch_names_the_token() { + let expected = [ + format!("virtio_mmio.device=0x1000@0xd0000000:{MICROVM_VIRTIO_NET_IRQ}"), + format!("virtio_mmio.device=0x1000@0xd0001000:{MICROVM_VIRTIO_FS_IRQ}"), + ]; + check_microvm_discovery_tokens( + &["quiet", expected[0].as_str(), expected[1].as_str()], + &expected, + ) + .unwrap(); + check_microvm_discovery_tokens(&["quiet"], &[]).unwrap(); + // A token that places its device on another interrupt, such as the + // NIC's ISA IRQ before each slot had an IOAPIC input of its own, is + // named, not only reported out of order. + let error = check_microvm_discovery_tokens( + &[ + "quiet", + "virtio_mmio.device=0x1000@0xd0000000:10", + expected[1].as_str(), + ], + &expected, + ) + .unwrap_err() + .to_string(); + assert!( + error.ends_with( + "found `virtio_mmio.device=0x1000@0xd0000000:10` where \ + `virtio_mmio.device=0x1000@0xd0000000:16` belongs" + ), + "{error}" + ); + let error = check_microvm_discovery_tokens( + &[expected[1].as_str(), expected[0].as_str()], + &expected, + ) + .unwrap_err() + .to_string(); + assert!( + error.ends_with(&format!( + "found `{}` where `{}` belongs", + expected[1], expected[0] + )), + "{error}" + ); + assert!(check_microvm_discovery_tokens(&[expected[1].as_str()], &expected).is_err()); + } + #[test] fn microvm_ioapic_inputs_have_backend_routes() { // The MP table routes every input of the IOAPIC, and every backend @@ -2766,11 +2903,11 @@ mod tests { slot.mmio_base, slot.irq )), - [("fs:microvm0", "microvm", 0xd000_1000, 6)] + [("fs:microvm0", "microvm", 0xd000_1000, 17)] ); assert_eq!( MICROVM_FILESYSTEM_SLOTS[0].discovery_token(), - "virtio_mmio.device=0x1000@0xd0001000:6" + "virtio_mmio.device=0x1000@0xd0001000:17" ); validate_microvm_virtio_reservations().unwrap(); assert!(!MICROVM_VIRTIO_MMIO_BASES.contains(&0xd000_8000)); @@ -3219,7 +3356,7 @@ mod tests { let mut cmdline = MICROVM_CONSOLE_COMMAND_LINE.to_owned(); append_microvm_virtio_discovery( &mut cmdline, - Some((&network, MICROVM_VIRTIO_NET_KVM_IRQ, None, None)), + Some((&network, None, None)), true, std::slice::from_ref(&aggregate), true, @@ -3231,10 +3368,10 @@ mod tests { cmdline, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0000000:10 \ - virtio_mmio.device=0x1000@0xd0001000:6 \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0000000:16 \ + virtio_mmio.device=0x1000@0xd0001000:17 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {MICROVM_CONTROL_TTY_COMMAND_LINE} \ virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 \ virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1 \ @@ -3261,7 +3398,7 @@ mod tests { assert_eq!( cmdline, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:6 \ + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:17 \ virtfs_dir=/workspace virtfs_tag=microvm virtfs_mode=rw" ) ); @@ -3347,8 +3484,8 @@ mod tests { let mut cmdline = format!("{MICROVM_BASE_COMMAND_LINE} nvx_overlay_upper=ramfs"); append_microvm_virtio_discovery(&mut cmdline, None, false, &[], false, false, &[distro]) .unwrap(); - assert!(cmdline.contains("virtio_mmio.device=0x1000@0xd0003000:4")); - assert!(!cmdline.contains("virtio_mmio.device=0x1000@0xd0006000:11")); + assert!(cmdline.contains("virtio_mmio.device=0x1000@0xd0003000:19")); + assert!(!cmdline.contains("virtio_mmio.device=0x1000@0xd0006000:22")); let mut cmdline = format!("{MICROVM_BASE_COMMAND_LINE} nvx_overlay_upper"); assert!( diff --git a/openvmm/openvmm_entry/src/cli_args/microvm.rs b/openvmm/openvmm_entry/src/cli_args/microvm.rs index faf40529131..6ca3490d47d 100644 --- a/openvmm/openvmm_entry/src/cli_args/microvm.rs +++ b/openvmm/openvmm_entry/src/cli_args/microvm.rs @@ -2118,7 +2118,7 @@ mod tests { assert_eq!( with_devices, format!( - "{MICROVM_CONSOLE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0002000:7 virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11" + "{MICROVM_CONSOLE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0002000:18 virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22" ) ); let mut with_control_console = build_microvm_control_command_line(&[], true).unwrap(); @@ -2136,8 +2136,8 @@ mod tests { with_control_console, format!( "{MICROVM_CONSOLE_COMMAND_LINE} \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ {}", openvmm_defs::microvm::MICROVM_CONTROL_TTY_COMMAND_LINE ) @@ -2147,12 +2147,7 @@ mod tests { let mut with_network = build_microvm_command_line(&[], false).unwrap(); append_microvm_virtio_discovery( &mut with_network, - Some(( - &network, - openvmm_defs::microvm::MICROVM_VIRTIO_NET_KVM_IRQ, - None, - None, - )), + Some((&network, None, None)), false, &[], false, @@ -2163,20 +2158,15 @@ mod tests { assert_eq!( with_network, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0000000:10 virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0000000:16 virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" ) ); // The guest names either gateway as its DNS server. for (server, dns) in [("10.0.0.1", "10.0.0.1"), ("fd00::a00:1", "fd00::a00:1")] { - let mut with_whp_network = build_microvm_command_line(&[], false).unwrap(); + let mut with_dns = build_microvm_command_line(&[], false).unwrap(); append_microvm_virtio_discovery( - &mut with_whp_network, - Some(( - &network, - openvmm_defs::microvm::MICROVM_VIRTIO_NET_WHP_IRQ, - Some(server.parse().unwrap()), - None, - )), + &mut with_dns, + Some((&network, Some(server.parse().unwrap()), None)), false, &[], false, @@ -2185,10 +2175,10 @@ mod tests { ) .unwrap(); assert!( - with_whp_network.ends_with(&format!( + with_dns.ends_with(&format!( "virtnet_dns={dns} virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" )), - "{with_whp_network}" + "{with_dns}" ); } @@ -2211,7 +2201,7 @@ mod tests { assert_eq!( with_filesystem, format!( - "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:6 virtfs_dir=/mnt/share virtfs_tag=microvm virtfs_mode=ro" + "{MICROVM_BASE_COMMAND_LINE} virtio_mmio.device=0x1000@0xd0001000:17 virtfs_dir=/mnt/share virtfs_tag=microvm virtfs_mode=ro" ) ); @@ -2236,7 +2226,7 @@ mod tests { "nvx-control-tty=hvc9", "driver_async_probe=virtio_console", "driver-async-probe=virtio_console", - "virtio-mmio.device=0x1000@0xd0007000:3", + "virtio-mmio.device=0x1000@0xd0007000:23", ] { assert!(build_microvm_control_command_line(&[reserved.into()], false).is_err()); assert!(build_microvm_command_line(&[reserved.into()], false).is_ok()); diff --git a/openvmm/openvmm_entry/src/microvm/config.rs b/openvmm/openvmm_entry/src/microvm/config.rs index dc3178f6ff8..3a3e4de7809 100644 --- a/openvmm/openvmm_entry/src/microvm/config.rs +++ b/openvmm/openvmm_entry/src/microvm/config.rs @@ -818,12 +818,6 @@ impl<'a> MicrovmConfigBuilder<'a> { let has_control_console = cfg.virtio_devices.iter().any(|(_, device)| { device.id() == openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID }); - let network_irq = cfg - .microvm - .network - .as_ref() - .map(|_| openvmm_defs::microvm::microvm_virtio_net_irq(requested_hypervisor)) - .transpose()?; let ingress = self .network .as_ref() @@ -832,8 +826,7 @@ impl<'a> MicrovmConfigBuilder<'a> { .microvm .network .as_ref() - .zip(network_irq) - .map(|(network, irq)| (network, irq, self.gateway_dns, ingress)); + .map(|network| (network, self.gateway_dns, ingress)); if let Some(snapshot_tier) = opt.microvm.snapshot_tier { anyhow::ensure!( !cmdline diff --git a/openvmm/openvmm_entry/src/microvm/network.rs b/openvmm/openvmm_entry/src/microvm/network.rs index 3106e9527a3..2c66cc24826 100644 --- a/openvmm/openvmm_entry/src/microvm/network.rs +++ b/openvmm/openvmm_entry/src/microvm/network.rs @@ -271,11 +271,10 @@ pub(crate) mod tests { policy: &net_backend_resources::egress::EgressPolicy, ) -> openvmm_helpers::snapshot::microvm::SnapshotMachineContract { let source_hypervisor = if cfg!(windows) { "whp" } else { "kvm" }; - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor)).unwrap(); let mut command_line = build_microvm_command_line(&[], false).unwrap(); openvmm_defs::microvm::append_microvm_virtio_discovery( &mut command_line, - Some((network, irq, policy.gateway_dns_server(), None)), + Some((network, policy.gateway_dns_server(), None)), false, &[], false, diff --git a/openvmm/openvmm_helpers/src/snapshot/microvm.rs b/openvmm/openvmm_helpers/src/snapshot/microvm.rs index 62af75d279d..b189479e825 100644 --- a/openvmm/openvmm_helpers/src/snapshot/microvm.rs +++ b/openvmm/openvmm_helpers/src/snapshot/microvm.rs @@ -1271,11 +1271,11 @@ pub fn microvm_machine_contract( && attachment.reconnect_timeout_ms == 0, "microVM network attachment has an unsupported endpoint policy" ); - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor))?; let discovery = format!( - "virtio_mmio.device={:#x}@{:#x}:{irq}", + "virtio_mmio.device={:#x}@{:#x}:{}", openvmm_defs::microvm::MICROVM_VIRTIO_MMIO_LEN, openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, + openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ, ); let tokens = effective_command_line .split_ascii_whitespace() @@ -1300,7 +1300,7 @@ pub fn microvm_machine_contract( openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE, openvmm_defs::microvm::MICROVM_VIRTIO_MMIO_LEN, )], - irq: Some(irq), + irq: Some(openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ), transport: "virtio-mmio".to_owned(), feature_banks: feature_banks(openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES), queue_count: 2, @@ -2702,9 +2702,8 @@ mod tests { mode: net_backend_resources::egress::EgressPolicyMode, ) -> SnapshotMachineContract { let egress_policy = network.bind_egress_policy(mode).unwrap(); - let irq = openvmm_defs::microvm::microvm_virtio_net_irq(Some(source_hypervisor)).unwrap(); let command_line = format!( - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0000000:{irq} {}", + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0000000:16 {}", network.command_line_fragment_with_dns(egress_policy.gateway_dns_server()) ); microvm_machine_contract( @@ -2748,7 +2747,7 @@ mod tests { microvm_machine_contract( "whp", MICROVM_BOOT_LAYOUT_VERSION, - "earlycon=xe9 console=hvc1 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0002000:7" + "earlycon=xe9 console=hvc1 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0002000:18" .to_owned(), None, false, @@ -2903,7 +2902,7 @@ mod tests { filesystem: &openvmm_defs::microvm::MicrovmFilesystemConfig, ) -> SnapshotMachineContract { let command_line = format!( - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:6 {}", + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:17 {}", filesystem.command_line_fragment(&openvmm_defs::microvm::MICROVM_FILESYSTEM_SLOTS[0]) ); microvm_machine_contract( @@ -2952,7 +2951,7 @@ mod tests { microvm_machine_contract( "whp", MICROVM_BOOT_LAYOUT_VERSION, - "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:6" + "earlycon=xe9 console=hvc0 reboot=t panic=-1 virtio_mmio.device=0x1000@0xd0001000:17" .to_owned(), None, true, @@ -3133,7 +3132,7 @@ mod tests { ); // The contract requires the slot's discovery token once and the // bootstrap tokens exactly. - let discovery = " virtio_mmio.device=0x1000@0xd0001000:6"; + let discovery = " virtio_mmio.device=0x1000@0xd0001000:17"; for invalid in [ command_line.replace(discovery, ""), command_line.replace(discovery, &format!("{discovery}{discovery}")), @@ -3402,7 +3401,7 @@ mod tests { assert_eq!(console.state_unit_name, "virtio-console-3489669120"); assert_eq!(console.ranges[0].start, 0xd000_2000); assert_eq!(console.ranges[0].length, 0x1000); - assert_eq!(console.irq, Some(7)); + assert_eq!(console.irq, Some(18)); assert_eq!(console.transport, "virtio-mmio"); // Indirect descriptors, event index, and virtio version 1: no console // size and no access-platform. @@ -3429,7 +3428,7 @@ mod tests { assert_eq!(control.state_unit_name, "virtio-control-console-3489689600"); assert_eq!(control.ranges[0].start, 0xd000_7000); assert_eq!(control.ranges[0].length, 0x1000); - assert_eq!(control.irq, Some(3)); + assert_eq!(control.irq, Some(23)); assert_eq!(control.transport, "virtio-mmio"); assert_eq!(control.feature_banks, boot.feature_banks); assert_eq!(control.queue_max_sizes, [256, 256]); @@ -3467,15 +3466,15 @@ mod tests { } #[test] - fn generated_microvm_network_contract_has_backend_specific_fixed_abi() { - for (source_hypervisor, irq) in [("kvm", 10), ("mshv", 10), ("whp", 5)] { + fn generated_microvm_network_contract_has_one_fixed_abi_on_every_backend() { + for source_hypervisor in ["kvm", "mshv", "whp"] { let contract = generated_network_contract(source_hypervisor); let network_device = contract.devices.last().unwrap(); assert_eq!(network_device.stable_id, "net:microvm0"); assert_eq!(network_device.state_unit_name, "virtio-net-3489660928"); assert_eq!(network_device.ranges[0].start, 0xd000_0000); assert_eq!(network_device.ranges[0].length, 0x1000); - assert_eq!(network_device.irq, Some(irq)); + assert_eq!(network_device.irq, Some(16)); assert_eq!(network_device.transport, "virtio-mmio"); assert_eq!(network_device.feature_banks, [0x20, 0x1]); assert_eq!(network_device.queue_count, 2); @@ -3783,7 +3782,7 @@ mod tests { assert_eq!(filesystem_device.state_unit_name, "virtiofs-3489665024"); assert_eq!(filesystem_device.ranges[0].start, 0xd000_1000); assert_eq!(filesystem_device.ranges[0].length, 0x1000); - assert_eq!(filesystem_device.irq, Some(6)); + assert_eq!(filesystem_device.irq, Some(17)); assert_eq!(filesystem_device.transport, "virtio-mmio"); assert_eq!(filesystem_device.feature_banks, [0x3000_0000, 0x0000_0001]); assert_eq!(filesystem_device.queue_count, 2); @@ -3828,7 +3827,7 @@ mod tests { assert_eq!(filesystem.stable_id, "fs:microvm0"); assert_eq!(filesystem.state_unit_name, "virtiofs-3489665024"); assert_eq!(filesystem.ranges[0].start, 0xd000_1000); - assert_eq!(filesystem.irq, Some(6)); + assert_eq!(filesystem.irq, Some(17)); } #[test] @@ -4046,14 +4045,17 @@ mod tests { #[test] fn abi_v2_contracts_are_refused_by_their_version() { // Version 2 offered access-platform on every device but the NIC, and - // the console size, so a console of version 2 records other feature - // banks than one of version 3. Restore refuses such a contract by its - // version, before it compares the devices. + // the console size, and put the console on IRQ 7, so a console of + // version 2 records other feature banks and another interrupt than one + // of version 3. Restore refuses such a contract by its version, before + // it compares the devices. const REFUSAL: &str = "snapshot microVM ABI version 2 is unsupported; this OpenVMM supports version 3"; let mut contract = generated_console_contract(); contract.microvm_abi_version = 2; - contract.devices.last_mut().unwrap().feature_banks = vec![0x3000_0001, 0x0000_0003]; + let console = contract.devices.last_mut().unwrap(); + console.feature_banks = vec![0x3000_0001, 0x0000_0003]; + console.irq = Some(7); let err = validate_supported_microvm_contract(&contract).unwrap_err(); assert_eq!(err.to_string(), REFUSAL); @@ -4747,10 +4749,10 @@ mod tests { "earlycon=xe9 console=hvc1 reboot=t panic=-1 \ nvx_sandbox=1 nvx_config=0xd0010000,65536 nr_cpus=2 \ nvx_snapshot_tier=platform \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0003000:4 \ - virtio_mmio.device=0x1000@0xd0006000:11 \ - virtio_mmio.device=0x1000@0xd0007000:3 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0003000:19 \ + virtio_mmio.device=0x1000@0xd0006000:22 \ + virtio_mmio.device=0x1000@0xd0007000:23 \ nvx_control_tty=hvc2" ); validate_manifest_contents(&manifest).unwrap();