diff --git a/doc/benchmarks.md b/doc/benchmarks.md index f83076301..99c429063 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -36,7 +36,8 @@ New runs emit ABI value 3 and are stored under `microvm-v3` paths, so they cannot collide with the legacy unsuffixed ABI-1 history or with the `microvm-v2` history. ABI 3 began new series because its devices stopped offering `VIRTIO_F_ACCESS_PLATFORM`, so that its guests stopped mapping virtio buffers -through the DMA API. +through the DMA API, and moved from ISA IRQs, which the PIC also sees, to +IOAPIC inputs of their own. The OpenVMM benchmark coordinator is implemented in `scripts/nvx_tools/benchmark.py` and exposed through the supported NVX CLI. Guest payloads live in `scripts/nvx_tools/benchmark_scripts`; the coordinator fills the `.sh.in` templates before use. diff --git a/doc/ci.md b/doc/ci.md index 5fadbdb6a..11da9762b 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -439,6 +439,30 @@ does a documented feature that the guest lacks. A unit test checks that the scenario's devices are those of Tables 2.7 and 2.8, and OpenVMM's unit tests check the exact feature mask that each device offers. +The `interrupt-map` scenario covers the interrupt map of +[2.4.2](design/2.4-interrupt-and-io-architecture.md#242-interrupt-routing) and +[Table 2.7](design/2.5-device-topology-transport-and-discovery.md). It boots a +guest with every fixed virtio device: the NIC, the host filesystem with a +share, both consoles, and both sandbox blocks. Once a driver has bound each of +the six, the guest reports, for each, the base of its MMIO window and the +interrupt that its driver requested, as `/sys/kernel/irq` describes it: its +controller and input, its trigger, its handlers, and how many times it fired. +It then drives each device, pinging the gateway, reading the share, writing to +both consoles, reading the `distro` block and writing the `scratch` block past +the page cache, and reports again. Each device must interrupt on the IOAPIC +input of its window, 16 through 23, edge-triggered and with no other handler, +its interrupt must have fired, and `/proc/interrupts` must count no APIC error +or spurious PIC interrupt (`ERR`), no level-triggered interrupt that the IOAPIC +misrouted (`MIS`), and no spurious APIC interrupt (`SPU`). The guest also +reports how many of each device's interrupts lately found no work for any +handler (`unhandled` in `/proc/irq//spurious`), which the scenario does not +check: a console's driver polls for its output's completion, so nearly every +interrupt of a console finds no work, and the NIC's polling can leave one such +interrupt too. A guest with the NIC and the share then captures a snapshot, +whose restore must report the same interrupts and fire them again. A unit test +checks that the scenario's interrupts are those of Table 2.7, and OpenVMM's +unit tests check the MP table's routing and each slot's input. + Shared guest artifacts are built with Docker on a GitHub-hosted Ubuntu runner. The kernel, Alpine initramfs, Ubuntu initramfs, and Ubuntu EROFS layer use separate cache keys. The EROFS layer's entry also holds the sandbox scratch diff --git a/doc/design/1.5-compatibility-and-versioning-model.md b/doc/design/1.5-compatibility-and-versioning-model.md index 27b42cc65..01097f645 100644 --- a/doc/design/1.5-compatibility-and-versioning-model.md +++ b/doc/design/1.5-compatibility-and-versioning-model.md @@ -24,7 +24,7 @@ contract carries its own version or identity, independent of the others. A change to a guest-visible address, interrupt, command-line token, feature mask, queue shape, time rule, or device behavior requires a new contract version. A backend-specific difference is valid only when the versioned -contract states it explicitly, as it does for the virtio-net interrupt. +contract states it explicitly. Guest-visible configuration is immutable for the lifetime represented by a snapshot. Restore derives the machine from the snapshot manifest; caller input diff --git a/doc/design/2.4-interrupt-and-io-architecture.md b/doc/design/2.4-interrupt-and-io-architecture.md index 5aa66575a..9f0a7230c 100644 --- a/doc/design/2.4-interrupt-and-io-architecture.md +++ b/doc/design/2.4-interrupt-and-io-architecture.md @@ -3,12 +3,13 @@ [Architecture documentation](README.md) > [2. Virtual Machine Architecture](2-virtual-machine-architecture.md) -As Figure 2.2 shows, interrupts and I/O follow separate paths. Devices signal -interrupts on interrupt request lines (IRQs) of the PIC and IOAPIC, which -deliver them to each processor's local advanced programmable interrupt -controller (LAPIC). The LAPIC also provides a timer. Processors reach devices -through port I/O and MMIO, while bulk data and interrupt status move through -guest RAM. +As Figure 2.2 shows, interrupts and I/O follow separate paths. The PIT and RTC +signal interrupts on interrupt request lines (IRQs) of the ISA bus, which both +the PIC and the IOAPIC see, and the virtio devices signal theirs on IOAPIC +inputs that the PIC does not see. The controllers deliver interrupts to each +processor's local advanced programmable interrupt controller (LAPIC). The +LAPIC also provides a timer. Processors reach devices through port I/O and +MMIO, while bulk data and interrupt status move through guest RAM. ```mermaid flowchart TB @@ -17,15 +18,18 @@ flowchart TB Virtio["Fixed virtio-mmio slots"] Timers["PIT and RTC"] Ram["Guest RAM
interrupt-status page
and descriptor chains"] - Controllers["PIC and IOAPIC
16 ISA interrupt lines"] + Pic["PIC
16 ISA IRQs"] + Ioapic["IOAPIC
24 inputs"] Vcpu -->|"port I/O"| Ports Vcpu -->|"MMIO configuration
and notification"| Virtio Vcpu <-->|"descriptors and
status words"| Ram Virtio <-->|"bulk data and
status bits"| Ram - Virtio -->|"dedicated edge IRQs"| Controllers - Timers -->|"IRQ 0 and IRQ 8"| Controllers - Controllers -->|"interrupts"| Vcpu + Virtio -->|"dedicated edge IRQs
16 to 23"| Ioapic + Timers -->|"IRQ 0 and IRQ 8"| Pic + Timers -->|"IRQ 0 and IRQ 8"| Ioapic + Pic -->|"interrupts"| Vcpu + Ioapic -->|"interrupts"| Vcpu ``` **Figure 2.2.** Interrupt and I/O paths. @@ -34,7 +38,7 @@ flowchart TB The base chipset contains exactly this allowlist: -- A generic PIC and IOAPIC. +- A generic PIC and a generic IOAPIC of 24 inputs. - The selected backend's LAPIC. - An i8253 PIT on IRQ 0. - A generic programmable CMOS RTC anchored to UTC. @@ -56,11 +60,26 @@ behavior of the LAPIC timer, PIT, and RTC. ## 2.4.2 Interrupt Routing -The MP table describes processors, the ISA bus, the IOAPIC, and legacy IRQ -routing, and it routes only the 16 ISA interrupts. The machine profile uses no -ACPI interrupt tables. Dedicated virtio interrupts use active-high edge -delivery. No virtio device uses the RTC's exclusive IRQ 8, and no device uses -IRQ 9, 12, or 13 on any backend. +The MP table describes processors, the ISA bus, the IOAPIC, and the routing of +every one of the IOAPIC's 24 inputs. The machine profile uses no ACPI interrupt +tables. The interrupt map is the same on every backend: + +- Inputs 0 through 15 carry the 16 ISA IRQs, which the PIC also sees. The + PIT's IRQ 0 reaches input 2, IRQ 2 cascades the PIC, and the RTC's IRQ 8 is + its own. ISA IRQs 1, 3 through 7, and 9 through 15 are routed but unused. +- Inputs 16 through 23 have no ISA IRQ and no PIC input. The MP table names + each one as the active-high, edge-triggered interrupt of its own number, so + the guest reaches them only through the IOAPIC. +- Each fixed virtio-mmio slot has an input of its own: the slot whose window + starts n windows of 4 KiB past `0xd0000000` interrupts on input 16 + n, as + [Table 2.7](2.5-device-topology-transport-and-discovery.md) lists. Inputs 20 + and 21, of the unused windows 4 and 5, are free. + +Dedicated virtio interrupts use active-high edge delivery, and no two devices +share an interrupt. The 24 inputs bound the map: a window past window 7 has no +input of its own, so a further fixed device takes window 4 or 5 with its input +20 or 21, an ISA IRQ that the PIC also sees, or an IOAPIC with more inputs, +each in a new contract version. ## 2.4.3 Port I/O diff --git a/doc/design/2.5-device-topology-transport-and-discovery.md b/doc/design/2.5-device-topology-transport-and-discovery.md index 7ff94a660..9e4a9139a 100644 --- a/doc/design/2.5-device-topology-transport-and-discovery.md +++ b/doc/design/2.5-device-topology-transport-and-discovery.md @@ -12,17 +12,20 @@ are validated before host resources are resolved. | Device | Stable identity | MMIO range | IRQ | Presence | | --- | --- | ---: | ---: | --- | -| Network | `net:microvm0` | `0xd0000000..0xd0000fff` | KVM/MSHV 10, WHP 5 | Optional | -| Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 6 | Reserved slot; attachment optional | -| Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 7 | Optional | -| `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 4 | Optional, read-only | -| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Optional, writable final role; absent in the RAM-backed overlay layout | -| Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 3 | Optional | +| Network | `net:microvm0` | `0xd0000000..0xd0000fff` | 16 | Optional | +| Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 17 | Reserved slot; attachment optional | +| Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 18 | Optional | +| `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 19 | Optional, read-only | +| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 22 | Optional, writable final role; absent in the RAM-backed overlay layout | +| Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 23 | Optional | Placement is explicit rather than allocated sequentially, so an omitted device -never shifts another. All six slots are reserved, and the windows -`0xd0004000..0xd0005fff` and `0xd0008000..0xd0008fff`, IRQs 9, 12, and 13, -and the status words at `0x30010`, `0x30014`, and `0x30020` are unused. +never shifts another. The slot of window n, at `0xd0000000` plus n times +`0x1000`, interrupts on IOAPIC input 16 + n, on every backend, and owns the +status word at `0x30000` plus 4n. All six slots are reserved, and windows 4 +and 5, `0xd0004000..0xd0005fff`, with inputs 20 and 21 and the status words at +`0x30010` and `0x30014`, and the window `0xd0008000..0xd0008fff` and status +word `0x30020` are unused; the IOAPIC has no input for a window past window 7. **Table 2.8.** Guest-visible virtio features. diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 8ab61229f..ed10a7501 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -35,9 +35,10 @@ flowchart TB The NIC occupies the fixed network slot, has one RX/TX queue pair, and advertises an exact feature mask: the MAC-address feature and virtio -version 1 ([Table 2.8](2.5-device-topology-transport-and-discovery.md)). Its -interrupt is the one backend-specific value in the machine contract: IRQ 10 on -KVM and MSHV, and IRQ 5 on WHP. The portable network profile is mandatory and +version 1 ([Table 2.8](2.5-device-topology-transport-and-discovery.md)). It +interrupts on IOAPIC input 16 on every backend +([2.4.2](2.4-interrupt-and-io-architecture.md#242-interrupt-routing)). The +portable network profile is mandatory and selects the same in-process data plane on every backend. Table 3.12 shows an example of the static network identity that the host owns. diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index 91507deaa..8334ab6b3 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -18,8 +18,9 @@ reservations, and guest addresses as untrusted. It: 2. Loads validated loadable segments at their physical addresses and zeroes their BSS tails. 3. Places an optional initramfs at the first 2-MiB boundary after the kernel. -4. Builds the Intel MP 1.4 processor, ISA bus, IOAPIC, and legacy interrupt - entries. +4. Builds the Intel MP 1.4 processor, ISA bus, and IOAPIC entries, and an + interrupt entry for every IOAPIC input: the ISA IRQs and inputs 16 through + 23. 5. Builds Linux `boot_params` with the canonical e820 RAM and reservation map. 6. Installs a bootstrap GDT and 4-GiB identity page tables. 7. Enters the kernel in the 64-bit execution mode of x86-64 (long mode) with diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 736de183f..bec8927a2 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -9,16 +9,17 @@ - Mapped fork tip: `82686c983907738d44a6eb9e3322589b573c0074`, position 144. The graph, PR summary, and commit mapping below cover positions 001-144 and the PR20 fixup at position 161. -- NVX's pinned OpenVMM revision: `b9b1f8f6d41bcb7cc9b2d1ec7c836b30f5dc7718`, position 203. Positions - 202 and 203, which microsoft/openvmm#4840 proposes for `feature/microvm/dev`, are two commits on - position 201, `d891dfd9d`, which microsoft/openvmm#4839 proposes, one commit on position 200, +- NVX's pinned OpenVMM revision: `35e76265fa8b63922edead112fcd5a38c59445c8`, position 205. Positions + 204 and 205, which microsoft/openvmm#4842 proposes for `feature/microvm/dev`, are two commits on + position 203, `b9b1f8f6d`, which microsoft/openvmm#4840 proposes as two commits on position 201, + `d891dfd9d`, which microsoft/openvmm#4839 proposes, one commit on position 200, `0ac254109`, the head of the `feature/microvm/release/v3.0` branch of microsoft/openvmm, to which `feature/microvm/dev` also points. The v3.0 reorganization regrouped the 385 commits of `feature/microvm/release/v2.4` that upstream had not landed into 200 commits: it folded fixups into the commits that they fix and combined the commits after the mapped tip into commits by concept, without changing what the chain adds to upstream; see - [The v3.0 reorganization](#the-v30-reorganization). The pin is 59 commits past the mapped tip - on the same first-parent chain. Positions 145-203, except the PR20 fixup at position + [The v3.0 reorganization](#the-v30-reorganization). The pin is 61 commits past the mapped tip + on the same first-parent chain. Positions 145-205, except the PR20 fixup at position 161, are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `b2676222b383bf50c7022f6a5983920421469a47` (2026-10-31), the fork base. @@ -516,7 +517,8 @@ b2676222b -> network [195] -> device DMA [196-197] -> command lines [198] -> KVP test [199-200] -> sandbox block roles [201] -> virtio feature masks [202-203] - -> b9b1f8f6d (NVX pin) + -> IOAPIC interrupt lines [204-205] + -> 35e76265f (NVX pin) ``` For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main, @@ -532,10 +534,10 @@ see [The v3.0 reorganization](#the-v30-reorganization). ## Commits after the mapped tip -The NVX pin extends the chain past position 144 without merges. Of the 59 later +The NVX pin extends the chain past position 144 without merges. Of the 61 later commits, position 161 is a PR20 fixup, and the others are not yet classified or assigned to PRs. Each of positions 145-200 combines the v2.4 commits that the table lists, which developed one -concept; "(part)" marks a v2.4 fixup that split across commits. Positions 201-203 follow the +concept; "(part)" marks a v2.4 fixup that split across commits. Positions 201-205 follow the reorganization, so they combine none. | Position | Commit | v2.4 positions | Topic | @@ -599,6 +601,8 @@ reorganization, so they combine none. | 201 | `d891dfd9d` | - | microVM: remove the runtime and custom sandbox block roles (microsoft/openvmm#4839) | | 202 | `b0dda7fea` | - | microVM: give every virtio device class an exact feature mask (microsoft/openvmm#4840) | | 203 | `b9b1f8f6d` | - | microVM: drop access-platform and the console size in microVM ABI 3 (microsoft/openvmm#4840) | +| 204 | `e13c1a644` | - | microVM: route every IOAPIC input in the MP table (microsoft/openvmm#4842) | +| 205 | `35e76265f` | - | microVM: give each virtio slot its own IOAPIC input in ABI 3 (microsoft/openvmm#4842) | Some of these commits change mapped work rather than add to it. Position 160 removes the legacy microVM clock paths, changing files that PR10 and PR17 also change, and makes manifest @@ -615,7 +619,14 @@ names and layer counts of PR19's position 105's snapshot capture and restore. Po fall-through among them, and that PR18's positions 095, 096, and 100, PR19's position 104, and PR23's position 129 give the NIC, the filesystem, the sandbox blocks, and the consoles and record in the snapshot contract. Position 203 also moves the microVM ABI version of PR17's position 087 -to 3, which PR20's position 108 and PR25's position 137 check. Assigning the later +to 3, which PR20's position 108 and PR25's position 137 check. Position 204 extends the MP table +builder of PR14's position 065, which microsoft/openvmm#4554 proposes upstream, and its tests and +those of PR14's positions 066 and 067, to route every IOAPIC input, and passes the microVM's 24 +inputs from the loader of PR15's position 075. Position 205 moves every virtio slot from the ISA +IRQ that PR16's position 080 gives it, and the NIC from the backend-specific IRQ that PR18's +position 096 selects, to the IOAPIC input of its window, which changes the interrupts and +discovery tokens that PR17's position 087 records in the snapshot contract and that the slots of +PR18, PR19, and PR23 use. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 191, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -646,6 +657,11 @@ does not apply to upstream main and follows those PRs upstream. Positions 202 an change only the microVM profile, its worker's device configuration, its snapshot contract, and its management RPC profile, which PR16-PR20, PR23, and PR25 add, and their Guide pages: replaying them on `b2676222b` conflicts in all nine of their files, so they follow those PRs upstream. +Positions 204 and 205 change the MP-table loader of PR14 and the microVM profile, worker, CLI, and +snapshot contract of PR15-PR19 and PR23, and their Guide pages: replaying them on `b2676222b` +conflicts in 16 of their 17 files, so they follow those PRs upstream. The 17th, the IOAPIC's +`chipset` tests, which position 204 extends with a save and restore of the inputs past the ISA +IRQs, applies to upstream main without conflicts. ## PR summary diff --git a/doc/run.md b/doc/run.md index 52f8494c5..d92b2d4ef 100644 --- a/doc/run.md +++ b/doc/run.md @@ -660,6 +660,10 @@ run retired ABI-1 guests or snapshots. ABI 3 offers no virtio device `VIRTIO_F_ACCESS_PLATFORM` and no console its size, which ABI 2 offered (see [Guest-Visible Capabilities](design/3.4-block-storage.md#342-guest-visible-capabilities)), +and gives each virtio device an IOAPIC input of its own past the ISA IRQs, the +same on every backend, where ABI 2 used ISA IRQs and another one for the NIC +on WHP (see +[Interrupt Routing](design/2.4-interrupt-and-io-architecture.md#242-interrupt-routing)), so OpenVMM refuses to restore a snapshot of ABI 2, with `snapshot microVM ABI version 2 is unsupported; this OpenVMM supports version 3`, and `nvx-rs` refuses a template of ABI 2. Capture them again; to restore them as they are, diff --git a/doc/usage.md b/doc/usage.md index bfa018b21..46c8f4292 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1074,8 +1074,8 @@ manifest, and the sandbox scratch template, which that selects one of them fails before its first scenario when one of those files is missing. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject -the Alpine-control-only `sandbox-blocks`, `scratch-snapshot`, and -`virtio-features` scenarios and +the Alpine-control-only `interrupt-map`, `sandbox-blocks`, `scratch-snapshot`, +and `virtio-features` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, `platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, `platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, @@ -1123,7 +1123,16 @@ The guest reports, for each device, the base of its MMIO window, its virtio device ID, and the features that its driver negotiated, and the scenario requires exactly the devices and features of [Table 2.8](design/2.5-device-topology-transport-and-discovery.md), naming each -bit that a device has besides its documented ones or lacks. The scenarios that +bit that a device has besides its documented ones or lacks. The +`interrupt-map` scenario boots the same devices, with a share in the host +filesystem slot, and reports the interrupt that each device's driver +requested before and after it drives traffic through every device; each must +be the IOAPIC input of its device's window that +[Table 2.7](design/2.5-device-topology-transport-and-discovery.md) lists, +edge-triggered, unshared, and firing, and `/proc/interrupts` must count no APIC +error, misrouted IOAPIC interrupt, or spurious interrupt. A snapshot of a guest +with the NIC and the share must then restore with the same interrupts, which +fire again. The scenarios that run microVMs or sandboxes as the public commands do, or that compile egress policies, run [`nvx-rs`](#nvx-rs). The command requires `build/vmlinux` (with `--debug-kernel`, `build/vmlinux-debug` diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index e09e6cf3e..ea1d46632 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -610,7 +610,7 @@ trap 'exit 143' TERM has_scratch=false case " $(cat /proc/cmdline) " in - *" virtio_mmio.device=0x1000@0xd0006000:11 "*) has_scratch=true ;; + *" virtio_mmio.device=0x1000@0xd0006000:22 "*) has_scratch=true ;; esac scratch_mounted=false mountpoint -q "$scratch_mount" && scratch_mounted=true @@ -631,7 +631,7 @@ if [ "$has_scratch" = true ]; then exit 1 fi elif [ "$scratch_policy" = fresh ]; then - echo "nvx-snapshot: fresh-scratch capture requires an ABI-v2 scratch device" >&2 + echo "nvx-snapshot: fresh-scratch capture requires an ABI-v3 scratch device" >&2 exit 1 fi diff --git a/nvx-rs/src/contract.rs b/nvx-rs/src/contract.rs index 85048f2f9..e7c820329 100644 --- a/nvx-rs/src/contract.rs +++ b/nvx-rs/src/contract.rs @@ -54,7 +54,8 @@ /// OpenVMM microVM ABI version this crate drives. /// /// Version 3 offers no device `VIRTIO_F_ACCESS_PLATFORM` and no console `VIRTIO_CONSOLE_F_SIZE`, -/// which version 2 offered, so OpenVMM and this crate refuse a template of version 2. +/// which version 2 offered, and gives each virtio device an IOAPIC input of its own past the ISA +/// IRQs, where version 2 used ISA IRQs, so OpenVMM and this crate refuse a template of version 2. pub const MICROVM_ABI_VERSION: u64 = 3; /// Version of the boot contract this crate speaks; see the [module](self) for when it changes. /// diff --git a/nvx-rs/src/host/cli/run/tests.rs b/nvx-rs/src/host/cli/run/tests.rs index 212bbb108..2aa8f99b5 100644 --- a/nvx-rs/src/host/cli/run/tests.rs +++ b/nvx-rs/src/host/cli/run/tests.rs @@ -15,7 +15,7 @@ use crate::protocol::cmdline::{self, parameters, virtio_mmio_devices}; fn effective_command_line(cmdline: &str) -> String { format!( "earlycon=xe9 console=hvc1 reboot=t panic=-1 {cmdline} \ - virtio_mmio.device=0x1000@0xd0001000:6 virtio_mmio.device=0x1000@0xd0002000:7{}", + virtio_mmio.device=0x1000@0xd0001000:17 virtio_mmio.device=0x1000@0xd0002000:18{}", aggregate_command_line_fragment(&[MountAccess::ReadWrite]) ) } diff --git a/nvx-rs/src/host/sandbox/plan/tests.rs b/nvx-rs/src/host/sandbox/plan/tests.rs index c92b9a169..da35e54d6 100644 --- a/nvx-rs/src/host/sandbox/plan/tests.rs +++ b/nvx-rs/src/host/sandbox/plan/tests.rs @@ -229,9 +229,9 @@ fn the_guest_reads_the_root_of_every_plan_from_its_command_line() { // OpenVMM appends its own tokens, the discovery of the plan's block devices among // them, which the guest reads alike. let mut effective = - format!("{command_line} nr_cpus=1 virtio_mmio.device=0x1000@0xd0003000:4"); + format!("{command_line} nr_cpus=1 virtio_mmio.device=0x1000@0xd0003000:19"); if upper.scratch_path().is_some() { - effective.push_str(" virtio_mmio.device=0x1000@0xd0006000:11"); + effective.push_str(" virtio_mmio.device=0x1000@0xd0006000:22"); } effective.push_str(&fragment(&plan)); assert_eq!( @@ -253,9 +253,9 @@ fn plans_refuse_the_command_lines_that_the_guest_refuses() { for image in images() { let plan = plan(image, SandboxUpper::RAM, SandboxSettings::default()); for user in [ - "virtio_mmio.device=0x1000@0xd0006000:11", + "virtio_mmio.device=0x1000@0xd0006000:22", "quiet virtio-mmio.device=4K@0xd0006000:11:3", - "\"virtio_mmio.device=0x1000@0xd0006000:11\"", + "\"virtio_mmio.device=0x1000@0xd0006000:22\"", "virtio_mmio.device=\"0x1000@ 0xd0006000:11\"", "virtio_mmio.device=0x1000@3489685504:11", ] { @@ -268,8 +268,8 @@ fn plans_refuse_the_command_lines_that_the_guest_refuses() { "virtio_mmio.device=0x1000@0xd0006000:0", "virtio_mmio.device=0x1000@0xd0006000", // A device elsewhere, and a token that the kernel reads inside another value. - "virtio_mmio.device=0x1000@0xd0007000:3", - "x=\"y virtio_mmio.device=0x1000@0xd0006000:11\"", + "virtio_mmio.device=0x1000@0xd0007000:23", + "x=\"y virtio_mmio.device=0x1000@0xd0006000:22\"", ] { let command_line = plan.kernel_command_line(user).unwrap(); assert_eq!( @@ -1440,7 +1440,7 @@ fn adapter_plans_refuse_what_their_root_and_openvmm_own() { ); // The guest must read the plan's root from the command line. assert_eq!( - message(plan.kernel_command_line("virtio_mmio.device=0x1000@0xd0006000:11")), + message(plan.kernel_command_line("virtio_mmio.device=0x1000@0xd0006000:22")), "the kernel command line of an adapter's plan conflicts with the boot tokens of its root: \ a RAM upper takes no scratch block device, but the command line adds a virtio-mmio \ device at 0xd0006000" diff --git a/nvx-rs/src/host/template/key/tests.rs b/nvx-rs/src/host/template/key/tests.rs index 74448b42e..70764df38 100644 --- a/nvx-rs/src/host/template/key/tests.rs +++ b/nvx-rs/src/host/template/key/tests.rs @@ -967,8 +967,9 @@ fn vector_class(name: &str) -> CaptureClass { } /// Returns the manifest of the capture that the vectors call `name`, as microVM ABI 2 captured it: -/// every virtio device but the NIC also offered access-platform (bit 33), and each console also -/// offered its size (bit 0). +/// every virtio device but the NIC also offered access-platform (bit 33), each console also +/// offered its size (bit 0), and each virtio device interrupted on an ISA IRQ, which its discovery +/// token named. fn vector_spec(name: &str) -> ManifestSpec { let mut spec = match name { "TEMPLATE" => ManifestSpec::template(), @@ -980,12 +981,35 @@ fn vector_spec(name: &str) -> ManifestSpec { } _ => panic!("no capture is named {name}"), }; - for device in &mut spec.contract().devices { - if device.transport == "virtio-mmio" && device.kind != "virtio-net" { - device.feature_banks[1] |= 1 << (33 - 32); - if device.kind.ends_with("console") { - device.feature_banks[0] |= 1; + // Each virtio device's MMIO base, interrupt in ABI 3, and ISA IRQ in ABI 2, on KVM. + const ISA_IRQS: [(u64, u32, u32); 6] = [ + (0xd000_0000, 16, 10), + (0xd000_1000, 17, 6), + (0xd000_2000, 18, 7), + (0xd000_3000, 19, 4), + (0xd000_6000, 22, 11), + (0xd000_7000, 23, 3), + ]; + let contract = spec.contract(); + for device in &mut contract.devices { + if device.transport == "virtio-mmio" { + if device.kind != "virtio-net" { + device.feature_banks[1] |= 1 << (33 - 32); + if device.kind.ends_with("console") { + device.feature_banks[0] |= 1; + } } + let (base, irq, isa_irq) = ISA_IRQS + .into_iter() + .find(|(base, ..)| device.ranges[0].1 == *base) + .expect("a virtio device has a fixed slot"); + assert_eq!(device.irq, Some(irq), "{base:#x}"); + device.irq = Some(isa_irq); + contract.command_line = contract.command_line.replacen( + &format!("virtio_mmio.device=0x1000@{base:#x}:{irq}"), + &format!("virtio_mmio.device=0x1000@{base:#x}:{isa_irq}"), + 1, + ); } } spec diff --git a/nvx-rs/src/host/template/manifest/tests.rs b/nvx-rs/src/host/template/manifest/tests.rs index 85d201f89..1dbc63d67 100644 --- a/nvx-rs/src/host/template/manifest/tests.rs +++ b/nvx-rs/src/host/template/manifest/tests.rs @@ -188,7 +188,7 @@ fn a_templates_manifest_is_read_whole() { start: 0xd000_3000, length: 0x1000, }], - irq: Some(4), + irq: Some(19), feature_banks: vec![0x3000_0664, 1], queue_count: 1, queue_max_sizes: vec![256], @@ -289,8 +289,8 @@ fn other_versions_of_the_format_are_unsupported() { unsupported, "describes a machine of profile \"standard\"", ); - // A template of microVM ABI version 2, whose devices offered access-platform and whose - // consoles offered their size, is refused, as are the versions around it. + // A template of microVM ABI version 2, whose devices offered access-platform and used ISA + // IRQs and whose consoles offered their size, is refused, as are the versions around it. for version in [1, 2, 4] { assert_refused( |spec| spec.contract().abi_version = version, diff --git a/nvx-rs/src/host/template/plan/tests.rs b/nvx-rs/src/host/template/plan/tests.rs index 369f99186..f0cc4ef5f 100644 --- a/nvx-rs/src/host/template/plan/tests.rs +++ b/nvx-rs/src/host/template/plan/tests.rs @@ -535,7 +535,7 @@ fn openvmms_platform_allowlist_admits_its_tokens_and_refuses_tenant_data() { "nr_cpus=8", "maxcpus=1", "nvx_control_tty=hvc2", - "virtio_mmio.device=0x1000@0xd0001000:6", + "virtio_mmio.device=0x1000@0xd0001000:17", "virtnet_ip=10.0.0.2", "virtnet_mask=255.255.255.0", "virtnet_gw=10.0.0.1", diff --git a/nvx-rs/src/host/template/tests.rs b/nvx-rs/src/host/template/tests.rs index 2cea241b6..3f5e50ede 100644 --- a/nvx-rs/src/host/template/tests.rs +++ b/nvx-rs/src/host/template/tests.rs @@ -605,11 +605,11 @@ pub(super) struct ManifestSpec { /// The kernel command line of the template of [`ManifestSpec::template`]. const COMMAND_LINE: &str = "earlycon=xe9 console=hvc1 reboot=t panic=-1 nvx_sandbox=1 \ nvx_config=0xd0010000,65536 nr_cpus=2 nvx_snapshot_tier=platform \ - virtio_mmio.device=0x1000@0xd0001000:6 \ - virtio_mmio.device=0x1000@0xd0002000:7 \ - virtio_mmio.device=0x1000@0xd0003000:4 \ - virtio_mmio.device=0x1000@0xd0006000:11 \ - virtio_mmio.device=0x1000@0xd0007000:3 nvx_control_tty=hvc2"; + virtio_mmio.device=0x1000@0xd0001000:17 \ + virtio_mmio.device=0x1000@0xd0002000:18 \ + virtio_mmio.device=0x1000@0xd0003000:19 \ + virtio_mmio.device=0x1000@0xd0006000:22 \ + virtio_mmio.device=0x1000@0xd0007000:23 nvx_control_tty=hvc2"; impl ManifestSpec { /// Returns the manifest of a template: a KVM microVM of 2 processors, all online at boot, and @@ -679,7 +679,7 @@ impl ManifestSpec { "virtio-fs", "virtiofs", 0xd000_1000, - 6, + 17, 0x1_3000_0000, 2, ), @@ -688,7 +688,7 @@ impl ManifestSpec { "virtio-console", "virtio-console", 0xd000_2000, - 7, + 18, 0x1_3000_0000, 2, ), @@ -697,7 +697,7 @@ impl ManifestSpec { "virtio-blk", "virtio-blk", 0xd000_3000, - 4, + 19, 0x1_3000_0664, 1, ), @@ -706,7 +706,7 @@ impl ManifestSpec { "virtio-blk", "virtio-blk", 0xd000_6000, - 11, + 22, 0x1_3000_0644, 1, ), @@ -715,7 +715,7 @@ impl ManifestSpec { "virtio-control-console", "virtio-control-console", 0xd000_7000, - 3, + 23, 0x1_3000_0000, 2, ), @@ -832,7 +832,7 @@ impl ManifestSpec { "virtio-net", "virtio-net", 0xd000_0000, - 10, + 16, 0x1_0000_0020, 2, ), @@ -842,8 +842,8 @@ impl ManifestSpec { .state_units .push("virtio-net-3489660928".to_owned()); contract.command_line = contract.command_line.replacen( - "virtio_mmio.device=0x1000@0xd0001000:6", - "virtio_mmio.device=0x1000@0xd0000000:10 virtio_mmio.device=0x1000@0xd0001000:6", + "virtio_mmio.device=0x1000@0xd0001000:17", + "virtio_mmio.device=0x1000@0xd0000000:16 virtio_mmio.device=0x1000@0xd0001000:17", 1, ) + " virtnet_ip=10.0.0.2 virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 \ virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1"; diff --git a/nvx-rs/tests/fixtures/export-tokens.json b/nvx-rs/tests/fixtures/export-tokens.json index ca865c0bf..f96043418 100644 --- a/nvx-rs/tests/fixtures/export-tokens.json +++ b/nvx-rs/tests/fixtures/export-tokens.json @@ -3,8 +3,8 @@ "cases": [ {"name": "a read-write export", "command_line": "virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "rw"}}, {"name": "a read-only export", "command_line": "virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=ro virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "ro"}}, - {"name": "the tokens of a managed sandbox and of the VMM", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_entrypoint=/bin/sh nvx_hostname=nvx-sandbox nr_cpus=1 nvx_workload_uid=65534 nvx_workload_gid=65534 nvx_lifecycle=managed nvx_maps=2 virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11 virtio_mmio.device=0x1000@0xd0007000:3 nvx_control_tty=hvc2 virtio_mmio.device=0x1000@0xd0001000:6 virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "rw"}}, - {"name": "the tokens of an adapter's layered root and of the VMM", "command_line": "console=hvc0 nvx_layer=distro,0xd0003000,gpt:fedcba98-7654-3210-fedc-ba9876543210 nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0001000:6 virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=ro virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "ro"}}, + {"name": "the tokens of a managed sandbox and of the VMM", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_entrypoint=/bin/sh nvx_hostname=nvx-sandbox nr_cpus=1 nvx_workload_uid=65534 nvx_workload_gid=65534 nvx_lifecycle=managed nvx_maps=2 virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22 virtio_mmio.device=0x1000@0xd0007000:23 nvx_control_tty=hvc2 virtio_mmio.device=0x1000@0xd0001000:17 virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "rw"}}, + {"name": "the tokens of an adapter's layered root and of the VMM", "command_line": "console=hvc0 nvx_layer=distro,0xd0003000,gpt:fedcba98-7654-3210-fedc-ba9876543210 nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0001000:17 virtfs_dir=/run/nvx/hostfs/root virtfs_tag=microvm virtfs_mode=ro virtfs_aggregate=1", "export": {"tag": "microvm", "mode": "ro"}}, {"name": "the tokens in reverse order", "command_line": "virtfs_aggregate=1 virtfs_mode=ro virtfs_tag=microvm virtfs_dir=/run/nvx/hostfs/root", "export": {"tag": "microvm", "mode": "ro"}}, {"name": "tabs and newlines between tokens", "command_line": "quiet\tvirtfs_dir=/run/nvx/hostfs/root\nvirtfs_tag=microvm virtfs_mode=rw\tvirtfs_aggregate=1\n", "export": {"tag": "microvm", "mode": "rw"}}, {"name": "a tag of every kind of byte that a tag may hold", "command_line": "virtfs_dir=/run/nvx/hostfs/root virtfs_tag=AZaz09_- virtfs_mode=rw virtfs_aggregate=1", "export": {"tag": "AZaz09_-", "mode": "rw"}}, diff --git a/nvx-rs/tests/fixtures/root-tokens.json b/nvx-rs/tests/fixtures/root-tokens.json index 3f717e019..1f3135d48 100644 --- a/nvx-rs/tests/fixtures/root-tokens.json +++ b/nvx-rs/tests/fixtures/root-tokens.json @@ -11,13 +11,13 @@ {"name": "a GPT image without an identity in RAM", "command_line": "nvx_layer=distro,0xd0003000,gpt nvx_overlay_upper=ramfs", "layout": {"format": "gpt", "identity": null, "upper": "ram"}, "shell_error": "GPT image"}, {"name": "the upper before the image", "command_line": "nvx_scratch=0xd0006000,ext4 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, {"name": "the RAM upper before the image", "command_line": "nvx_overlay_upper=ramfs nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, - {"name": "the tokens of a managed sandbox and of the VMM", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_entrypoint=/bin/sh nvx_hostname=nvx-sandbox nvx_arg=-c nvx_memory_max=268435456 nvx_pids_max=65 nvx_share=0-4f9cd4a1a4e2b5c3d6e7f8091a2b3c4d,/workspace,rw nr_cpus=1 nvx_workload_uid=65534 nvx_workload_gid=65534 nvx_lifecycle=managed virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11 virtio_mmio.device=0x1000@0xd0007000:3 nvx_control_tty=hvc2 virtfs_dir=/run/nvx/shares virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, - {"name": "a RAM upper beside the distro and control devices", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_entrypoint=/bin/sh virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0007000:3", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, + {"name": "the tokens of a managed sandbox and of the VMM", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_entrypoint=/bin/sh nvx_hostname=nvx-sandbox nvx_arg=-c nvx_memory_max=268435456 nvx_pids_max=65 nvx_share=0-4f9cd4a1a4e2b5c3d6e7f8091a2b3c4d,/workspace,rw nr_cpus=1 nvx_workload_uid=65534 nvx_workload_gid=65534 nvx_lifecycle=managed virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22 virtio_mmio.device=0x1000@0xd0007000:23 nvx_control_tty=hvc2 virtfs_dir=/run/nvx/shares virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, + {"name": "a RAM upper beside the distro and control devices", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_entrypoint=/bin/sh virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0007000:23", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, {"name": "tabs and newlines between tokens", "command_line": "quiet\tnvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef\nnvx_scratch=0xd0006000,ext4\tnvx_sandbox=1", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, {"name": "parameters whose names only begin like the boot tokens", "command_line": "nvx_layer_capacity_sectors=2048 nvx_layers=x nvx_scratchpad=1 nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 nvx_overlay_upper_size=1 xnvx_layer=1", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}}, {"name": "no boot tokens, as for a blockless root", "command_line": "quiet nvx_sandbox=1 nvx_entrypoint=/bin/sh", "layout": null}, {"name": "an empty command line", "command_line": "", "layout": null}, - {"name": "a scratch device without boot tokens", "command_line": "quiet virtio_mmio.device=0x1000@0xd0006000:11", "layout": null}, + {"name": "a scratch device without boot tokens", "command_line": "quiet virtio_mmio.device=0x1000@0xd0006000:22", "layout": null}, {"name": "the form without a format, which names the UUID alone", "command_line": "nvx_layer=distro,0xd0003000,11111111-1111-1111-1111-111111111111 nvx_scratch=0xd0006000,ext4", "error": "the nvx_layer kernel parameter names no image format, erofs or gpt: 'distro,0xd0003000,11111111-1111-1111-1111-111111111111'", "shell_error": "image format"}, {"name": "an empty format", "command_line": "nvx_layer=distro,0xd0003000, nvx_scratch=0xd0006000,ext4", "error": "the nvx_layer kernel parameter names no image format, erofs or gpt: 'distro,0xd0003000,'", "shell_error": "image format"}, {"name": "an identity without a format", "command_line": "nvx_layer=distro,0xd0003000,:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4", "error": "the nvx_layer kernel parameter names no image format, erofs or gpt: 'distro,0xd0003000,:0123abcd-4567-89ef-0123-456789abcdef'", "shell_error": "image format"}, @@ -64,21 +64,21 @@ {"name": "another RAM upper", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=tmpfs", "error": "the nvx_overlay_upper kernel parameter must be ramfs, not 'tmpfs'"}, {"name": "a RAM upper with an empty value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=", "error": "the nvx_overlay_upper kernel parameter must be ramfs, not ''"}, {"name": "a RAM upper without a value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper", "error": "the nvx_overlay_upper kernel parameter has no value"}, - {"name": "a RAM upper with a scratch block device", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, - {"name": "a RAM upper with a scratch block device that a dashed parameter adds", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio-mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device that a dashed parameter adds", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio-mmio.device=0x1000@0xd0006000:22", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device at an address in capitals", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4K@0XD0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device at a decimal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@3489685504:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device at an octal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@032000060000:11:5", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device at a negative address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@-0xffffffff2fffa000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device at an address past 64 bits", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0x100000000d0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, - {"name": "a RAM upper with a scratch block device in a quoted parameter", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs \"virtio_mmio.device=0x1000@0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device in a quoted parameter", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs \"virtio_mmio.device=0x1000@0xd0006000:22\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper with a scratch block device whose quoted value spaces its address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=\"0x1000@ 0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, - {"name": "a RAM upper with a scratch block device after a vertical tab", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs quiet\u000bvirtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device after a vertical tab", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs quiet\u000bvirtio_mmio.device=0x1000@0xd0006000:22", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, {"name": "a RAM upper beside a device token without an address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000 virtio_mmio.device=0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, {"name": "a RAM upper beside a scratch device token whose IRQ is 0", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:0", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, - {"name": "a RAM upper beside a scratch device token with text after its IRQ", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:11:", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, - {"name": "a RAM upper beside a scratch device token in a quoted value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs x=\"y virtio_mmio.device=0x1000@0xd0006000:11\"", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, - {"name": "a RAM upper beside a scratch device token after --", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs -- virtio_mmio.device=0x1000@0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, - {"name": "a scratch upper beside its block device", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}} + {"name": "a RAM upper beside a scratch device token with text after its IRQ", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:22:", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token in a quoted value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs x=\"y virtio_mmio.device=0x1000@0xd0006000:22\"", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token after --", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs -- virtio_mmio.device=0x1000@0xd0006000:22", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a scratch upper beside its block device", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 virtio_mmio.device=0x1000@0xd0003000:19 virtio_mmio.device=0x1000@0xd0006000:22", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}} ] } diff --git a/nvx-rs/tests/fixtures/template/README.md b/nvx-rs/tests/fixtures/template/README.md index 6a29af744..0a6c891d4 100644 --- a/nvx-rs/tests/fixtures/template/README.md +++ b/nvx-rs/tests/fixtures/template/README.md @@ -13,7 +13,8 @@ against its own idea of them. | [`platform-mshv-network`](platform-mshv-network) | The same on MSHV, with a fixed NIC at `10.0.0.2/24`, its IPv6 identity, and an allow-list policy | Both are platform-tier captures for clones of microVM ABI version 3, whose -devices offer no access-platform and whose consoles offer no size, after the +devices offer no access-platform, whose consoles offer no size, and whose +virtio devices interrupt on IOAPIC inputs past the ISA IRQs, after the invariant configuration alone, with OpenVMM's test records of the time contract, of time ABI version 2, and of the CPU profile `intel.icelake-sp.v1`. @@ -22,7 +23,7 @@ contract, of time ABI version 2, and of the CPU profile `intel.icelake-sp.v1`. [`generate.rs`](generate.rs) builds each manifest as a capture does, with `microvm_machine_contract` of OpenVMM's `openvmm_helpers`, checks it with OpenVMM's own validation, and encodes it with `mesh`. It wrote these files at -OpenVMM commit `b9b1f8f6d41bcb7cc9b2d1ec7c836b30f5dc7718`. +OpenVMM commit `35e76265fa8b63922edead112fcd5a38c59445c8`. To write them again, as when a new version of the snapshot format, the time ABI, or the microVM ABI retires these, append `generate.rs` to diff --git a/nvx-rs/tests/fixtures/template/generate.rs b/nvx-rs/tests/fixtures/template/generate.rs index 00d4597fb..a21b1e19f 100644 --- a/nvx-rs/tests/fixtures/template/generate.rs +++ b/nvx-rs/tests/fixtures/template/generate.rs @@ -62,7 +62,6 @@ mod nvx_fixture { ]), ) .unwrap(); - let irq = defs::microvm_virtio_net_irq(Some(hypervisor)).unwrap(); let blocks = [ defs::MicrovmSandboxBlockConfig { role: defs::MicrovmSandboxBlockRole::Distro, @@ -75,7 +74,7 @@ mod nvx_fixture { ]; defs::append_microvm_virtio_discovery( &mut command_line, - with_network.then(|| (&network, irq, egress_policy.gateway_dns_server(), None)), + with_network.then(|| (&network, egress_policy.gateway_dns_server(), None)), true, &[], true, diff --git a/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin b/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin index 18068fcca..76b89c953 100644 Binary files a/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin and b/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin differ diff --git a/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin b/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin index 5f4d85461..9f4f7e08e 100644 Binary files a/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin and b/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin differ diff --git a/nvx-rs/tests/vectors/template-key-v1.txt b/nvx-rs/tests/vectors/template-key-v1.txt index 596d7ee1b..1694806a6 100644 --- a/nvx-rs/tests/vectors/template-key-v1.txt +++ b/nvx-rs/tests/vectors/template-key-v1.txt @@ -17,8 +17,10 @@ # # The keys and classes are those of the captures that tests/fixtures/template/README.md # describes, as microVM ABI 2 captured them, when every virtio device but the NIC also offered -# access-platform (feature bit 33) and each console also offered its size (feature bit 0), with -# the OpenVMM, kernel, and initramfs identities sha256:a..., sha256:b..., and +# access-platform (feature bit 33), each console also offered its size (feature bit 0), and the +# virtio devices interrupted on ISA IRQs (the NIC on 10, the filesystem on 6, the consoles on 7 +# and 3, and the distro and scratch blocks on 4 and 11), with the OpenVMM, kernel, and initramfs +# identities sha256:a..., sha256:b..., and # sha256:c..., the guest build guest-build-1, boot contract 3, control contract # nvx-microvm-v2-control-v2, control session 1, snapshot format 6, device-state schema 1, time # ABI 1, restore packet 4, and activation contract 1. diff --git a/openvmm b/openvmm index b9b1f8f6d..35e76265f 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit b9b1f8f6d41bcb7cc9b2d1ec7c836b30f5dc7718 +Subproject commit 35e76265fa8b63922edead112fcd5a38c59445c8 diff --git a/scripts/nvx_tools/benchmark.py b/scripts/nvx_tools/benchmark.py index 2f8435a87..12f4264f2 100644 --- a/scripts/nvx_tools/benchmark.py +++ b/scripts/nvx_tools/benchmark.py @@ -131,6 +131,9 @@ VIRTFS_HOST_TO_GUEST = b"host-to-guest\n" SMP_PROBE_COMPLETION_MARKER = b"NVX-SMP-PROBE-OK" SMP_PROBE_PATH = "/tmp/nvx-smp-probe" +# The microVM NIC's interrupt: IOAPIC input 16 on every backend, as Table 2.7 +# of the architecture guide lists it. +MICROVM_NETWORK_IRQ = 16 SNAPSHOT_CAPTURE_PATH = "/tmp/nvx-c" SNAPSHOT_POST_RESTORE_PATH = "/tmp/nvx-post-restore" SNAPSHOT_GUEST_DISPATCH_MARKER = b"NVX-SNAPSHOT-DISPATCHED" @@ -4812,13 +4815,12 @@ def benchmark_network_snapshot_workload( command_prefix=command_prefix, network=network, ) - ioapic_irq = 5 if backend == "whp" else 10 run_guest_script( cold_command, smp_probe_script( args.processors, network_gateway=gateway, - ioapic_irq=ioapic_irq, + ioapic_irq=MICROVM_NETWORK_IRQ, ), SMP_PROBE_COMPLETION_MARKER, timeout=max(args.timeout, 40.0), diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index f47cde74c..e789bf4ef 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -39,6 +39,7 @@ "guest-boot": 0.7, "guest-identity": 0.7, "host-loopback-policy": 23.1, + "interrupt-map": 4.1, "ipv6-egress-policy": 29.4, "l3-l4-egress-policy": 19.3, "l3-l4-egress-port-ranges": 22.4, @@ -71,7 +72,7 @@ }, "shards": [ { - "seconds": 266.1, + "seconds": 267.8, "scenarios": [ "command-line-limit", "console-snapshot", @@ -79,6 +80,7 @@ "endpoint-policy-snapshot", "filesystem-comma-paths", "filesystem-snapshot", + "lifecycle", "managed-lifecycle", "network-snapshot", "platform-restore-processors", @@ -94,7 +96,7 @@ ] }, { - "seconds": 265.4, + "seconds": 267.8, "scenarios": [ "console-exit", "cpu-profile-fallback", @@ -105,11 +107,11 @@ "guest-boot", "guest-identity", "host-loopback-policy", + "interrupt-map", "ipv6-egress-policy", "l3-l4-egress-policy", "l3-l4-egress-port-ranges", "l3-l4-egress-protocols", - "lifecycle", "platform-aggregate-export", "platform-filesystem", "platform-network-policy", @@ -143,6 +145,7 @@ "guest-boot": 0.5, "guest-identity": 0.5, "host-loopback-policy": 23.5, + "interrupt-map": 3.8, "ipv6-egress-policy": 30.0, "l3-l4-egress-policy": 19.3, "l3-l4-egress-port-ranges": 22.5, @@ -175,14 +178,15 @@ }, "shards": [ { - "seconds": 246.7, + "seconds": 248.8, "scenarios": [ + "command-line-limit", "console-snapshot", "directional-network-policy", "endpoint-policy-snapshot", - "filesystem-comma-paths", "filesystem-shares", "filesystem-snapshot", + "interrupt-map", "platform-filesystem", "platform-restore-processors", "platform-zero-fill", @@ -198,12 +202,12 @@ ] }, { - "seconds": 247.0, + "seconds": 248.7, "scenarios": [ - "command-line-limit", "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", + "filesystem-comma-paths", "filesystem-owner", "filesystem-policy", "guest-boot", @@ -247,6 +251,7 @@ "guest-boot": 0.8, "guest-identity": 0.8, "host-loopback-policy": 28.2, + "interrupt-map": 8.6, "ipv6-egress-policy": 38.6, "l3-l4-egress-policy": 19.9, "l3-l4-egress-port-ranges": 27.4, @@ -279,15 +284,15 @@ }, "shards": [ { - "seconds": 348.4, + "seconds": 353.1, "scenarios": [ - "console-snapshot", "denied-filesystem-paths", "directional-network-policy", "endpoint-policy-snapshot", "filesystem-policy", "filesystem-snapshot", "host-loopback-policy", + "interrupt-map", "network-snapshot", "platform-filesystem", "platform-restore-processors", @@ -302,10 +307,11 @@ ] }, { - "seconds": 349.3, + "seconds": 353.2, "scenarios": [ "command-line-limit", "console-exit", + "console-snapshot", "cpu-profile-fallback", "filesystem-comma-paths", "filesystem-owner", diff --git a/scripts/nvx_tools/microvm_test_scripts/console-snapshot.sh.in b/scripts/nvx_tools/microvm_test_scripts/console-snapshot.sh.in index 9d01b59b6..011cd7b68 100644 --- a/scripts/nvx_tools/microvm_test_scripts/console-snapshot.sh.in +++ b/scripts/nvx_tools/microvm_test_scripts/console-snapshot.sh.in @@ -7,7 +7,7 @@ fail() { } grep -q 'console=hvc1' /proc/cmdline || fail 50 -grep -q 'virtio_mmio.device=0x1000@0xd0002000:7' /proc/cmdline || fail 51 +grep -q 'virtio_mmio.device=0x1000@0xd0002000:18' /proc/cmdline || fail 51 [ -e /sys/class/tty/hvc1 ] || fail 52 stty -F /dev/hvc1 raw -echo printf '\000\015\012\177\377NVX-CONSOLE-BINARY\n' diff --git a/scripts/nvx_tools/microvm_test_scripts/filesystem-dormant.sh b/scripts/nvx_tools/microvm_test_scripts/filesystem-dormant.sh index 0d6c7d601..8801e11cf 100644 --- a/scripts/nvx_tools/microvm_test_scripts/filesystem-dormant.sh +++ b/scripts/nvx_tools/microvm_test_scripts/filesystem-dormant.sh @@ -1,5 +1,5 @@ set -eu -grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline +grep -q 'virtio_mmio.device=0x1000@0xd0001000:17' /proc/cmdline ! grep -q 'virtfs_tag=' /proc/cmdline echo NVX-FILESYSTEM-DORMANT-BEFORE nvx-snapshot diff --git a/scripts/nvx_tools/microvm_test_scripts/filesystem-read-only.sh b/scripts/nvx_tools/microvm_test_scripts/filesystem-read-only.sh index 1b53f6c9c..c81d73f27 100644 --- a/scripts/nvx_tools/microvm_test_scripts/filesystem-read-only.sh +++ b/scripts/nvx_tools/microvm_test_scripts/filesystem-read-only.sh @@ -14,7 +14,7 @@ fi if ln -s seed /mnt/share/link 2>/dev/null; then fail 26 fi -grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline || fail 23 +grep -q 'virtio_mmio.device=0x1000@0xd0001000:17' /proc/cmdline || fail 23 grep -q 'virtfs_tag=microvm' /proc/cmdline || fail 24 grep -q 'virtfs_mode=ro' /proc/cmdline || fail 25 echo NVX-FILESYSTEM-READ-ONLY-OK diff --git a/scripts/nvx_tools/microvm_test_scripts/filesystem-shares.sh b/scripts/nvx_tools/microvm_test_scripts/filesystem-shares.sh index c6d54f353..e9a11af6b 100644 --- a/scripts/nvx_tools/microvm_test_scripts/filesystem-shares.sh +++ b/scripts/nvx_tools/microvm_test_scripts/filesystem-shares.sh @@ -18,7 +18,7 @@ expect_read_only() { # One virtio-fs slot carries every share as a child of one aggregate, which # the guest mounts where only root can enter and binds at each target. -grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline || fail 40 +grep -q 'virtio_mmio.device=0x1000@0xd0001000:17' /proc/cmdline || fail 40 if grep -q '0xd0008000' /proc/cmdline; then fail 41 fi diff --git a/scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in b/scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in new file mode 100644 index 000000000..25c79e466 --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/interrupt-map.sh.in @@ -0,0 +1,169 @@ +set -eu +fail() { + code="$1" + echo "NVX-INTERRUPT-MAP-FAIL code=$code" + nvx-exit "$code" + exit "$code" +} + +# Counts the virtio devices that a driver has bound. +bound_devices() { + bound=0 + for device in /sys/bus/virtio/drivers/*/virtio*; do + [ -e "$device" ] && bound=$((bound + 1)) + done + echo "$bound" +} + +# Prints the base of the MMIO window that /proc/iomem lists under the name of +# the virtio-mmio platform device $1. +window_base() { + while IFS=' ' read -r range separator name _; do + [ "$separator" = : ] && [ "$name" = "$1" ] || continue + echo "${range%%-*}" + return 0 + done /dev/null || fail 30 ;; + d0001000) cat /mnt/share/seed >/dev/null || fail 31 ;; + d0002000) + # When a console's driver takes its buffer back before the + # device reads the used event, the device rightly skips the + # interrupt, so write until one interrupts. + while [ "$(irq_count "$irq")" -le "$fired" ] && [ "$writes" -lt 16 ]; do + echo NVX-INTERRUPT-MAP-CONSOLE + writes=$((writes + 1)) + await_irq "$irq" "$fired" 10 + done + ;; + d0003000) + block=$(ls "$device/block") || fail 33 + blockdev --flushbufs "/dev/$block" || fail 33 + dd if="/dev/$block" of=/dev/null bs=4096 count=16 2>/dev/null || fail 33 + ;; + d0006000) + block=$(ls "$device/block") || fail 34 + dd if=/dev/zero of="/dev/$block" bs=4096 count=16 conv=fsync 2>/dev/null || + fail 34 + ;; + d0007000) + # As for the boot console, a byte at a time, each in a buffer + # of its own: fewer bytes than the 44-byte header of the + # control-session protocol, which the console parses its + # input as. + while [ "$(irq_count "$irq")" -le "$fired" ] && [ "$writes" -lt 16 ]; do + printf . >/dev/hvc2 || fail 35 + writes=$((writes + 1)) + await_irq "$irq" "$fired" 10 + done + ;; + *) fail 36 ;; + esac + await_irq "$irq" "$fired" + if [ "$(irq_count "$irq")" -le "$fired" ]; then + echo "NVX-INTERRUPT-MAP-QUIET base=0x$base device=$name irq=$irq" \ + "count=$(irq_count "$irq")" + fail 37 + fi + done +} + +tries=0 +while [ "$(bound_devices)" -lt @DEVICES@ ] && [ "$tries" -lt 200 ]; do + sleep 0.05 + tries=$((tries + 1)) +done +[ "$(bound_devices)" -eq @DEVICES@ ] || fail 20 + +report boot +@CAPTURE@ +traffic +report traffic +# The APIC errors and spurious PIC interrupts (ERR), the level-triggered +# interrupts that the IOAPIC misrouted (MIS), and the spurious APIC +# interrupts (SPU). +echo "NVX-INTERRUPT-MAP-ERRORS" \ + "err=$(awk '$1 == "ERR:" { print $2 }' /proc/interrupts)" \ + "mis=$(awk '$1 == "MIS:" { print $2 }' /proc/interrupts)" \ + "spu=$(awk '$1 == "SPU:" { print $2 }' /proc/interrupts)" +# The whole map, for the log. +cat /proc/interrupts +echo NVX-INTERRUPT-MAP-OK +nvx-exit 0 diff --git a/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh b/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh index 928d67082..97d28c17a 100644 --- a/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh +++ b/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh @@ -19,7 +19,7 @@ expect_error() { # The restore attached an aggregate to the slot that the platform capture left # dormant. The guest keeps its captured command line, which discovers the slot # but names no filesystem, so the guest mounts the aggregate itself. -grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline || fail "discovery" 80 +grep -q 'virtio_mmio.device=0x1000@0xd0001000:17' /proc/cmdline || fail "discovery" 80 if grep -Eq '(^| )(virtfs_|nvx_share=)' /proc/cmdline; then fail "command line" 81 fi diff --git a/scripts/nvx_tools/microvm_test_scripts/sandbox-blocks.sh b/scripts/nvx_tools/microvm_test_scripts/sandbox-blocks.sh index 7f19c212f..9b4b0b668 100644 --- a/scripts/nvx_tools/microvm_test_scripts/sandbox-blocks.sh +++ b/scripts/nvx_tools/microvm_test_scripts/sandbox-blocks.sh @@ -15,8 +15,8 @@ for name in vda vdb; do [ -b "/dev/$name" ] || fail 20 done [ ! -b /dev/vdc ] || fail 30 -grep -q 'virtio_mmio.device=0x1000@0xd0003000:4' /proc/cmdline || fail 21 -grep -q 'virtio_mmio.device=0x1000@0xd0006000:11' /proc/cmdline || fail 24 +grep -q 'virtio_mmio.device=0x1000@0xd0003000:19' /proc/cmdline || fail 21 +grep -q 'virtio_mmio.device=0x1000@0xd0006000:22' /proc/cmdline || fail 24 # The windows and interrupts of the removed runtime and custom layers stay unassigned. ! grep -q '0xd0004000' /proc/cmdline || fail 31 ! grep -q '0xd0005000' /proc/cmdline || fail 32 diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 8fbb34de2..ed7584608 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -112,6 +112,7 @@ "guest-boot", "guest-identity", "host-loopback-policy", + "interrupt-map", "ipv6-egress-policy", "lifecycle", "l3-l4-egress-policy", @@ -149,6 +150,7 @@ UBUNTU_UNSUPPORTED_SCENARIOS = frozenset(("console-snapshot",)) SANDBOX_CONTROL_SCENARIOS = frozenset( ( + "interrupt-map", "managed-exec-config", "platform-aggregate-export", "platform-filesystem", @@ -209,31 +211,50 @@ # features that its driver negotiated, one character per bit from bit 0. VIRTIO_FEATURES_REPORT_PREFIX = "NVX-VIRTIO-FEATURES " VIRTIO_FEATURES_NETWORK = "10.0.0.2/24" +INTERRUPT_MAP_COMPLETION_MARKER = b"NVX-INTERRUPT-MAP-OK" +INTERRUPT_MAP_CAPTURE_MARKER = b"NVX-INTERRUPT-MAP-CAPTURE" +# The prefix of the lines on which the interrupt-map scenario's guest reports, +# after a step, the window of a virtio device and the interrupt that its +# driver requested, and of the line of /proc/interrupts' counts of APIC errors +# and spurious PIC interrupts (ERR), of level-triggered interrupts that the +# IOAPIC misrouted (MIS), and of spurious APIC interrupts (SPU). +INTERRUPT_MAP_REPORT_PREFIX = "NVX-INTERRUPT-MAP " +INTERRUPT_MAP_ERRORS_PREFIX = "NVX-INTERRUPT-MAP-ERRORS " +INTERRUPT_MAP_ERROR_COUNTS = ("err", "mis", "spu") +INTERRUPT_MAP_NETWORK = "10.0.0.2/24" +INTERRUPT_MAP_SHARE_SEED = "NVX-INTERRUPT-MAP-SHARE" +# The devices of the interrupt-map scenario's snapshot, by the bases of their +# windows: the NIC and the host filesystem with a share. +INTERRUPT_MAP_SNAPSHOT_DEVICES = (0xD000_0000, 0xD000_1000) class MicrovmVirtioDevice(NamedTuple): """A fixed virtio device of the microVM, as Table 2.8 of the architecture guide (doc/design/2.5-device-topology-transport-and-discovery.md) lists it: its name, virtio device ID, and the feature bits that it offers the - guest, every one of which Linux negotiates.""" + guest, every one of which Linux negotiates, and as Table 2.7 lists its + interrupt, the IOAPIC input of its window.""" name: str device_id: int features: frozenset[int] + irq: int # Each fixed virtio device of the microVM, by the base of its MMIO window. MICROVM_VIRTIO_DEVICES = { - 0xD000_0000: MicrovmVirtioDevice("Network", 1, frozenset({5, 32})), - 0xD000_1000: MicrovmVirtioDevice("Host filesystem", 26, frozenset({28, 29, 32})), - 0xD000_2000: MicrovmVirtioDevice("Boot console", 3, frozenset({28, 29, 32})), + 0xD000_0000: MicrovmVirtioDevice("Network", 1, frozenset({5, 32}), 16), + 0xD000_1000: MicrovmVirtioDevice( + "Host filesystem", 26, frozenset({28, 29, 32}), 17 + ), + 0xD000_2000: MicrovmVirtioDevice("Boot console", 3, frozenset({28, 29, 32}), 18), 0xD000_3000: MicrovmVirtioDevice( - "`distro` block", 2, frozenset({2, 5, 6, 9, 10, 28, 29, 32}) + "`distro` block", 2, frozenset({2, 5, 6, 9, 10, 28, 29, 32}), 19 ), 0xD000_6000: MicrovmVirtioDevice( - "`scratch` block", 2, frozenset({2, 6, 9, 10, 28, 29, 32}) + "`scratch` block", 2, frozenset({2, 6, 9, 10, 28, 29, 32}), 22 ), - 0xD000_7000: MicrovmVirtioDevice("Control console", 3, frozenset({28, 29, 32})), + 0xD000_7000: MicrovmVirtioDevice("Control console", 3, frozenset({28, 29, 32}), 23), } # The x86 guest kernel's COMMAND_LINE_SIZE. OpenVMM requires the effective # kernel command line, with the tokens that it appends after the caller's, to @@ -5131,6 +5152,312 @@ def run_virtio_features( check_virtio_features(result["output"]) +class InterruptMapEntry(NamedTuple): + """A virtio device's interrupt, as the interrupt-map scenario's guest + reports it after a step: the device's name, the interrupt that its driver + requested, that interrupt's controller (chip) and input (hwirq), its + trigger, its handlers, how many times it fired, and how many of its + interrupts lately found no work for any handler (``unhandled`` of + /proc/irq//spurious). + + The scenario reports that last count but does not check it. Linux keeps + it to detect a stuck line, and resets it to 1 when 100 ms pass without + such an interrupt, so it counts no history. A virtio console's driver + waits for its output by polling the used ring, so nearly every interrupt + of a console finds no work, and the NIC's polling can leave one such + interrupt too, such as after a restore. Each device's single handler, its + IOAPIC input, and its count rising with its own traffic catch an + interrupt delivered to the wrong line.""" + + device: str + irq: int + chip: str + hwirq: int + trigger: str + actions: str + fired: int + unhandled: int + + +def parse_interrupt_map( + output: str, +) -> tuple[dict[str, dict[int, InterruptMapEntry]], dict[str, int]]: + """Return the interrupt-map scenario's reports in ``output``, by step and + by the base of each device's window, and the guest's ERR, MIS, and SPU + counts. + + Raises RuntimeError for a malformed line or a device that a step reports + twice.""" + steps: dict[str, dict[int, InterruptMapEntry]] = {} + errors: dict[str, int] = {} + for line in output.splitlines(): + line = line.strip() + for prefix in (INTERRUPT_MAP_ERRORS_PREFIX, INTERRUPT_MAP_REPORT_PREFIX): + if line.startswith(prefix): + break + else: + continue + try: + fields = dict(field.split("=", 1) for field in line[len(prefix) :].split()) + if prefix == INTERRUPT_MAP_ERRORS_PREFIX: + errors = { + name: int(fields[name]) for name in INTERRUPT_MAP_ERROR_COUNTS + } + continue + step = fields["step"] + base = int(fields["base"], 16) + entry = InterruptMapEntry( + fields["device"], + int(fields["irq"]), + fields["chip"], + int(fields["hwirq"]), + fields["type"], + fields["actions"], + int(fields["count"]), + int(fields["unhandled"]), + ) + except (KeyError, ValueError) as error: + raise RuntimeError( + f"malformed interrupt-map report {line!r}: {error}" + ) from error + entries = steps.setdefault(step, {}) + if base in entries: + raise RuntimeError( + f"the device at {base:#x} is reported twice after step {step}" + ) + entries[base] = entry + return steps, errors + + +def check_interrupt_map( + steps: dict[str, dict[int, InterruptMapEntry]], + errors: dict[str, int], + bases: Collection[int], + *, + before: str, + after: str = "traffic", +) -> None: + """Check the interrupt-map scenario's reports against the map of the + architecture guide (2.4.2 and Table 2.7): after the steps ``before`` and + ``after``, exactly the devices of ``bases`` are present, and each + interrupts on the IOAPIC input of its window in MICROVM_VIRTIO_DEVICES, + edge-triggered and with no other handler; each interrupt fired after + ``before``, with the traffic that the guest drove; and /proc/interrupts + counts no APIC error or spurious PIC interrupt (ERR), no misrouted IOAPIC + interrupt (MIS), and no spurious APIC interrupt (SPU). + + Raises RuntimeError naming each departure.""" + problems: list[str] = [] + for step in (before, after): + entries = steps.get(step) + if entries is None: + problems.append(f"no device is reported after step {step}") + continue + for base in sorted(set(entries) - set(bases)): + problems.append( + f"an unexpected device {entries[base].device} at {base:#x} is " + f"reported after step {step}" + ) + for base in sorted(bases): + expected = MICROVM_VIRTIO_DEVICES[base] + entry = entries.get(base) + if entry is None: + problems.append( + f"no {expected.name} is reported at {base:#x} after step {step}" + ) + continue + if entry.irq != expected.irq: + problems.append( + f"{expected.name} at {base:#x} interrupts on IRQ {entry.irq}, " + f"not {expected.irq}" + ) + if entry.chip != "IO-APIC" or entry.hwirq != entry.irq: + problems.append( + f"{expected.name}'s IRQ {entry.irq} is input {entry.hwirq} of " + f"{entry.chip}, not the IOAPIC input of its number" + ) + if entry.trigger != "edge": + problems.append( + f"{expected.name}'s IRQ {entry.irq} is {entry.trigger}-triggered" + ) + if entry.actions != entry.device: + problems.append( + f"{expected.name} shares IRQ {entry.irq}, whose handlers are " + f"{entry.actions}" + ) + if not problems: + for base in sorted(bases): + expected = MICROVM_VIRTIO_DEVICES[base] + first, last = steps[before][base], steps[after][base] + if last.fired <= first.fired: + problems.append( + f"{expected.name}'s IRQ {first.irq} fired {first.fired} times " + f"after step {before} and {last.fired} after step {after}" + ) + if not errors: + problems.append("the guest reported no ERR, MIS, and SPU counts") + elif any(errors.values()): + problems.append( + f"/proc/interrupts counts {errors['err']} APIC errors or spurious PIC " + f"interrupts (ERR), {errors['mis']} misrouted IOAPIC interrupts " + f"(MIS), and {errors['spu']} spurious APIC interrupts (SPU)" + ) + if problems: + raise RuntimeError( + "the microVM's interrupts differ from the architecture guide's map: " + + "; ".join(problems) + ) + + +def run_interrupt_map( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Check the interrupt map of the architecture guide (2.4.2 and Table + 2.7) end to end. + + A microVM with every fixed virtio device, the NIC, the host filesystem + with a share, both consoles, and both sandbox blocks, reports the + interrupt that each device's driver requested, drives traffic through + each, and reports the interrupts again: each must be the IOAPIC input of + its window, past the ISA IRQs, edge-triggered, with no other handler, and + must have fired, and /proc/interrupts must count no APIC error, misrouted + IOAPIC interrupt, or spurious interrupt. A snapshot of a microVM with the + NIC and the share then restores with the same interrupts, which fire + again after the restore.""" + with tempfile.TemporaryDirectory(prefix="nvx-interrupt-map-") as temporary: + root = Path(temporary) + share = root / "share" + share.mkdir() + (share / "seed").write_text(INTERRUPT_MAP_SHARE_SEED, encoding="utf-8") + mount = f"/mnt/share,{share},ro" + command = workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + network=INTERRUPT_MAP_NETWORK, + mount=mount, + ) + command.extend( + ( + "--virtio-console", + "console", + "--microvm-control-console", + "none", + ) + ) + for role in ("distro", "scratch"): + path = root / f"{role}.img" + with path.open("wb") as disk: + disk.truncate(SANDBOX_BLOCK_SIZE) + read_only = ",ro" if role != "scratch" else "" + command.extend( + ("--microvm-sandbox-block", f"{role}:file:{path}{read_only}") + ) + result = run_guest_script( + command, + _render_script( + "interrupt-map.sh.in", + DEVICES=str(len(MICROVM_VIRTIO_DEVICES)), + CAPTURE="", + ), + INTERRUPT_MAP_COMPLETION_MARKER, + timeout=timeout, + log_path=output_dir / "interrupt-map.log", + ) + steps, errors = parse_interrupt_map(result["output"]) + check_interrupt_map(steps, errors, MICROVM_VIRTIO_DEVICES, before="boot") + + snapshot_path = root / "snapshot" + capture_command = [ + *workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + network=INTERRUPT_MAP_NETWORK, + mount=mount, + ), + "--snapshot-destination", + str(snapshot_path), + ] + capture_marker = INTERRUPT_MAP_CAPTURE_MARKER.decode() + with OpenvmmProcess( + capture_command, + output_dir / "interrupt-map-capture.log", + ) as process: + process.wait_for(BOOT_MARKER, timeout) + _stage_script( + process, + "/tmp/nvx-interrupt-map", + "NVX_INTERRUPT_MAP", + _render_script( + "interrupt-map.sh.in", + DEVICES=str(len(INTERRUPT_MAP_SNAPSHOT_DEVICES)), + CAPTURE=f"echo {capture_marker}\nnvx-snapshot\nreport restore", + ), + ) + source = process.wait(timeout) + if source.returncode != 0: + raise RuntimeError( + f"interrupt-map capture source exited with {source.returncode}" + ) + if _output_lines(source.output).count(INTERRUPT_MAP_CAPTURE_MARKER) != 1: + raise RuntimeError( + "interrupt-map source did not reach capture exactly once" + ) + captured, _ = parse_interrupt_map( + source.output.decode("utf-8", errors="replace") + ) + if set(captured) != {"boot"}: + raise RuntimeError("interrupt-map source crossed its capture boundary") + + restore_command = snapshot_restore_command( + executable, + backend, + snapshot_path, + network_profile="portable", + ) + restore_command.extend(("--mount", mount)) + with OpenvmmProcess( + restore_command, + output_dir / "interrupt-map-restore.log", + ) as process: + process.wait_for(INTERRUPT_MAP_COMPLETION_MARKER, timeout) + restored = process.wait(timeout) + if restored.returncode != 0: + raise RuntimeError( + f"interrupt-map restore exited with {restored.returncode}" + ) + steps, errors = parse_interrupt_map( + restored.output.decode("utf-8", errors="replace") + ) + if "boot" in steps: + raise RuntimeError("interrupt-map restore replayed the steps before capture") + check_interrupt_map(steps, errors, INTERRUPT_MAP_SNAPSHOT_DEVICES, before="restore") + for base in INTERRUPT_MAP_SNAPSHOT_DEVICES: + before = captured["boot"].get(base) + after = steps["restore"][base] + if before is None or before._replace(fired=0, unhandled=0) != after._replace( + fired=0, unhandled=0 + ): + raise RuntimeError( + f"{MICROVM_VIRTIO_DEVICES[base].name}'s interrupt changed across " + f"the snapshot: {before} before capture, {after} after restore" + ) + + def _measure_restore( command: Sequence[str], *, @@ -10872,6 +11199,20 @@ def _run_scenarios( timeout=args.timeout, log_path=output_dir / "virtio-features.log", ) + if "interrupt-map" in scenarios: + report.start( + "interrupt-map", + f"Running the microVM interrupt map on OpenVMM/{args.backend}", + ) + run_interrupt_map( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "command-line-limit" in scenarios: report.start( "command-line-limit", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index e9ac7dbe1..4ec3fb9ca 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -23,7 +23,7 @@ import threading import time import unittest -from collections.abc import Callable, Collection, Sequence +from collections.abc import Callable, Collection, Mapping, Sequence from contextlib import ExitStack, nullcontext, redirect_stderr, redirect_stdout from datetime import datetime from pathlib import Path @@ -7912,7 +7912,7 @@ def test_command_line_limit_fills_the_guest_kernel_buffer(self): # network bootstrap. head = "earlycon=xe9 console=hvc0 reboot=t panic=-1 quiet loglevel=0" tail = ( - "nr_cpus=1 virtio_mmio.device=0x1000@0xd0000000:5 virtnet_ip=10.0.0.2 " + "nr_cpus=1 virtio_mmio.device=0x1000@0xd0000000:16 virtnet_ip=10.0.0.2 " "virtnet_mask=255.255.255.0 virtnet_gw=10.0.0.1 " "virtnet_ip6=fd00::a00:2/120 virtnet_gw6=fd00::a00:1" ) @@ -10394,7 +10394,8 @@ def table(number: str) -> list[list[str]]: bases = { row[0]: int(row[2].strip("`").split("..")[0], 16) for row in table("2.7") } - documented = {} + irqs = {row[0]: int(row[3]) for row in table("2.7")} + documented: dict[int, microvm_tests.MicrovmVirtioDevice] = {} for device, device_id, bits, features in table("2.8"): numbers = frozenset(int(bit) for bit in bits.split(", ")) self.assertEqual( @@ -10403,9 +10404,313 @@ def table(number: str) -> list[list[str]]: device, ) documented[bases[device]] = microvm_tests.MicrovmVirtioDevice( - device, int(device_id), numbers + device, int(device_id), numbers, irqs[device] ) self.assertEqual(documented, microvm_tests.MICROVM_VIRTIO_DEVICES) + # The device of window n, n windows of 4 KiB past 0xd0000000, has IOAPIC + # input 16 + n, past the ISA IRQs. + for base, device in documented.items(): + self.assertEqual(device.irq, 16 + (base - 0xD000_0000) // 0x1000, device) + + @staticmethod + def _interrupt_report( + steps: dict[str, dict[int, tuple[str, int, int]]], + *, + chip: str = "IO-APIC", + trigger: str = "edge", + shared: str = "", + unhandled: Mapping[int, int] | None = None, + errors: tuple[int, int, int] | None = (0, 0, 0), + ) -> str: + """Returns the interrupt-map scenario's report of `steps`: after each + step, each device by the base of its window, with its name, interrupt, + interrupt count, and count of interrupts that found no work, which + `unhandled` gives by base and is otherwise 0, as the scenario's script + prints it, with the console's carriage returns.""" + lines = ["NVX-GUEST-SCRIPT-START echo is not a report"] + for step, devices in steps.items(): + for base, (name, irq, count) in devices.items(): + actions = name + (f",{shared}" if shared else "") + lines.append( + f"NVX-INTERRUPT-MAP step={step} base={base:#x} device={name} " + f"irq={irq} chip={chip} hwirq={irq} type={trigger} " + f"actions={actions} count={count} " + f"unhandled={(unhandled or {}).get(base, 0)}\r" + ) + if errors is not None: + lines.append( + f"NVX-INTERRUPT-MAP-ERRORS err={errors[0]} mis={errors[1]} " + f"spu={errors[2]}\r" + ) + lines.append("NVX-INTERRUPT-MAP-OK\r") + return "\n".join(lines) + "\n" + + @staticmethod + def _documented_interrupts( + bases: Collection[int], step: str, count: int + ) -> dict[str, dict[int, tuple[str, int, int]]]: + return { + step: { + base: ( + f"virtio{index}", + microvm_tests.MICROVM_VIRTIO_DEVICES[base].irq, + count, + ) + for index, base in enumerate(sorted(bases)) + } + } + + def _check_interrupts(self, report: str) -> None: + steps, errors = microvm_tests.parse_interrupt_map(report) + microvm_tests.check_interrupt_map( + steps, errors, microvm_tests.MICROVM_VIRTIO_DEVICES, before="boot" + ) + + def test_interrupt_map_names_each_departure_from_the_guide(self): + bases = microvm_tests.MICROVM_VIRTIO_DEVICES + accepted = { + **self._documented_interrupts(bases, "boot", 1), + **self._documented_interrupts(bases, "traffic", 5), + } + self._check_interrupts(self._interrupt_report(accepted)) + # Interrupts that found no work are reported, not checked: a console's + # driver polls for its output, so nearly all of its interrupts find + # none, and the NIC's polling can leave one too. + self._check_interrupts( + self._interrupt_report( + accepted, + unhandled={0xD000_0000: 1, 0xD000_2000: 4177, 0xD000_7000: 1}, + ) + ) + + # ABI 2 put each device on an ISA IRQ, and the NIC on 10 on KVM. + isa = {0xD000_0000: 10, 0xD000_1000: 6, 0xD000_2000: 7} + abi_2 = { + step: { + base: (name, isa.get(base, irq), count) + for base, (name, irq, count) in devices.items() + } + for step, devices in accepted.items() + } + with self.assertRaises(RuntimeError) as raised: + self._check_interrupts(self._interrupt_report(abi_2)) + message = str(raised.exception) + self.assertIn("Network at 0xd0000000 interrupts on IRQ 10, not 16", message) + self.assertIn( + "Host filesystem at 0xd0001000 interrupts on IRQ 6, not 17", message + ) + self.assertIn("Boot console at 0xd0002000 interrupts on IRQ 7, not 18", message) + self.assertNotIn("`distro` block", message) + + quiet = { + **accepted, + "traffic": { + base: (name, irq, 1 if base == 0xD000_6000 else count) + for base, (name, irq, count) in accepted["traffic"].items() + }, + } + missing = { + step: { + base: entry for base, entry in devices.items() if base != 0xD000_3000 + } + for step, devices in accepted.items() + } + unknown = { + step: {**devices, 0xD000_4000: ("virtio9", 20, 1)} + for step, devices in accepted.items() + } + for name, report, problem in ( + ( + "level", + self._interrupt_report(accepted, trigger="level"), + "Network's IRQ 16 is level-triggered", + ), + ( + "pic", + self._interrupt_report(accepted, chip="XT-PIC"), + "Network's IRQ 16 is input 16 of XT-PIC, not the IOAPIC input of " + "its number", + ), + ( + "shared", + self._interrupt_report(accepted, shared="rtc0"), + "Control console shares IRQ 23, whose handlers are virtio5,rtc0", + ), + ( + "quiet", + self._interrupt_report(quiet), + "`scratch` block's IRQ 22 fired 1 times after step boot and 1 " + "after step traffic", + ), + ( + "missing", + self._interrupt_report(missing), + "no `distro` block is reported at 0xd0003000 after step boot", + ), + ( + "unknown", + self._interrupt_report(unknown), + "an unexpected device virtio9 at 0xd0004000 is reported after step " + "boot", + ), + ( + "errors", + self._interrupt_report(accepted, errors=(2, 1, 0)), + "/proc/interrupts counts 2 APIC errors or spurious PIC interrupts " + "(ERR), 1 misrouted IOAPIC interrupts (MIS), and 0 spurious APIC " + "interrupts (SPU)", + ), + ( + "spurious", + self._interrupt_report(accepted, errors=(0, 0, 3)), + "and 3 spurious APIC interrupts (SPU)", + ), + ( + "no errors", + self._interrupt_report(accepted, errors=None), + "the guest reported no ERR, MIS, and SPU counts", + ), + ( + "no traffic", + self._interrupt_report({"boot": accepted["boot"]}), + "no device is reported after step traffic", + ), + ): + with ( + self.subTest(name=name), + self.assertRaisesRegex(RuntimeError, re.escape(problem)), + ): + self._check_interrupts(report) + for line in ( + "NVX-INTERRUPT-MAP step=boot base=0xd0000000 device=virtio0 irq=16", + "NVX-INTERRUPT-MAP step=boot base=0xd0000000 device=virtio0 irq=x " + "chip=IO-APIC hwirq=16 type=edge actions=virtio0 count=1 unhandled=0", + "NVX-INTERRUPT-MAP step=boot base=0xd0000000 device=virtio0 irq=16 " + "chip=IO-APIC hwirq=16 type=edge actions=virtio0 count=1", + "NVX-INTERRUPT-MAP-ERRORS err=0 mis=0", + ): + with ( + self.subTest(line=line), + self.assertRaisesRegex(RuntimeError, "malformed interrupt-map report"), + ): + microvm_tests.parse_interrupt_map(line + "\n") + report = self._interrupt_report(accepted) + with self.assertRaisesRegex(RuntimeError, "is reported twice after step boot"): + microvm_tests.parse_interrupt_map(report + report) + + def test_interrupt_map_boots_every_device_and_restores_the_nic_and_share(self): + bases = microvm_tests.MICROVM_VIRTIO_DEVICES + snapshot_bases = microvm_tests.INTERRUPT_MAP_SNAPSHOT_DEVICES + cold = self._interrupt_report( + { + **self._documented_interrupts(bases, "boot", 1), + **self._documented_interrupts(bases, "traffic", 5), + } + ) + captured = self._interrupt_report( + self._documented_interrupts(snapshot_bases, "boot", 3), errors=None + ).replace("NVX-INTERRUPT-MAP-OK", "NVX-INTERRUPT-MAP-CAPTURE") + # The NIC's polling can leave an interrupt without work after a + # restore, which the comparison with the capture ignores. + restored = self._interrupt_report( + { + **self._documented_interrupts(snapshot_bases, "restore", 3), + **self._documented_interrupts(snapshot_bases, "traffic", 9), + }, + unhandled={0xD000_0000: 1}, + ) + launched: list[list[str]] = [] + staged: list[str] = [] + + class FakeProcess: + def __init__(self, command: list[str], _log_path: Path) -> None: + launched.append(command) + self.command = command + + def __enter__(self) -> "FakeProcess": + return self + + def __exit__(self, *_exception: object) -> None: + return None + + def wait_for(self, _marker: bytes, _timeout: float) -> None: + return None + + def send_bytes(self, data: bytes) -> None: + staged.append(data.decode()) + + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: + output = restored if "--restore-snapshot" in self.command else captured + return openvmm_process.OpenvmmProcessResult(0, output.encode()) + + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object( + microvm_tests, + "run_guest_script", + return_value={"text": cold, "output": cold}, + ) as run_guest_script, + patch.object(microvm_tests, "OpenvmmProcess", FakeProcess), + ): + microvm_tests.run_interrupt_map( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "whp", + memory_mib=128, + timeout=60, + output_dir=Path(temporary), + ) + + command, script, marker = run_guest_script.call_args.args + self.assertEqual(marker, microvm_tests.INTERRUPT_MAP_COMPLETION_MARKER) + self.assertIn("--net", command) + self.assertTrue(command[command.index("--mount") + 1].startswith("/mnt/share,")) + self.assertEqual( + command[command.index("--virtio-console") + 1 :][:3], + ["console", "--microvm-control-console", "none"], + ) + self.assertEqual(command.count("--microvm-sandbox-block"), 2) + self.assertIn('[ "$(bound_devices)" -eq 6 ]', script) + self.assertNotIn("nvx-snapshot", script) + capture, restore = launched + self.assertIn("--snapshot-destination", capture) + self.assertNotIn("--microvm-sandbox-block", capture) + self.assertNotIn("--virtio-console", capture) + self.assertEqual( + restore[restore.index("--mount") + 1], capture[capture.index("--mount") + 1] + ) + self.assertEqual(restore[restore.index("--network-profile") + 1], "portable") + self.assertIn('[ "$(bound_devices)" -eq 2 ]', staged[0]) + self.assertIn( + "echo NVX-INTERRUPT-MAP-CAPTURE\nnvx-snapshot\nreport restore\n", staged[0] + ) + + # A restore whose interrupts differ from those before the capture fails. + restored = restored.replace("irq=17 ", "irq=21 ").replace( + "hwirq=17", "hwirq=21" + ) + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object( + microvm_tests, + "run_guest_script", + return_value={"text": cold, "output": cold}, + ), + patch.object(microvm_tests, "OpenvmmProcess", FakeProcess), + self.assertRaisesRegex( + RuntimeError, "Host filesystem at 0xd0001000 interrupts on IRQ 21" + ), + ): + microvm_tests.run_interrupt_map( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + memory_mib=128, + timeout=60, + output_dir=Path(temporary), + ) def test_smp_snapshot_runs_the_warp_probe_after_each_restore(self): def measure(command: list[str], **kwargs: object) -> None: @@ -13107,7 +13412,7 @@ def test_platform_aggregate_export_script_mounts_and_checks_the_aggregate(self): # The clone keeps its captured command line, which discovers the slot # but names no filesystem, so the guest mounts the aggregate itself. self.assertIn( - "\ngrep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline ||", + "\ngrep -q 'virtio_mmio.device=0x1000@0xd0001000:17' /proc/cmdline ||", script, ) self.assertIn( diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index af6773414..6ab2fbd7a 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -23148,7 +23148,7 @@ def test_smp_probe_uses_explicit_topology_and_worker_rendezvous(self): network_script = benchmark.smp_probe_script( 4, network_gateway="10.0.0.1", - ioapic_irq=10, + ioapic_irq=16, ) self.assertIn('ping -c 2 -W 1 "10.0.0.1"', network_script) self.assertIn("SMP-IOAPIC-FAIL", network_script) @@ -23186,7 +23186,7 @@ def test_prepare_snapshot_capture_stages_waiting_controller(self): 4, teardown_mode="guest-exit", network_gateway="10.0.0.1", - ioapic_irq=10, + ioapic_irq=16, ) self.assertTrue( @@ -23680,23 +23680,28 @@ def test_network_workload_runs_unmeasured_smp_interrupt_preflight(self): "teardown_min_ms": 1.0, "teardown_max_ms": 1.0, } - with ( - patch.object(benchmark, "run_guest_script") as probe, - patch.object(benchmark, "benchmark", return_value=result), - patch.object(benchmark, "capture_automatic_snapshot"), - ): - benchmark.benchmark_network_snapshot_workload( - args, - Path("openvmm"), - Path("vmlinux"), - Path("initrd"), - "kvm", - ) + # The NIC interrupts on IOAPIC input 16 on every backend. + for backend in ("kvm", "mshv", "whp"): + with ( + patch.object(benchmark, "run_guest_script") as probe, + patch.object(benchmark, "benchmark", return_value=result), + patch.object(benchmark, "capture_automatic_snapshot"), + ): + benchmark.benchmark_network_snapshot_workload( + args, + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + backend, + ) - self.assertEqual(probe.call_count, 1) - self.assertIn("--network-profile", probe.call_args.args[0]) - self.assertIn("SMP-IOAPIC-OK", probe.call_args.args[1]) - self.assertEqual(probe.call_args.args[2], benchmark.SMP_PROBE_COMPLETION_MARKER) + self.assertEqual(probe.call_count, 1) + self.assertIn("--network-profile", probe.call_args.args[0]) + self.assertIn("SMP-IOAPIC-OK", probe.call_args.args[1]) + self.assertIn("ioapic_irq=16\n", probe.call_args.args[1]) + self.assertEqual( + probe.call_args.args[2], benchmark.SMP_PROBE_COMPLETION_MARKER + ) def test_shell_snapshot_restore_suite_only_measures_restore(self): args = argparse.Namespace(