Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ New runs emit ABI value 3 and are stored under `microvm-v3` paths, so they
cannot collide with the legacy unsuffixed ABI-1 history or with the `microvm-v2`
history. ABI 3 began new series because its devices stopped offering
`VIRTIO_F_ACCESS_PLATFORM`, so that its guests stopped mapping virtio buffers
through the DMA API.
through the DMA API, and moved from ISA IRQs, which the PIC also sees, to
IOAPIC inputs of their own.
The OpenVMM benchmark coordinator is implemented in `scripts/nvx_tools/benchmark.py` and exposed
through the supported NVX CLI. Guest payloads live in `scripts/nvx_tools/benchmark_scripts`; the
coordinator fills the `.sh.in` templates before use.
Expand Down
24 changes: 24 additions & 0 deletions doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,30 @@ does a documented feature that the guest lacks. A unit test checks that the
scenario's devices are those of Tables 2.7 and 2.8, and OpenVMM's unit tests
check the exact feature mask that each device offers.

The `interrupt-map` scenario covers the interrupt map of
[2.4.2](design/2.4-interrupt-and-io-architecture.md#242-interrupt-routing) and
[Table 2.7](design/2.5-device-topology-transport-and-discovery.md). It boots a
guest with every fixed virtio device: the NIC, the host filesystem with a
share, both consoles, and both sandbox blocks. Once a driver has bound each of
the six, the guest reports, for each, the base of its MMIO window and the
interrupt that its driver requested, as `/sys/kernel/irq` describes it: its
controller and input, its trigger, its handlers, and how many times it fired.
It then drives each device, pinging the gateway, reading the share, writing to
both consoles, reading the `distro` block and writing the `scratch` block past
the page cache, and reports again. Each device must interrupt on the IOAPIC
input of its window, 16 through 23, edge-triggered and with no other handler,
its interrupt must have fired, and `/proc/interrupts` must count no APIC error
or spurious PIC interrupt (`ERR`), no level-triggered interrupt that the IOAPIC
misrouted (`MIS`), and no spurious APIC interrupt (`SPU`). The guest also
reports how many of each device's interrupts lately found no work for any
handler (`unhandled` in `/proc/irq/<n>/spurious`), which the scenario does not
check: a console's driver polls for its output's completion, so nearly every
interrupt of a console finds no work, and the NIC's polling can leave one such
interrupt too. A guest with the NIC and the share then captures a snapshot,
whose restore must report the same interrupts and fire them again. A unit test
checks that the scenario's interrupts are those of Table 2.7, and OpenVMM's
unit tests check the MP table's routing and each slot's input.

Shared guest artifacts are built with Docker on a GitHub-hosted Ubuntu runner.
The kernel, Alpine initramfs, Ubuntu initramfs, and Ubuntu EROFS layer use
separate cache keys. The EROFS layer's entry also holds the sandbox scratch
Expand Down
2 changes: 1 addition & 1 deletion doc/design/1.5-compatibility-and-versioning-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ contract carries its own version or identity, independent of the others.
A change to a guest-visible address, interrupt, command-line token, feature
mask, queue shape, time rule, or device behavior requires a new contract
version. A backend-specific difference is valid only when the versioned
contract states it explicitly, as it does for the virtio-net interrupt.
contract states it explicitly.

Guest-visible configuration is immutable for the lifetime represented by a
snapshot. Restore derives the machine from the snapshot manifest; caller input
Expand Down
51 changes: 35 additions & 16 deletions doc/design/2.4-interrupt-and-io-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
[Architecture documentation](README.md) >
[2. Virtual Machine Architecture](2-virtual-machine-architecture.md)

As Figure 2.2 shows, interrupts and I/O follow separate paths. Devices signal
interrupts on interrupt request lines (IRQs) of the PIC and IOAPIC, which
deliver them to each processor's local advanced programmable interrupt
controller (LAPIC). The LAPIC also provides a timer. Processors reach devices
through port I/O and MMIO, while bulk data and interrupt status move through
guest RAM.
As Figure 2.2 shows, interrupts and I/O follow separate paths. The PIT and RTC
signal interrupts on interrupt request lines (IRQs) of the ISA bus, which both
the PIC and the IOAPIC see, and the virtio devices signal theirs on IOAPIC
inputs that the PIC does not see. The controllers deliver interrupts to each
processor's local advanced programmable interrupt controller (LAPIC). The
LAPIC also provides a timer. Processors reach devices through port I/O and
MMIO, while bulk data and interrupt status move through guest RAM.

```mermaid
flowchart TB
Expand All @@ -17,15 +18,18 @@ flowchart TB
Virtio["Fixed virtio-mmio slots"]
Timers["PIT and RTC"]
Ram["Guest RAM<br/>interrupt-status page<br/>and descriptor chains"]
Controllers["PIC and IOAPIC<br/>16 ISA interrupt lines"]
Pic["PIC<br/>16 ISA IRQs"]
Ioapic["IOAPIC<br/>24 inputs"]

Vcpu -->|"port I/O"| Ports
Vcpu -->|"MMIO configuration<br/>and notification"| Virtio
Vcpu <-->|"descriptors and<br/>status words"| Ram
Virtio <-->|"bulk data and<br/>status bits"| Ram
Virtio -->|"dedicated edge IRQs"| Controllers
Timers -->|"IRQ 0 and IRQ 8"| Controllers
Controllers -->|"interrupts"| Vcpu
Virtio -->|"dedicated edge IRQs<br/>16 to 23"| Ioapic
Timers -->|"IRQ 0 and IRQ 8"| Pic
Timers -->|"IRQ 0 and IRQ 8"| Ioapic
Pic -->|"interrupts"| Vcpu
Ioapic -->|"interrupts"| Vcpu
```

**Figure 2.2.** Interrupt and I/O paths.
Expand All @@ -34,7 +38,7 @@ flowchart TB

The base chipset contains exactly this allowlist:

- A generic PIC and IOAPIC.
- A generic PIC and a generic IOAPIC of 24 inputs.
- The selected backend's LAPIC.
- An i8253 PIT on IRQ 0.
- A generic programmable CMOS RTC anchored to UTC.
Expand All @@ -56,11 +60,26 @@ behavior of the LAPIC timer, PIT, and RTC.

## 2.4.2 Interrupt Routing

The MP table describes processors, the ISA bus, the IOAPIC, and legacy IRQ
routing, and it routes only the 16 ISA interrupts. The machine profile uses no
ACPI interrupt tables. Dedicated virtio interrupts use active-high edge
delivery. No virtio device uses the RTC's exclusive IRQ 8, and no device uses
IRQ 9, 12, or 13 on any backend.
The MP table describes processors, the ISA bus, the IOAPIC, and the routing of
every one of the IOAPIC's 24 inputs. The machine profile uses no ACPI interrupt
tables. The interrupt map is the same on every backend:

- Inputs 0 through 15 carry the 16 ISA IRQs, which the PIC also sees. The
PIT's IRQ 0 reaches input 2, IRQ 2 cascades the PIC, and the RTC's IRQ 8 is
its own. ISA IRQs 1, 3 through 7, and 9 through 15 are routed but unused.
- Inputs 16 through 23 have no ISA IRQ and no PIC input. The MP table names
each one as the active-high, edge-triggered interrupt of its own number, so
the guest reaches them only through the IOAPIC.
- Each fixed virtio-mmio slot has an input of its own: the slot whose window
starts n windows of 4 KiB past `0xd0000000` interrupts on input 16 + n, as
[Table 2.7](2.5-device-topology-transport-and-discovery.md) lists. Inputs 20
and 21, of the unused windows 4 and 5, are free.

Dedicated virtio interrupts use active-high edge delivery, and no two devices
share an interrupt. The 24 inputs bound the map: a window past window 7 has no
input of its own, so a further fixed device takes window 4 or 5 with its input
20 or 21, an ISA IRQ that the PIC also sees, or an IOAPIC with more inputs,
each in a new contract version.

## 2.4.3 Port I/O

Expand Down
21 changes: 12 additions & 9 deletions doc/design/2.5-device-topology-transport-and-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,20 @@ are validated before host resources are resolved.

| Device | Stable identity | MMIO range | IRQ | Presence |
| --- | --- | ---: | ---: | --- |
| Network | `net:microvm0` | `0xd0000000..0xd0000fff` | KVM/MSHV 10, WHP 5 | Optional |
| Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 6 | Reserved slot; attachment optional |
| Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 7 | Optional |
| `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 4 | Optional, read-only |
| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Optional, writable final role; absent in the RAM-backed overlay layout |
| Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 3 | Optional |
| Network | `net:microvm0` | `0xd0000000..0xd0000fff` | 16 | Optional |
| Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 17 | Reserved slot; attachment optional |
| Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 18 | Optional |
| `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 19 | Optional, read-only |
| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 22 | Optional, writable final role; absent in the RAM-backed overlay layout |
| Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 23 | Optional |

Placement is explicit rather than allocated sequentially, so an omitted device
never shifts another. All six slots are reserved, and the windows
`0xd0004000..0xd0005fff` and `0xd0008000..0xd0008fff`, IRQs 9, 12, and 13,
and the status words at `0x30010`, `0x30014`, and `0x30020` are unused.
never shifts another. The slot of window n, at `0xd0000000` plus n times
`0x1000`, interrupts on IOAPIC input 16 + n, on every backend, and owns the
status word at `0x30000` plus 4n. All six slots are reserved, and windows 4
and 5, `0xd0004000..0xd0005fff`, with inputs 20 and 21 and the status words at
`0x30010` and `0x30014`, and the window `0xd0008000..0xd0008fff` and status
word `0x30020` are unused; the IOAPIC has no input for a window past window 7.

**Table 2.8.** Guest-visible virtio features.

Expand Down
7 changes: 4 additions & 3 deletions doc/design/3.5-network.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ flowchart TB

The NIC occupies the fixed network slot, has one RX/TX queue pair, and
advertises an exact feature mask: the MAC-address feature and virtio
version 1 ([Table 2.8](2.5-device-topology-transport-and-discovery.md)). Its
interrupt is the one backend-specific value in the machine contract: IRQ 10 on
KVM and MSHV, and IRQ 5 on WHP. The portable network profile is mandatory and
version 1 ([Table 2.8](2.5-device-topology-transport-and-discovery.md)). It
interrupts on IOAPIC input 16 on every backend
([2.4.2](2.4-interrupt-and-io-architecture.md#242-interrupt-routing)). The
portable network profile is mandatory and
selects the same in-process data plane on every backend.

Table 3.12 shows an example of the static network identity that the host owns.
Expand Down
5 changes: 3 additions & 2 deletions doc/design/5.1-boot-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ reservations, and guest addresses as untrusted. It:
2. Loads validated loadable segments at their physical addresses and zeroes
their BSS tails.
3. Places an optional initramfs at the first 2-MiB boundary after the kernel.
4. Builds the Intel MP 1.4 processor, ISA bus, IOAPIC, and legacy interrupt
entries.
4. Builds the Intel MP 1.4 processor, ISA bus, and IOAPIC entries, and an
interrupt entry for every IOAPIC input: the ISA IRQs and inputs 16 through
23.
5. Builds Linux `boot_params` with the canonical e820 RAM and reservation map.
6. Installs a bootstrap GDT and 4-GiB identity page tables.
7. Enters the kernel in the 64-bit execution mode of x86-64 (long mode) with
Expand Down
34 changes: 25 additions & 9 deletions doc/openvmm-upstream-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,17 @@
- Mapped fork tip: `82686c983907738d44a6eb9e3322589b573c0074`, position 144. The graph, PR
summary, and commit mapping below cover positions 001-144 and the PR20 fixup at
position 161.
- NVX's pinned OpenVMM revision: `b9b1f8f6d41bcb7cc9b2d1ec7c836b30f5dc7718`, position 203. Positions
202 and 203, which microsoft/openvmm#4840 proposes for `feature/microvm/dev`, are two commits on
position 201, `d891dfd9d`, which microsoft/openvmm#4839 proposes, one commit on position 200,
- NVX's pinned OpenVMM revision: `35e76265fa8b63922edead112fcd5a38c59445c8`, position 205. Positions
204 and 205, which microsoft/openvmm#4842 proposes for `feature/microvm/dev`, are two commits on
position 203, `b9b1f8f6d`, which microsoft/openvmm#4840 proposes as two commits on position 201,
`d891dfd9d`, which microsoft/openvmm#4839 proposes, one commit on position 200,
`0ac254109`, the head of the `feature/microvm/release/v3.0` branch of microsoft/openvmm, to
which `feature/microvm/dev` also points. The v3.0 reorganization regrouped the 385 commits of
`feature/microvm/release/v2.4` that upstream had not landed into 200 commits: it folded
fixups into the commits that they fix and combined the commits after the mapped tip into commits
by concept, without changing what the chain adds to upstream; see
[The v3.0 reorganization](#the-v30-reorganization). The pin is 59 commits past the mapped tip
on the same first-parent chain. Positions 145-203, except the PR20 fixup at position
[The v3.0 reorganization](#the-v30-reorganization). The pin is 61 commits past the mapped tip
on the same first-parent chain. Positions 145-205, except the PR20 fixup at position
161, are not yet assigned to PRs; see
[Commits after the mapped tip](#commits-after-the-mapped-tip).
- Upstream main: `b2676222b383bf50c7022f6a5983920421469a47` (2026-10-31), the fork base.
Expand Down Expand Up @@ -516,7 +517,8 @@ b2676222b
-> network [195] -> device DMA [196-197]
-> command lines [198] -> KVP test [199-200]
-> sandbox block roles [201] -> virtio feature masks [202-203]
-> b9b1f8f6d (NVX pin)
-> IOAPIC interrupt lines [204-205]
-> 35e76265f (NVX pin)
```

For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main,
Expand All @@ -532,10 +534,10 @@ see [The v3.0 reorganization](#the-v30-reorganization).

## Commits after the mapped tip

The NVX pin extends the chain past position 144 without merges. Of the 59 later
The NVX pin extends the chain past position 144 without merges. Of the 61 later
commits, position 161 is a PR20 fixup, and the others are not yet classified or assigned to
PRs. Each of positions 145-200 combines the v2.4 commits that the table lists, which developed one
concept; "(part)" marks a v2.4 fixup that split across commits. Positions 201-203 follow the
concept; "(part)" marks a v2.4 fixup that split across commits. Positions 201-205 follow the
reorganization, so they combine none.

| Position | Commit | v2.4 positions | Topic |
Expand Down Expand Up @@ -599,6 +601,8 @@ reorganization, so they combine none.
| 201 | `d891dfd9d` | - | microVM: remove the runtime and custom sandbox block roles (microsoft/openvmm#4839) |
| 202 | `b0dda7fea` | - | microVM: give every virtio device class an exact feature mask (microsoft/openvmm#4840) |
| 203 | `b9b1f8f6d` | - | microVM: drop access-platform and the console size in microVM ABI 3 (microsoft/openvmm#4840) |
| 204 | `e13c1a644` | - | microVM: route every IOAPIC input in the MP table (microsoft/openvmm#4842) |
| 205 | `35e76265f` | - | microVM: give each virtio slot its own IOAPIC input in ABI 3 (microsoft/openvmm#4842) |

Some of these commits change mapped work rather than add to it. Position 160 removes the
legacy microVM clock paths, changing files that PR10 and PR17 also change, and makes manifest
Expand All @@ -615,7 +619,14 @@ names and layer counts of PR19's position 105's snapshot capture and restore. Po
fall-through among them, and that PR18's positions 095, 096, and 100, PR19's position 104, and
PR23's position 129 give the NIC, the filesystem, the sandbox blocks, and the consoles and record
in the snapshot contract. Position 203 also moves the microVM ABI version of PR17's position 087
to 3, which PR20's position 108 and PR25's position 137 check. Assigning the later
to 3, which PR20's position 108 and PR25's position 137 check. Position 204 extends the MP table
builder of PR14's position 065, which microsoft/openvmm#4554 proposes upstream, and its tests and
those of PR14's positions 066 and 067, to route every IOAPIC input, and passes the microVM's 24
inputs from the loader of PR15's position 075. Position 205 moves every virtio slot from the ISA
IRQ that PR16's position 080 gives it, and the NIC from the backend-specific IRQ that PR18's
position 096 selects, to the IOAPIC input of its window, which changes the interrupts and
discovery tokens that PR17's position 087 records in the snapshot contract and that the slots of
PR18, PR19, and PR23 use. Assigning the later
commits therefore means revisiting those PRs, not only appending new ones.

Position 191, by contrast, changes only `disk_blob`, which upstream main and the fork base
Expand Down Expand Up @@ -646,6 +657,11 @@ does not apply to upstream main and follows those PRs upstream. Positions 202 an
change only the microVM profile, its worker's device configuration, its snapshot contract, and
its management RPC profile, which PR16-PR20, PR23, and PR25 add, and their Guide pages: replaying
them on `b2676222b` conflicts in all nine of their files, so they follow those PRs upstream.
Positions 204 and 205 change the MP-table loader of PR14 and the microVM profile, worker, CLI, and
snapshot contract of PR15-PR19 and PR23, and their Guide pages: replaying them on `b2676222b`
conflicts in 16 of their 17 files, so they follow those PRs upstream. The 17th, the IOAPIC's
`chipset` tests, which position 204 extends with a save and restore of the inputs past the ISA
IRQs, applies to upstream main without conflicts.

## PR summary

Expand Down
4 changes: 4 additions & 0 deletions doc/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -660,6 +660,10 @@ run retired ABI-1 guests or snapshots.
ABI 3 offers no virtio device `VIRTIO_F_ACCESS_PLATFORM` and no console its
size, which ABI 2 offered (see
[Guest-Visible Capabilities](design/3.4-block-storage.md#342-guest-visible-capabilities)),
and gives each virtio device an IOAPIC input of its own past the ISA IRQs, the
same on every backend, where ABI 2 used ISA IRQs and another one for the NIC
on WHP (see
[Interrupt Routing](design/2.4-interrupt-and-io-architecture.md#242-interrupt-routing)),
so OpenVMM refuses to restore a snapshot of ABI 2, with `snapshot microVM ABI
version 2 is unsupported; this OpenVMM supports version 3`, and `nvx-rs`
refuses a template of ABI 2. Capture them again; to restore them as they are,
Expand Down
15 changes: 12 additions & 3 deletions doc/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -1074,8 +1074,8 @@ manifest, and the sandbox scratch template, which
that selects one of them fails before its first scenario when one of those
files is missing. Alpine remains the
default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject
the Alpine-control-only `sandbox-blocks`, `scratch-snapshot`, and
`virtio-features` scenarios and
the Alpine-control-only `interrupt-map`, `sandbox-blocks`, `scratch-snapshot`,
and `virtio-features` scenarios and
the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`,
`platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`,
`platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`,
Expand Down Expand Up @@ -1123,7 +1123,16 @@ The guest reports, for each device, the base of its MMIO window, its virtio
device ID, and the features that its driver negotiated, and the scenario
requires exactly the devices and features of
[Table 2.8](design/2.5-device-topology-transport-and-discovery.md), naming each
bit that a device has besides its documented ones or lacks. The scenarios that
bit that a device has besides its documented ones or lacks. The
`interrupt-map` scenario boots the same devices, with a share in the host
filesystem slot, and reports the interrupt that each device's driver
requested before and after it drives traffic through every device; each must
be the IOAPIC input of its device's window that
[Table 2.7](design/2.5-device-topology-transport-and-discovery.md) lists,
edge-triggered, unshared, and firing, and `/proc/interrupts` must count no APIC
error, misrouted IOAPIC interrupt, or spurious interrupt. A snapshot of a guest
with the NIC and the share must then restore with the same interrupts, which
fire again. The scenarios that
run microVMs or sandboxes as the public commands do, or that compile egress
policies, run [`nvx-rs`](#nvx-rs). The
command requires `build/vmlinux` (with `--debug-kernel`, `build/vmlinux-debug`
Expand Down
Loading
Loading