Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Guide/src/reference/devices/firmware/linux_direct.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ uncompressed ELF kernel, but selects an ACPI-free platform mode. OpenVMM places
an Intel MP 1.4 floating pointer at `0x0`, its configuration table at `0x400`,
the bootstrap GDT at `0x1000`, `boot_params` at `0x2000`, and the command line
in the 64-KiB region starting at `0x20000`. The MP table describes contiguous
APIC IDs, the ISA bus, the IOAPIC, and legacy IRQ routing. No RSDP, ACPI table,
APIC IDs, the ISA bus, the IOAPIC, and the routing of each of the IOAPIC's 24
inputs: the ISA IRQs, and inputs 16 through 23, which the PIC does not see, as
active-high, edge-triggered interrupts of their own numbers. No RSDP, ACPI table,
SMBIOS anchor, or SMBIOS structure table is imported.

On AArch64, pass the uncompressed `Image` file (not `Image.gz`).
Expand Down
2 changes: 1 addition & 1 deletion Guide/src/reference/devices/virtio/virtio-fs.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ fixes the remaining guest-visible configuration:
| Stable ID | `fs:microvm0` |
| Tag | `microvm` |
| Transport | virtio-mmio at `0xd0001000` |
| Interrupt | IRQ 6 |
| Interrupt | IRQ 17, an IOAPIC input past the ISA IRQs |
| Queues | One high-priority and one request queue |
| Features | Indirect descriptors, event index, version 1 |
| DAX window | None |
Expand Down
38 changes: 22 additions & 16 deletions Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,17 +52,20 @@ describes the source definitions.
rejects a longer command line, or one with an embedded NUL, rather than let
the guest truncate it.

One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 6 and remains
One virtio-fs slot is exposed at MMIO `0xd0001000`, IRQ 17 and remains
dormant when neither `--mount` nor `--mount-aggregate` is given; either one
binds HostFs to it;
one optional `--virtio-console <BACKEND>` is exposed at MMIO `0xd0002000`,
IRQ 7 as the boot/log console (`hvc1`); and `--microvm-sandbox-block`
exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 4 and the fixed
`scratch` slot at MMIO `0xd0006000`, IRQ 11. The windows between them and
IRQs 9 and 12 stay unassigned. The profile also reserves MMIO `0xd0007000`,
IRQ 3 for the dedicated control console (`hvc2`), selected by the host-owned
`nvx_control_tty=hvc2` token. The reserved slot and resource identity do not
expose a live control endpoint. Both consoles
IRQ 18 as the boot/log console (`hvc1`); and `--microvm-sandbox-block`
exposes the fixed `distro` slot at MMIO `0xd0003000`, IRQ 19 and the fixed
`scratch` slot at MMIO `0xd0006000`, IRQ 22. The windows between them and
IRQs 20 and 21 stay unassigned. The profile also reserves MMIO `0xd0007000`,
IRQ 23 for the dedicated control console (`hvc2`), selected by the host-owned
`nvx_control_tty=hvc2` token. Each virtio-mmio window, `0xd0000000 + n *
0x1000`, interrupts on its own IOAPIC input, `16 + n`, with active-high edges
on every backend; the PIC sees none of these inputs, and the ISA IRQs carry
only the PIT (IRQ 0) and the RTC (IRQ 8). The reserved slot and resource
identity do not expose a live control endpoint. Both consoles
remain virtio-console devices from the guest's perspective. Ordinary
`--virtio-blk` is rejected. All use split rings, and each offers an exact
feature mask of virtio 1.0 features: the NIC its MAC address; the block
Expand All @@ -71,14 +74,15 @@ describes the source definitions.
notification suppression. No device offers `VIRTIO_F_ACCESS_PLATFORM`, since
the microVM has no IOMMU, and no console offers its size. Firmware, ACPI,
SMBIOS, PCI, VMBus, UARTs, graphics, isolation, nested virtualization, and
other devices are rejected. Linux discovers contiguous APIC IDs and the
IOAPIC from Intel MP 1.4 tables at `0x0` and `0x400`; `boot_params` is at
other devices are rejected. Linux discovers contiguous APIC IDs, the IOAPIC,
and the routing of its 24 inputs from Intel MP 1.4 tables at `0x0` and
`0x400`; `boot_params` is at
`0x2000`, the command line starts at `0x20000`, and no ACPI or SMBIOS data is
exposed. Host-driven save/restore, pulse-save/restore, and worker restart
remain unavailable.

`microvm` may also expose a dedicated control virtio-console at MMIO
`0xd0007000`, IRQ 3. It requires the boot virtio-console, preserves
`0xd0007000`, IRQ 23. It requires the boot virtio-console, preserves
`console=hvc1`, and publishes `nvx_control_tty=hvc2`. The profile fixes
boot-before-control discovery order and rejects user overrides that could
change it.
Expand All @@ -102,11 +106,13 @@ describes the source definitions.
as `microvm-v2`; the `microvm-v2` selector and the former ABI-v1 behavior are
removed. Snapshots record microVM ABI value 3 and boot layout value 2. ABI
value 3 offers no device `VIRTIO_F_ACCESS_PLATFORM` and no console
`VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, so restore refuses a
`VIRTIO_CONSOLE_F_SIZE`, which value 2 offered, and gives each virtio-mmio
slot its own IOAPIC input past the ISA IRQs on every backend, where value 2
used ISA IRQs and put the NIC on another one on WHP, so restore refuses a
snapshot of value 2, which must be recaptured.
```
* `--net <IPv4/PREFIX>`: With `--machine microvm`, attach one virtio-net NIC
at MMIO `0xd0000000`. KVM and MSHV use IRQ 10; WHP uses IRQ 5. Prefixes
at MMIO `0xd0000000`, IRQ 16, on every backend. Prefixes
`/1` through `/30` are accepted. The first usable subnet address becomes
the gateway; network, broadcast, and gateway addresses cannot be assigned
to the guest. Guest and gateway MAC addresses are derived as
Expand Down Expand Up @@ -266,7 +272,7 @@ describes the source definitions.
protocol does not retain pre-capture endpoint completions; restored guest
software must establish new host-side flows.
* `--mount <GUEST_TARGET,HOST_PATH[,ro|rw]>`: With `--machine microvm`, attach
one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 6, with tag `microvm`.
one no-DAX HostFs device at MMIO `0xd0001000`, IRQ 17, with tag `microvm`.
`--mount` may appear once. The default mode is read-only; `rw` must be
explicit. The guest target ends at the first comma, and the mode follows
the last comma, so a host path may contain commas when the mode is given.
Expand Down Expand Up @@ -1006,7 +1012,7 @@ Serial devices can be configured to appear as different devices inside the guest
behaves normally.
* `--virtio-console <BACKEND>`: Expose a virtio console device. It normally
appears as `/dev/hvc0`. Under `--machine microvm`, it occupies fixed MMIO
`0xd0002000`, IRQ 7, and is selected as `/dev/hvc1`; the raw portb path
`0xd0002000`, IRQ 18, and is selected as `/dev/hvc1`; the raw portb path
remains available as `hvc0` for early output and recovery.

A microVM accepts these explicit attachment policies:
Expand Down Expand Up @@ -1034,7 +1040,7 @@ Serial devices can be configured to appear as different devices inside the guest

* `--microvm-control-console <BACKEND>`: With `--machine microvm`, expose a
second independent single-port virtio console at fixed MMIO `0xd0007000`, IRQ
3. `--virtio-console` is required on a fresh boot and remains the only kernel
23. `--virtio-console` is required on a fresh boot and remains the only kernel
console. The control device normally appears as the profile-owned
`nvx_control_tty=hvc2`.

Expand Down
7 changes: 4 additions & 3 deletions Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ On cold boot, `DevicesConfig.virtio_consoles` may hold one entry, which
configures a microVM Unix socket or named-pipe endpoint. Listener mode
recreates the path on restore.
Client mode is required and uses a five-second connection timeout
before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 7, and selects
before vCPUs start. The device uses MMIO `0xd0002000`, IRQ 18, and selects
`hvc1`; its canonical path and policy become the stable restore attachment.

`DevicesConfig.virtiofs_config` may bind one HostFs attachment to the fixed
Expand All @@ -232,7 +232,7 @@ target, and access mode, and the live root must retain its saved identity;
the snapshot's denied, allowed, and writable paths apply again. A snapshot
captured with the slot dormant may
omit the attachment or supply a new one; after `ResumeVM`, the guest explicitly
mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 6, one
mounts tag `microvm`. The fixed device uses MMIO `0xd0001000`, IRQ 17, one
request queue, and no DAX window. The management API binds only a single
directory: it cannot attach an aggregate of several host directories, as the
command line's `--mount-aggregate` does, and it cannot restore a snapshot of
Expand Down Expand Up @@ -264,7 +264,8 @@ currently blockless; role-bearing sandbox blocks remain CLI-only. Snapshots
record microVM ABI value 3 and boot layout value 2, which identifies the
Linux-direct MP-table layout. Snapshots of ABI values 1 and 2 are unsupported:
value 2 offered `VIRTIO_F_ACCESS_PLATFORM` on every device but the NIC, and the
console size on the consoles.
console size on the consoles, and put the virtio devices on ISA IRQs, where
value 3 gives each one its own IOAPIC input past them.

`VMConfig.cpu_profile` selects the microVM's CPU profile as
[`--cpu-profile`](cli.md) does, with the same values and checks: `auto`, the
Expand Down
15 changes: 9 additions & 6 deletions Guide/src/user_guide/openvmm/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,12 @@ locations:

| Role | Access | MMIO address | IRQ |
| --- | --- | ---: | ---: |
| `distro` | read-only | `0xd0003000` | 4 |
| `scratch` | writable | `0xd0006000` | 11 |
| `distro` | read-only | `0xd0003000` | 19 |
| `scratch` | writable | `0xd0006000` | 22 |

The windows `0xd0004000` and `0xd0005000` between them, and IRQs 9 and 12,
stay unassigned, so that no slot moves.
The windows `0xd0004000` and `0xd0005000` between them, and their IRQs 20 and
21, stay unassigned, so that no slot moves. Each IRQ is an IOAPIC input past
the ISA IRQs, which the PIC does not see.

Use `--microvm-sandbox-block ROLE:DISK`, in the order shown. The `distro`
role requires the normal disk `,ro` option and a non-empty topology must end
Expand Down Expand Up @@ -94,8 +95,10 @@ Custom socket, SMT, APIC, x2APIC, and NUMA options are rejected.
The microVM uses fixed virtio device slots and sandbox block roles. Its
persisted ABI is value 3 and its boot layout value 2. Each fixed virtio device
offers an exact feature mask, which never includes `VIRTIO_F_ACCESS_PLATFORM`,
since the microVM has no IOMMU, nor packed virtqueues. The
MP floating pointer begins at `0x0`, the MP configuration table at `0x400`,
since the microVM has no IOMMU, nor packed virtqueues, and interrupts on an
IOAPIC input of its own past the ISA IRQs, the same on every backend. The
MP floating pointer begins at `0x0`, the MP configuration table, which routes
all 24 IOAPIC inputs, at `0x400`,
the boot GDT at `0x1000`, and the Linux zero page at `0x2000`. No ACPI MADT or
SMBIOS data is exposed.

Expand Down
2 changes: 1 addition & 1 deletion Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -606,7 +606,7 @@ immediately with a clear error if any active device does not support it.
`--processors`. MicroVM restore reads them authoritatively from the manifest
and rejects overrides. Persisted microVM ABI value 3 and boot-layout value 2
are supported; a snapshot of ABI value 2, whose devices offered
`VIRTIO_F_ACCESS_PLATFORM`, must be recaptured.
`VIRTIO_F_ACCESS_PLATFORM` and used ISA IRQs, must be recaptured.
- MicroVM restore requires the backend, CPU generation, and CPU profile of
the capture, and an OpenVMM build that computes the same effective CPUID.
A host profile's snapshot, including one of a cold boot whose `auto` fell
Expand Down
32 changes: 17 additions & 15 deletions openvmm/openvmm_core/src/worker/dispatch/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ pub(super) fn load_linux_x86_mptable(
&vm.gm,
&apic_ids,
&openvmm_defs::microvm::MICROVM_LEVEL_TRIGGERED_IRQS,
openvmm_defs::microvm::MICROVM_IOAPIC_INPUTS,
&reserved_memory_ranges,
)?)
}
Expand Down Expand Up @@ -484,7 +485,7 @@ fn virtio_mmio_config(
let (start, irq, features) = match id {
"virtio-net" => (
openvmm_defs::microvm::MICROVM_VIRTIO_NET_MMIO_BASE,
openvmm_defs::microvm::microvm_virtio_net_irq(None)?,
openvmm_defs::microvm::MICROVM_VIRTIO_NET_IRQ,
openvmm_defs::microvm::MICROVM_VIRTIO_NET_FEATURES,
),
// The configuration orders virtio-fs devices by slot.
Expand Down Expand Up @@ -805,8 +806,7 @@ pub(super) fn x86_topology_builder(
}
}

// The microVM runs on Linux and Windows hosts alone, where its NIC has an
// interrupt.
// The microVM runs on Linux and Windows hosts alone.
#[cfg(all(test, any(target_os = "linux", windows)))]
mod tests {
use super::virtio_mmio_config;
Expand Down Expand Up @@ -916,24 +916,25 @@ mod tests {
];
let mut sandbox_block_index = 0;
let mut filesystem_index = 0;
// Each device in the order of the slots of its kind, with its window
// and the features that the guest sees: never access-platform, which
// the transport offers every device, nor the console size, which the
// virtio-console device offers.
let devices: [(&str, u64, &[u32]); 6] = [
("virtio-net", 0xd000_0000, &[5, 32]),
("virtiofs", 0xd000_1000, &[28, 29, 32]),
("virtio-console", 0xd000_2000, &[28, 29, 32]),
("virtio-blk", 0xd000_3000, &[2, 5, 6, 9, 10, 28, 29, 32]),
("virtio-blk", 0xd000_6000, &[2, 6, 9, 10, 28, 29, 32]),
// Each device in the order of the slots of its kind, with its window,
// its IOAPIC input, and the features that the guest sees: never
// access-platform, which the transport offers every device, nor the
// console size, which the virtio-console device offers.
let devices: [(&str, u64, u32, &[u32]); 6] = [
("virtio-net", 0xd000_0000, 16, &[5, 32]),
("virtiofs", 0xd000_1000, 17, &[28, 29, 32]),
("virtio-console", 0xd000_2000, 18, &[28, 29, 32]),
("virtio-blk", 0xd000_3000, 19, &[2, 5, 6, 9, 10, 28, 29, 32]),
("virtio-blk", 0xd000_6000, 22, &[2, 6, 9, 10, 28, 29, 32]),
(
openvmm_defs::microvm::MICROVM_VIRTIO_CONTROL_CONSOLE_ID,
0xd000_7000,
23,
&[28, 29, 32],
),
];
for (id, mmio_base, bits) in devices {
let (start, len, _irq, features, interrupt_mode) = virtio_mmio_config(
for (id, mmio_base, input, bits) in devices {
let (start, len, irq, features, interrupt_mode) = virtio_mmio_config(
id,
&blocks,
&mut sandbox_block_index,
Expand All @@ -942,6 +943,7 @@ mod tests {
)
.unwrap();
assert_eq!(start, mmio_base, "{id}");
assert_eq!(irq, input, "{id}");
let mut device = VirtioMmioDevice::new_with_disabled_features_and_interrupt_mode(
Box::new(EveryFeatureDevice),
&driver_source.simple(),
Expand Down
2 changes: 2 additions & 0 deletions openvmm/openvmm_core/src/worker/vm_loaders/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ pub fn load_linux_x86_mptable(
gm: &GuestMemory,
apic_ids: &[u32],
level_triggered_irqs: &[u32],
ioapic_inputs: u8,
reserved_memory_ranges: &[MemoryRange],
) -> Result<InitialLoad<X86Register>, LinuxError> {
if !matches!(cfg.isolation, KernelIsolationConfig::None) {
Expand Down Expand Up @@ -54,6 +55,7 @@ pub fn load_linux_x86_mptable(
&loader::mptable::MpTableConfig {
apic_ids,
level_triggered_irqs,
ioapic_inputs,
},
reserved_memory_ranges,
)
Expand Down
Loading
Loading