Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,11 @@ describes the source definitions.
Restore completes the restoration of every device while the VM is stopped,
before any vCPU starts, even when the VM stays paused: invalid
device-private state, or a saved virtio-fs object that OpenVMM cannot
reopen, fails the restore itself rather than a later resume.
reopen, fails the restore itself rather than a later resume. The resume does
not wait for the devices to start their queues: a queue that fails to start
sets `DEVICE_NEEDS_RESET` on its device instead of failing the resume, as
[Snapshots](../../../user_guide/openvmm/snapshots.md#restoring-a-snapshot)
describes.

When the snapshot contains virtio-net, restore also requires
`--network-profile portable`; the snapshot's profile and canonical egress
Expand Down
6 changes: 5 additions & 1 deletion Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,11 @@ state, or a saved virtio-fs object that OpenVMM cannot reopen, fails the
restore rather than `ResumeVM`.
The readiness endpoint is a process-local orchestration attachment and is not
part of saved state. Each successful restore publishes one event; validation,
attachment, or worker-start failure publishes none.
attachment, or worker-start failure publishes none. Neither `ResumeVM` nor the
event waits for the devices to start their queues: a queue that fails to start
sets `DEVICE_NEEDS_RESET` on its device instead of failing either, as
[Snapshots](../../../user_guide/openvmm/snapshots.md#restoring-a-snapshot)
describes.

`VMConfig.MICROVM` is numeric value 1 and accepts exactly 1, 2, 4, or 8
processors. RPC construction is currently blockless; role-bearing sandbox
Expand Down
11 changes: 11 additions & 0 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,17 @@ releasing gated input. The peer must accept and read while resume is in
progress; on Windows, flush completion waits until the named-pipe peer consumes
the complete frame.

Restore completes the restoration of every device before any vCPU runs, even
when the VM stays paused: invalid device state, or a host resource that a
device's state refers to and that OpenVMM cannot reacquire, fails the restore
itself. Restoring starts no virtqueue. Each device starts its queues when the
VM first runs, and neither the resume nor the readiness event waits for them.
A queue that fails to start then, such as one whose network backend cannot
create its queues, fails neither: its transport sets `DEVICE_NEEDS_RESET`
instead, with a configuration-change notification if the driver has set
`DRIVER_OK`, and OpenVMM logs the failure. Once the guest resets the device,
setting `DRIVER_OK` starts its queues again.

For a tiered microVM restore, and for a restore that replaces the network
policy of a snapshot without a tier, OpenVMM starts device workers with
network, control, and filesystem input gated. The guest performs post-restore
Expand Down
7 changes: 7 additions & 0 deletions vm/devices/virtio/virtio/src/device.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,13 @@ pub trait VirtioDevice: InspectMut + Send {
/// `initial_state` provides restored queue indices for save/restore
/// or vhost-user `SET_VRING_BASE`. If `None`, the queue starts fresh
/// (indices at 0).
///
/// An error fails the start. When the guest sets DRIVER_OK, the
/// transport fails DRIVER_OK; when the VM resumes, which does not wait
/// for queues to start, the transport marks the device as needing reset.
/// Acquire the host resources that restored state refers to in
/// [`restore_device`](Self::restore_device) instead, so that a missing
/// one fails the restore.
fn start_queue(
&mut self,
idx: u16,
Expand Down
87 changes: 79 additions & 8 deletions vm/devices/virtio/virtio/src/tests/saved_state.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

//! Tests for device-private saved state and restore: private-state round trips,
//! restore validation, the restore of private state into the stopped device,
//! and the start, reset, and removal that follow it.
//! and the start, reset, and removal that follow it. Also tests that ordinary
//! stops and starts, which restore nothing, keep queue progress.

use super::*;
use std::sync::atomic::AtomicBool;
Expand Down Expand Up @@ -782,6 +783,17 @@ async fn pci_reset_discards_restored_queue_progress(driver: DefaultDriver) {
assert_eq!(probe.initial_used.load(Ordering::Relaxed), usize::MAX);
}

/// Waits for the device task of a dropped transport to drop its device.
async fn assert_device_dropped(probe: &PrivateStateProbe) {
for _ in 0..10 {
if probe.dropped.load(Ordering::Relaxed) {
break;
}
yield_now().await;
}
assert!(probe.dropped.load(Ordering::Relaxed));
}

#[async_test]
async fn mmio_removal_before_first_start_releases_device(driver: DefaultDriver) {
use vmcore::save_restore::AsyncSaveRestore;
Expand All @@ -794,13 +806,24 @@ async fn mmio_removal_before_first_start_releases_device(driver: DefaultDriver)
assert!(!probe.dropped.load(Ordering::Relaxed));

drop(destination);
for _ in 0..10 {
if probe.dropped.load(Ordering::Relaxed) {
break;
}
yield_now().await;
}
assert!(probe.dropped.load(Ordering::Relaxed));
assert_device_dropped(&probe).await;
assert_eq!(probe.start_count.load(Ordering::Relaxed), 0);
}

#[async_test]
async fn pci_removal_before_first_start_releases_device(driver: DefaultDriver) {
use vmcore::save_restore::AsyncSaveRestore;

let test_mem = VirtioTestMemoryAccess::new();
let saved = save_active_private_pci(&driver, &test_mem).await;
let (device, probe) = PrivateStateTestDevice::new(0, 1, false);
let mut destination =
VirtioPciTestDevice::new_with_device(&driver, &test_mem, Box::new(device));
destination.pci_device.restore(saved).await.unwrap();
assert!(!probe.dropped.load(Ordering::Relaxed));

drop(destination);
assert_device_dropped(&probe).await;
assert_eq!(probe.start_count.load(Ordering::Relaxed), 0);
}

Expand Down Expand Up @@ -915,6 +938,54 @@ async fn stop_applies_failed_start_pci(driver: DefaultDriver) {
verify_stop_applies_failed_start(&mut transport, &probe).await;
}

/// Stops and starts an active device twice, as ordinary pauses and resumes
/// do: each start resumes both queues from the progress of the stop before
/// it, and nothing restores or resets the device.
async fn verify_stop_start_keeps_queue_progress(
transport: &mut impl TestTransport,
probe: &PrivateStateProbe,
) {
transport.write_driver_ok();
yield_and_poll(transport).await;
assert_eq!(probe.running_count.load(Ordering::Relaxed), 2);
probe.value.store(42, Ordering::Relaxed);
transport.stop().await;
assert_eq!(probe.running_count.load(Ordering::Relaxed), 0);

for starts in [4, 6] {
probe.initial_avail.store(usize::MAX, Ordering::Relaxed);
probe.initial_used.store(usize::MAX, Ordering::Relaxed);
transport.start();
// The stop waits for the start, then stops the queues again.
transport.stop().await;
assert_eq!(probe.start_count.load(Ordering::Relaxed), starts);
assert_eq!(probe.running_count.load(Ordering::Relaxed), 0);
assert_eq!(probe.initial_avail.load(Ordering::Relaxed), 3);
assert_eq!(probe.initial_used.load(Ordering::Relaxed), 1);
}

assert_eq!(probe.restore_count.load(Ordering::Relaxed), 0);
assert_eq!(probe.reset_count.load(Ordering::Relaxed), 0);
assert_eq!(probe.value.load(Ordering::Relaxed), 42);
let status = transport.read_status();
assert_ne!(status & VIRTIO_DRIVER_OK, 0);
assert_eq!(status & VIRTIO_DEVICE_NEEDS_RESET, 0);
}

#[async_test]
async fn stop_start_keeps_queue_progress_mmio(driver: DefaultDriver) {
let (device, probe) = PrivateStateTestDevice::new(0, 2, false);
let mut transport = MmioTestTransport::new(Box::new(device), &driver, 2);
verify_stop_start_keeps_queue_progress(&mut transport, &probe).await;
}

#[async_test]
async fn stop_start_keeps_queue_progress_pci(driver: DefaultDriver) {
let (device, probe) = PrivateStateTestDevice::new(0, 2, false);
let mut transport = PciTestTransport::new(Box::new(device), &driver, 2);
verify_stop_start_keeps_queue_progress(&mut transport, &probe).await;
}

#[async_test]
async fn mmio_restore_skips_disabled_zero_address_ring(driver: DefaultDriver) {
use vmcore::save_restore::AsyncSaveRestore;
Expand Down
28 changes: 27 additions & 1 deletion vm/devices/virtio/virtio_net/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,11 @@ impl VirtioDevice for Device {
);

if first_pair {
if let Err(err) = self.create_endpoint_queues().await {
self.coordinator.remove();
self.pairs[pair_idx] = QueuePairState::Empty;
return Err(err);
}
self.coordinator.start();
}
}
Expand Down Expand Up @@ -748,11 +753,27 @@ impl Device {
num_queues,
enabled_rx_offloads: negotiated_rx_offload_support(negotiated_features),
restart: true,
endpoint_queues_created: false,
input_quiesced: self.lifecycle.input_quiesced,
},
);
}

/// Creates the endpoint queues of the coordinator that `start_queue` has
/// just inserted, before the coordinator starts, so that an endpoint that
/// cannot provide them fails the queue start instead of leaving the device
/// without endpoint queues.
async fn create_endpoint_queues(&mut self) -> anyhow::Result<()> {
let (state, coordinator) = self.coordinator.get_mut();
let coordinator = coordinator.context("virtio-net coordinator state is unavailable")?;
coordinator
.restart_queues(state)
.await
.context("failed to create the network endpoint queues")?;
coordinator.endpoint_queues_created = true;
Ok(())
}

/// Allocates and inserts a worker.
///
/// The coordinator must be stopped.
Expand Down Expand Up @@ -789,6 +810,9 @@ struct Coordinator {
num_queues: u16,
enabled_rx_offloads: RxOffloadSupport,
restart: bool,
/// Whether `start_queue` has created the endpoint queues that the first
/// restart would otherwise create.
endpoint_queues_created: bool,
input_quiesced: bool,
}

Expand Down Expand Up @@ -844,7 +868,9 @@ impl Coordinator {
stop.until_stopped(self.stop_workers()).await?;
// The queue restart operation is not restartable, so do not
// poll on `stop` here.
if let Err(err) = self.restart_queues(state).await {
if !std::mem::take(&mut self.endpoint_queues_created)
&& let Err(err) = self.restart_queues(state).await
{
tracing::error!(
error = &err as &dyn std::error::Error,
"failed to restart queues"
Expand Down
1 change: 1 addition & 0 deletions vm/devices/virtio/virtio_net/src/tests.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

mod activation;
mod egress;
mod harness;
mod quiesce;
Expand Down
Loading
Loading