Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,9 +482,9 @@ describes the source definitions.
Restore completes the restoration of every device while the VM is stopped,
before any vCPU starts, even when the VM stays paused: invalid
device-private state, or a saved virtio-fs object that OpenVMM cannot
reopen, fails the restore itself rather than a later resume. The resume does
not wait for the devices to start their queues: a queue that fails to start
sets `DEVICE_NEEDS_RESET` on its device instead of failing the resume, as
reopen, fails the restore itself rather than a later resume. The resume
waits for the devices to start their queues before any vCPU runs: a queue
that fails to start fails the resume, as
[Snapshots](../../../user_guide/openvmm/snapshots.md#restoring-a-snapshot)
describes.

Expand Down
6 changes: 3 additions & 3 deletions Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,9 +335,9 @@ state, or a saved virtio-fs object that OpenVMM cannot reopen, fails the
restore rather than `ResumeVM`.
The readiness endpoint is a process-local orchestration attachment and is not
part of saved state. Each successful restore publishes one event; validation,
attachment, or worker-start failure publishes none. Neither `ResumeVM` nor the
event waits for the devices to start their queues: a queue that fails to start
sets `DEVICE_NEEDS_RESET` on its device instead of failing either, as
attachment, or worker-start failure publishes none. `ResumeVM` waits for the
devices to start their queues before any vCPU runs: a queue that fails to start
fails it, and the restore then publishes no event, as
[Snapshots](../../../user_guide/openvmm/snapshots.md#restoring-a-snapshot)
describes.

Expand Down
12 changes: 6 additions & 6 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,12 +179,12 @@ Restore completes the restoration of every device before any vCPU runs, even
when the VM stays paused: invalid device state, or a host resource that a
device's state refers to and that OpenVMM cannot reacquire, fails the restore
itself. Restoring starts no virtqueue. Each device starts its queues when the
VM first runs, and neither the resume nor the readiness event waits for them.
A queue that fails to start then, such as one whose network backend cannot
create its queues, fails neither: its transport sets `DEVICE_NEEDS_RESET`
instead, with a configuration-change notification if the driver has set
`DRIVER_OK`, and OpenVMM logs the failure. Once the guest resets the device,
setting `DRIVER_OK` starts its queues again.
VM first runs, and the resume waits for them before any vCPU runs. A queue that
fails to start then, such as one whose network backend cannot create its
queues, fails the resume: OpenVMM stops the devices that started again and
logs the failure, no vCPU runs, and the restore publishes no readiness event.
The devices keep their restored state, so a later resume starts their queues
again.

For a tiered microVM restore, and for a restore that replaces the network
policy of a snapshot without a tier, OpenVMM starts device workers with
Expand Down
8 changes: 8 additions & 0 deletions vm/chipset_device_resources/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ pub trait ConfigureChipsetDevice: Send {
#[async_trait]
trait DynChipsetDevice: ChipsetDevice + AsyncProtobufSaveRestore + InspectMut {
fn start(&mut self);
async fn confirm_start(&mut self) -> anyhow::Result<()>;
async fn quiesce_input(&mut self) -> anyhow::Result<()>;
async fn resume_input(&mut self) -> anyhow::Result<()>;
async fn stop(&mut self);
Expand All @@ -108,6 +109,9 @@ impl<T: ChangeDeviceState + ChipsetDevice + AsyncProtobufSaveRestore + InspectMu
fn start(&mut self) {
self.start()
}
async fn confirm_start(&mut self) -> anyhow::Result<()> {
ChangeDeviceState::confirm_start(self).await
}
async fn quiesce_input(&mut self) -> anyhow::Result<()> {
ChangeDeviceState::quiesce_input(self).await
}
Expand Down Expand Up @@ -152,6 +156,10 @@ impl ChangeDeviceState for ErasedChipsetDevice {
self.0.start()
}

async fn confirm_start(&mut self) -> anyhow::Result<()> {
self.0.confirm_start().await
}

async fn quiesce_input(&mut self) -> anyhow::Result<()> {
self.0.quiesce_input().await
}
Expand Down
5 changes: 3 additions & 2 deletions vm/devices/virtio/virtio/src/device.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,9 @@ pub trait VirtioDevice: InspectMut + Send {
/// (indices at 0).
///
/// An error fails the start. When the guest sets DRIVER_OK, the
/// transport fails DRIVER_OK; when the VM resumes, which does not wait
/// for queues to start, the transport marks the device as needing reset.
/// transport fails DRIVER_OK; when the VM starts, the transport fails the
/// confirmation of the start, and the VM stops the device again before
/// any vCPU runs, which keeps the progress of the queues that did start.
/// Acquire the host resources that restored state refers to in
/// [`restore_device`](Self::restore_device) instead, so that a missing
/// one fails the restore.
Expand Down
9 changes: 9 additions & 0 deletions vm/devices/virtio/virtio/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
mod dma;
mod interrupt;
mod saved_state;
mod start;

use crate::DeviceTraits;
use crate::DynVirtioDevice;
Expand Down Expand Up @@ -4334,6 +4335,8 @@ trait TestTransport {
fn stop(&mut self) -> impl Future<Output = ()>;
/// Drive ChangeDeviceState::start().
fn start(&mut self);
/// Drive ChangeDeviceState::confirm_start().
fn confirm_start(&mut self) -> impl Future<Output = anyhow::Result<()>>;
fn teardown(self) -> impl Future<Output = ()>;
}

Expand Down Expand Up @@ -4409,6 +4412,9 @@ impl TestTransport for MmioTestTransport {
fn start(&mut self) {
ChangeDeviceState::start(&mut self.dev);
}
async fn confirm_start(&mut self) -> anyhow::Result<()> {
ChangeDeviceState::confirm_start(&mut self.dev).await
}
async fn teardown(self) {
self.dev.teardown().await;
}
Expand Down Expand Up @@ -4550,6 +4556,9 @@ impl TestTransport for PciTestTransport {
fn start(&mut self) {
ChangeDeviceState::start(&mut self.dev);
}
async fn confirm_start(&mut self) -> anyhow::Result<()> {
ChangeDeviceState::confirm_start(&mut self.dev).await
}
async fn teardown(self) {
self.dev.teardown().await;
}
Expand Down
39 changes: 23 additions & 16 deletions vm/devices/virtio/virtio/src/tests/saved_state.rs
Original file line number Diff line number Diff line change
Expand Up @@ -877,29 +877,36 @@ async fn fail_resume(transport: &mut impl TestTransport, probe: &PrivateStatePro
transport.start();
}

async fn verify_failed_resume_marks_device_needs_reset(
/// A resume whose second queue fails to start fails its confirmation and
/// leaves the device as it was: the queue that started is stopped again, and
/// the device is not marked as needing reset, since the guest does not run
/// until a start succeeds. The VM's next start then starts both queues.
async fn verify_failed_resume_fails_its_confirmation(
transport: &mut impl TestTransport,
probe: &PrivateStateProbe,
) {
fail_resume(transport, probe).await;
yield_and_poll(transport).await;
let status = transport.read_status();
assert_ne!(
status & VIRTIO_DEVICE_NEEDS_RESET,
0,
"a failed start must mark the device as needing reset"
let error = transport.confirm_start().await.unwrap_err();
assert!(
format!("{error:#}").contains("intentional queue start failure"),
"{error:#}"
);
let status = transport.read_status();
assert_eq!(status & VIRTIO_DEVICE_NEEDS_RESET, 0);
assert_ne!(status & VIRTIO_DRIVER_OK, 0);
// The queue that started before the failure is stopped again.
assert_eq!(probe.running_count.load(Ordering::Relaxed), 0);

// The guest's reset recovers the device.
// The VM stops the device again, and its next start succeeds.
transport.stop().await;
assert_eq!(transport.read_status() & VIRTIO_DEVICE_NEEDS_RESET, 0);
probe.fail_start_at.store(usize::MAX, Ordering::Relaxed);
transport.write_status_zero();
yield_and_poll(transport).await;
assert_eq!(transport.read_status(), 0);
transport.start();
transport.confirm_start().await.unwrap();
assert_eq!(probe.running_count.load(Ordering::Relaxed), 2);
}

/// A resume that nothing confirms leaves the failure to the next stop, which
/// marks the device as needing reset.
async fn verify_stop_applies_failed_start(
transport: &mut impl TestTransport,
probe: &PrivateStateProbe,
Expand All @@ -911,17 +918,17 @@ async fn verify_stop_applies_failed_start(
}

#[async_test]
async fn failed_resume_marks_device_needs_reset_mmio(driver: DefaultDriver) {
async fn failed_resume_fails_its_confirmation_mmio(driver: DefaultDriver) {
let (device, probe) = PrivateStateTestDevice::new(0, 2, false);
let mut transport = MmioTestTransport::new(Box::new(device), &driver, 2);
verify_failed_resume_marks_device_needs_reset(&mut transport, &probe).await;
verify_failed_resume_fails_its_confirmation(&mut transport, &probe).await;
}

#[async_test]
async fn failed_resume_marks_device_needs_reset_pci(driver: DefaultDriver) {
async fn failed_resume_fails_its_confirmation_pci(driver: DefaultDriver) {
let (device, probe) = PrivateStateTestDevice::new(0, 2, false);
let mut transport = PciTestTransport::new(Box::new(device), &driver, 2);
verify_failed_resume_marks_device_needs_reset(&mut transport, &probe).await;
verify_failed_resume_fails_its_confirmation(&mut transport, &probe).await;
}

#[async_test]
Expand Down
Loading
Loading