Skip to content

openvmm: failed snapshot rollback leaves restarted devices running #1149

Closed

Description

@tangruize

If one device fails to restart during snapshot rollback, successfully restarted siblings remain running even though the VM reports that its state-unit set is stopped. Teardown can then skip their Stop.

In vmm_core/state_unit/src/quiesce.rs, StateUnits::resume_after_failed_save records successful restarts directly in the unit table:

        let mut inner = self.inner.lock();
        for (name, id, result) in results {
            let Some(unit) = inner.units.get_mut(&id) else {
                continue;
            };
            match result {
                StartResult::Started => unit.state = State::Running,
                StartResult::DependencyFailed => {
                    unit.state = State::Stopped;
                    failures.push(format!("{name}: a dependency did not restart"));

Other result arms record failures, and the method returns an error before setting the set's running flag. It does not undo the successful Starts. LoadedVm::resume_after_failed_snapshot consequently leaves the VM marked stopped, and LoadedVm::pause returns without stopping those running units.

Trigger

Start independent devices, quiesce them successfully, and encounter a capture failure for which rollback is allowed. Send ResumeAfterFailedSnapshot with one device able to restart and another failing its Start. The rollback reports failure, but the successfully restarted device continues running. The vCPU snapshot boundary still holds the guest; it does not stop that device activity.

Fix

fix.patch collects successful restart IDs and stops that subset in reverse dependency order before returning the rollback error, using the existing start-rollback cleanup mechanism. The unit-table lock is released before awaiting cleanup.

Cleanup Stop remains unbounded and retains the existing dead-channel failure behavior. This repairs partial-restart cleanup, not the separate problem of accepting later requests against terminal units.

Activity

  1. tangruize commented on Nov 1, 2026

    @tangruize
    ContributorAuthor

    Proposed patch

    Patch against OpenVMM 998b9f2402050cd16d1faa897bc831ef1434be48. Save the contents below as fix.patch and run git apply fix.patch.

    fix.patch
    diff --git a/vmm_core/state_unit/src/quiesce.rs b/vmm_core/state_unit/src/quiesce.rs
    --- a/vmm_core/state_unit/src/quiesce.rs
    +++ b/vmm_core/state_unit/src/quiesce.rs
    @@ -237,28 +237,49 @@
     
             let results = join_all(operations).await;
             let mut failures = Vec::new();
    -        let mut inner = self.inner.lock();
    -        for (name, id, result) in results {
    -            let Some(unit) = inner.units.get_mut(&id) else {
    -                continue;
    -            };
    -            match result {
    -                StartResult::Started => unit.state = State::Running,
    -                StartResult::DependencyFailed => {
    -                    unit.state = State::Stopped;
    -                    failures.push(format!("{name}: a dependency did not restart"));
    +        let mut started_ids = Vec::new();
    +        {
    +            let mut inner = self.inner.lock();
    +            for (name, id, result) in results {
    +                let Some(unit) = inner.units.get_mut(&id) else {
    +                    continue;
    +                };
    +                match result {
    +                    StartResult::Started => {
    +                        unit.state = State::Running;
    +                        started_ids.push(id);
    +                    }
    +                    StartResult::DependencyFailed => {
    +                        unit.state = State::Stopped;
    +                        failures.push(format!("{name}: a dependency did not restart"));
    +                    }
    +                    StartResult::Failed(error) => {
    +                        unit.state = State::QuiesceUncertain;
    +                        failures.push(format!("{name}: {error:#}"));
    +                    }
    +                    StartResult::Uncertain(error) => {
    +                        unit.state = State::QuiesceUncertain;
    +                        failures.push(format!("{name}: {error:#}"));
    +                    }
                     }
    -                StartResult::Failed(error) => {
    -                    unit.state = State::QuiesceUncertain;
    -                    failures.push(format!("{name}: {error:#}"));
    -                }
    -                StartResult::Uncertain(error) => {
    -                    unit.state = State::QuiesceUncertain;
    -                    failures.push(format!("{name}: {error:#}"));
    -                }
    -            }
    -        }
    -        drop(inner);
    +            }
    +        }
    +
    +        if !failures.is_empty() && !started_ids.is_empty() {
    +            // Stop the units that did restart again, as `start_with_rollback`
    +            // does, so that no unit runs while the set reports stopped.
    +            self.run_op(
    +                "failed_snapshot_rollback",
    +                Some(&started_ids),
    +                State::Running,
    +                State::Stopping,
    +                State::Stopped,
    +                StateRequest::Stop,
    +                |_, _| Some(()),
    +                |unit| Some(&unit.dependents),
    +            )
    +            .await;
    +        }
     
             anyhow::ensure!(
                 failures.is_empty(),
    diff --git a/vmm_core/state_unit/src/tests/quiesce.rs b/vmm_core/state_unit/src/tests/quiesce.rs
    --- a/vmm_core/state_unit/src/tests/quiesce.rs
    +++ b/vmm_core/state_unit/src/tests/quiesce.rs
    @@ -170,3 +170,111 @@
         assert!(started.elapsed() < Duration::from_millis(500));
         assert!(!units.is_running());
     }
    +
    +/// A unit whose first start succeeds and every later start fails.
    +struct FailsRestartUnit {
    +    starts: u32,
    +}
    +
    +impl StateUnit for FailsRestartUnit {
    +    async fn start(&mut self) -> anyhow::Result<()> {
    +        self.starts += 1;
    +        anyhow::ensure!(self.starts == 1, "intentional restart failure");
    +        Ok(())
    +    }
    +
    +    async fn stop(&mut self) {}
    +
    +    async fn reset(&mut self) -> anyhow::Result<()> {
    +        Ok(())
    +    }
    +
    +    async fn save(&mut self) -> Result<Option<SavedStateBlob>, SaveError> {
    +        Ok(None)
    +    }
    +
    +    async fn restore(&mut self, _state: SavedStateBlob) -> Result<(), RestoreError> {
    +        Err(RestoreError::SavedStateNotSupported)
    +    }
    +}
    +
    +impl InspectMut for FailsRestartUnit {
    +    fn inspect_mut(&mut self, req: inspect::Request<'_>) {
    +        req.respond();
    +    }
    +}
    +
    +/// A unit that records whether it is running.
    +struct RecordingUnit(std::sync::Arc<std::sync::atomic::AtomicBool>);
    +
    +impl StateUnit for RecordingUnit {
    +    async fn start(&mut self) -> anyhow::Result<()> {
    +        self.0.store(true, std::sync::atomic::Ordering::SeqCst);
    +        Ok(())
    +    }
    +
    +    async fn stop(&mut self) {
    +        self.0.store(false, std::sync::atomic::Ordering::SeqCst);
    +    }
    +
    +    async fn reset(&mut self) -> anyhow::Result<()> {
    +        Ok(())
    +    }
    +
    +    async fn save(&mut self) -> Result<Option<SavedStateBlob>, SaveError> {
    +        Ok(None)
    +    }
    +
    +    async fn restore(&mut self, _state: SavedStateBlob) -> Result<(), RestoreError> {
    +        Err(RestoreError::SavedStateNotSupported)
    +    }
    +}
    +
    +impl InspectMut for RecordingUnit {
    +    fn inspect_mut(&mut self, req: inspect::Request<'_>) {
    +        req.respond();
    +    }
    +}
    +
    +/// Regression test (S-3): when `resume_after_failed_save` fails because one unit does not
    +/// restart, the units it did restart are stopped again, so that no unit runs while the set
    +/// reports not running (the worker's teardown, `LoadedVm::pause`, skips `stop` then). Before the
    +/// fix the restarted unit stayed running, also after a later `start` refused the terminal state.
    +#[async_test]
    +async fn test_failed_resume_stops_restarted_units(driver: DefaultDriver) {
    +    let mut units = StateUnits::new();
    +    let running = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
    +    let _restarts = units
    +        .add("restarts")
    +        .spawn(&driver, |recv| {
    +            run_unit(RecordingUnit(running.clone()), recv)
    +        })
    +        .unwrap();
    +    let _fails = units
    +        .add("fails")
    +        .spawn(&driver, |recv| {
    +            run_unit(FailsRestartUnit { starts: 0 }, recv)
    +        })
    +        .unwrap();
    +    units.start().await.unwrap();
    +    units
    +        .quiesce_for_save(Duration::from_secs(1))
    +        .await
    +        .unwrap();
    +    assert!(!running.load(std::sync::atomic::Ordering::SeqCst));
    +
    +    let error = units
    +        .resume_after_failed_save(Duration::from_secs(1))
    +        .await
    +        .unwrap_err();
    +    assert!(error.to_string().contains("intentional restart failure"));
    +    // The set reports not running, and the unit that restarted was stopped again.
    +    assert!(!units.is_running());
    +    assert!(!running.load(std::sync::atomic::Ordering::SeqCst));
    +
    +    // `start` refuses (terminal state) and starts nothing.
    +    let error = units.start().await.unwrap_err();
    +    assert!(error.to_string().contains("terminal state"));
    +    assert!(!units.is_running());
    +    assert!(!running.load(std::sync::atomic::Ordering::SeqCst));
    +}
  2. ppenna commented on Nov 5, 2026

    @ppenna
    Contributor

    Re-triaged after #1260 (#1288, microsoft/openvmm#4921): resume_after_failed_save is gone. The rollback of a capture that fails after its quiesce now uses StateUnits::start, whose rollback of a partial start (fork position 018, microsoft/openvmm#4872 upstream) stops the units that started again when one does not answer, as this issue's fix proposed. So in the fork this issue is resolved once #1288 merges; upstream, microsoft/openvmm#4872 carries the same rollback.

  3. ppenna commented on Nov 10, 2026

    @ppenna
    Contributor

    Already fixed on dev by #1288, merged on 2026-11-05 as 4cfe8330e, which pinned microsoft/openvmm#4921 at 14dc4116e. The fix still holds at dev fdd1985f5, which pins OpenVMM 6ddd6ca50.

    What fixed it.

    Evidence. Each state_unit check ran cargo nextest run --profile agent -p state_unit in a scratch checkout of the named revision. The hosts are virtual machines.

    OpenVMM revision Host Check Result
    998b9f240, this issue's base Windows, local This issue's test_failed_resume_stops_restarted_units Fails at its second assert!(!running.load(..)): the restarted unit still runs
    998b9f240 with this issue's fix.patch Windows, local state_unit 12 of 12 pass
    9cec2b892, the pin before #1288 Windows, local The same scenario, with a unit whose task does not answer its restart Fails: the restarted unit still runs while the set reports stopped
    6ddd6ca50, the pin of dev at fdd1985f5 Windows, local; nvx-kvm-test The same scenario with the pinned API: stop, then a start in which one unit fails its confirmation, or its task does not answer Passes: every restarted unit stops again, dependents first. The 20 state_unit tests and both scratch tests pass
    6ddd6ca50 with the rollback's stop disabled Windows, local state_unit a_failed_confirmation_rolls_back_and_blocks_dependents, an_unanswered_start_rolls_back_and_blocks_dependents, and requests_fail_after_a_start_leaves_a_terminal_unit fail, so the existing tests already catch a regression
    6ddd6ca50, built with nvx.py build-openvmm over the other artifacts of release v0.1.0-dev.656af07e549e, at dev fdd1985f5 nvx-kvm-test (kvm), nvx-mshv-test (mshv), nvx-whp-test (whp) python3 scripts/nvx.py test-microvm --backend BACKEND --scenario snapshot-rollback Passes on all three. Each built binary reports 0.4.0+g6ddd6ca50, and its SHA-256 differs from the release binary's: kvm 90228ec16d81…, mshv 85289814d241…, whp c9b657d620b3…

    The rest of the failed-transition handling is tracked elsewhere. #1148, fixed by #1435, covers later requests against units in a terminal state, and #1363 covers units whose task has exited.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions