You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If one device fails to restart during snapshot rollback, successfully restarted siblings remain running even though the VM reports that its state-unit set is stopped. Teardown can then skip their Stop.
In vmm_core/state_unit/src/quiesce.rs, StateUnits::resume_after_failed_save records successful restarts directly in the unit table:
letmut inner = self.inner.lock();for(name, id, result)in results {letSome(unit) = inner.units.get_mut(&id)else{continue;};match result {StartResult::Started => unit.state = State::Running,StartResult::DependencyFailed => {
unit.state = State::Stopped;
failures.push(format!("{name}: a dependency did not restart"));
Other result arms record failures, and the method returns an error before setting the set's running flag. It does not undo the successful Starts. LoadedVm::resume_after_failed_snapshot consequently leaves the VM marked stopped, and LoadedVm::pause returns without stopping those running units.
Trigger
Start independent devices, quiesce them successfully, and encounter a capture failure for which rollback is allowed. Send ResumeAfterFailedSnapshot with one device able to restart and another failing its Start. The rollback reports failure, but the successfully restarted device continues running. The vCPU snapshot boundary still holds the guest; it does not stop that device activity.
Fix
fix.patch collects successful restart IDs and stops that subset in reverse dependency order before returning the rollback error, using the existing start-rollback cleanup mechanism. The unit-table lock is released before awaiting cleanup.
Cleanup Stop remains unbounded and retains the existing dead-channel failure behavior. This repairs partial-restart cleanup, not the separate problem of accepting later requests against terminal units.
Re-triaged after #1260 (#1288, microsoft/openvmm#4921): resume_after_failed_save is gone. The rollback of a capture that fails after its quiesce now uses StateUnits::start, whose rollback of a partial start (fork position 018, microsoft/openvmm#4872 upstream) stops the units that started again when one does not answer, as this issue's fix proposed. So in the fork this issue is resolved once #1288 merges; upstream, microsoft/openvmm#4872 carries the same rollback.
Already fixed on dev by #1288, merged on 2026-11-05 as 4cfe8330e, which pinned microsoft/openvmm#4921 at 14dc4116e. The fix still holds at devfdd1985f5, which pins OpenVMM 6ddd6ca50.
What fixed it.
StateUnits::resume_after_failed_save no longer exists.
LoadedVm::resume_after_failed_snapshot, which rolls back a failed capture, now calls StateUnits::start. It is in chain position 072, 31ea2adca.
StateUnits::start rolls back a partial start, as this issue's fix.patch proposed. When a unit fails to confirm its start, or its task does not answer, every unit that started is stopped again, in reverse dependency order. The set then reports stopped. That rollback is chain position 015, 179106cd9, which state_unit: roll back a partial start instead of panicking openvmm#4872 proposes upstream.
Evidence. Each state_unit check ran cargo nextest run --profile agent -p state_unit in a scratch checkout of the named revision. The hosts are virtual machines.
OpenVMM revision
Host
Check
Result
998b9f240, this issue's base
Windows, local
This issue's test_failed_resume_stops_restarted_units
Fails at its second assert!(!running.load(..)): the restarted unit still runs
The same scenario, with a unit whose task does not answer its restart
Fails: the restarted unit still runs while the set reports stopped
6ddd6ca50, the pin of dev at fdd1985f5
Windows, local; nvx-kvm-test
The same scenario with the pinned API: stop, then a start in which one unit fails its confirmation, or its task does not answer
Passes: every restarted unit stops again, dependents first. The 20 state_unit tests and both scratch tests pass
6ddd6ca50 with the rollback's stop disabled
Windows, local
state_unit
a_failed_confirmation_rolls_back_and_blocks_dependents, an_unanswered_start_rolls_back_and_blocks_dependents, and requests_fail_after_a_start_leaves_a_terminal_unit fail, so the existing tests already catch a regression
6ddd6ca50, built with nvx.py build-openvmm over the other artifacts of release v0.1.0-dev.656af07e549e, at devfdd1985f5
Passes on all three. Each built binary reports 0.4.0+g6ddd6ca50, and its SHA-256 differs from the release binary's: kvm 90228ec16d81…, mshv 85289814d241…, whp c9b657d620b3…
The rest of the failed-transition handling is tracked elsewhere. #1148, fixed by #1435, covers later requests against units in a terminal state, and #1363 covers units whose task has exited.
Activity
tangruize commentedon Nov 1, 2026
Proposed patch
Patch against OpenVMM
998b9f2402050cd16d1faa897bc831ef1434be48. Save the contents below asfix.patchand rungit apply fix.patch.fix.patchppenna commentedon Nov 5, 2026
Re-triaged after #1260 (#1288, microsoft/openvmm#4921):
resume_after_failed_saveis gone. The rollback of a capture that fails after its quiesce now usesStateUnits::start, whose rollback of a partial start (fork position 018, microsoft/openvmm#4872 upstream) stops the units that started again when one does not answer, as this issue's fix proposed. So in the fork this issue is resolved once #1288 merges; upstream, microsoft/openvmm#4872 carries the same rollback.microVM: quiesce snapshots without a deadline
microVM: quiesce snapshots without a deadline
ppenna commentedon Nov 10, 2026
Already fixed on
devby #1288, merged on 2026-11-05 as4cfe8330e, which pinned microsoft/openvmm#4921 at14dc4116e. The fix still holds atdevfdd1985f5, which pins OpenVMM6ddd6ca50.What fixed it.
StateUnits::resume_after_failed_saveno longer exists.LoadedVm::resume_after_failed_snapshot, which rolls back a failed capture, now callsStateUnits::start. It is in chain position 072,31ea2adca.StateUnits::startrolls back a partial start, as this issue'sfix.patchproposed. When a unit fails to confirm its start, or its task does not answer, every unit that started is stopped again, in reverse dependency order. The set then reports stopped. That rollback is chain position 015,179106cd9, which state_unit: roll back a partial start instead of panicking openvmm#4872 proposes upstream.49afa32c0, openvmm: pin device starts confirmed before the VM runs #1316) added the start confirmation: a unit whose confirmation fails is stopped again with the others. Positions 240 to 242 (openvmm: pin the rollback as the only way to run a quiesced microVM #1430 and openvmm: pin the handling of state units in a terminal state #1435) left the rollback's code unchanged.Evidence. Each
state_unitcheck rancargo nextest run --profile agent -p state_unitin a scratch checkout of the named revision. The hosts are virtual machines.998b9f240, this issue's basetest_failed_resume_stops_restarted_unitsassert!(!running.load(..)): the restarted unit still runs998b9f240with this issue'sfix.patchstate_unit9cec2b892, the pin before #12886ddd6ca50, the pin ofdevatfdd1985f5nvx-kvm-teststop, then astartin which one unit fails its confirmation, or its task does not answerstate_unittests and both scratch tests pass6ddd6ca50with the rollback's stop disabledstate_unita_failed_confirmation_rolls_back_and_blocks_dependents,an_unanswered_start_rolls_back_and_blocks_dependents, andrequests_fail_after_a_start_leaves_a_terminal_unitfail, so the existing tests already catch a regression6ddd6ca50, built withnvx.py build-openvmmover the other artifacts of releasev0.1.0-dev.656af07e549e, atdevfdd1985f5nvx-kvm-test(kvm),nvx-mshv-test(mshv),nvx-whp-test(whp)python3 scripts/nvx.py test-microvm --backend BACKEND --scenario snapshot-rollback0.4.0+g6ddd6ca50, and its SHA-256 differs from the release binary's: kvm90228ec16d81…, mshv85289814d241…, whpc9b657d620b3…The rest of the failed-transition handling is tracked elsewhere. #1148, fixed by #1435, covers later requests against units in a terminal state, and #1363 covers units whose task has exited.