Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,16 @@ settings, and it must restore them when the request returns
(`nvx-time cancel-capture`). Afterwards, `nvx-time status` must exit 0 with a
passing boot line at generation 0 and no restore line, and
`/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress` must read 0.
The `snapshot-rollback` scenario fails a capture after its quiesce. The
guest's RAM has a file of its own, so the snapshot destination's parent holds
nothing, and it disappears after boot: the capture quiesces and saves the VM,
then fails to publish, and OpenVMM rolls it back. The request must return in
the source with status 0, after OpenVMM's capture profile records exactly one
quiesce and one save but no publication, and the source must pass the checks
that `snapshot-core` runs after its released request. Once the parent is back,
a second request of the same source must publish a snapshot and end the
source with status 0, leaving only the snapshot in the parent, and the
snapshot's restore must continue past that request exactly once.
The `platform-zero-fill` scenario covers a platform restore whose `distro`
image is shorter than the snapshot's and whose fresh scratch is absent. It
captures a platform snapshot with a 1 MiB `distro` placeholder and a 1 MiB
Expand Down
5 changes: 3 additions & 2 deletions doc/design/1.4-trust-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,9 @@ fallback, from automatic CPU-profile selection to a host-derived CPU profile,
warns at every cold boot that takes it.

The VMM uses checked address arithmetic, bounded buffers and tables, typed
validation errors, rate-limited guest-triggerable logs, and fallible quiesce
and restore operations. Native I/O callbacks only record bounded state or
validation errors, rate-limited guest-triggerable logs, and fallible restore
operations, and it rolls back a capture that fails after the quiesce and
before its commit point. Native I/O callbacks only record bounded state or
enqueue notifications; they never perform blocking snapshot or host-resource
work. Malformed saved device state is rejected before device workers start,
and no virtual processor runs after a partial restore failure. A failure after
Expand Down
4 changes: 3 additions & 1 deletion doc/design/3.4-block-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,5 +98,7 @@ Writable scratch follows one of two policies:

Transport and queue progress are saved with the other device state. Stopping
a queue prevents new descriptor intake and drains every accepted I/O to one
used-ring completion within the bounded snapshot quiesce timeout. The
used-ring completion. The drain has no deadline of its own: a backend that
never completes an I/O stalls the capture until its launcher ends the VMM; see
[Capture Transaction](5.3-snapshot-protocol.md#533-capture-transaction). The
controller flushes and copies scratch only after that drain.
66 changes: 44 additions & 22 deletions doc/design/5.3-snapshot-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ A capture requires up to three settings:
a restore must supply the same one; see
[Snapshot Identity](5.4-snapshot-artifact-and-attachment-model.md#545-snapshot-identity).

A capture also accepts four optional settings:
A capture also accepts three optional settings:

- **Memory backing file.** The file that backs guest RAM. When none is
supplied, the VMM creates one at launch in the parent of the destination
Expand All @@ -44,8 +44,6 @@ A capture also accepts four optional settings:
reserves so that a restore can give the VM more memory. Without it, a
restore cannot add memory; see
[Memory Activation](5.5-restore-protocol.md#554-memory-activation).
- **Quiesce timeout.** The longest the VMM may spend quiescing the VM,
including draining block I/O.
- **Scratch restore mode.** How a restore recreates
[paired scratch](5.5-restore-protocol.md#552-memory-and-scratch-materialization):
a private copy; a copy-on-write clone, only for a `workload-start` snapshot;
Expand Down Expand Up @@ -186,22 +184,31 @@ sequenceDiagram
Units-->>Controller: State, inventory, time contract, and CPU profile
Controller->>Storage: Copy state, memory, and optional scratch
Controller->>Storage: Publish with no-replace rename
Controller->>Worker: Terminate committed source
alt Published
Controller->>Worker: Terminate committed source
else Failed before the rename
Controller->>Worker: Roll back
Worker->>Units: Start in dependency order and resume host input
Worker-->>Guest: Continue execution
end
end
```

**Figure 5.2.** Capture transaction.

The VMM performs the capture as one bounded transaction:
The VMM performs the capture as one transaction:

1. Gate host input and defer completion of the snapshot-port write.
2. Stop the virtual processor at the I/O boundary while completing the write,
so saved state starts at the instruction immediately after the output.
3. Preflight the machine profile, destination, backend, external attachments,
and snapshot eligibility; a missing destination or a rejected preflight
releases the boundary and lets the guest continue.
releases the boundary and lets the guest continue. Otherwise, deliver the
console output that the guest wrote before its request; output accepted
while no endpoint is connected stays in the snapshot, and the restored VM
writes it.
4. Quiesce the remaining device workers and VM time in dependency order,
including the bounded block queue drain.
including the block queue drain.
5. Establish the time contract: require that no LAPIC timer is periodic or in
TSC-deadline mode and that PIT channel 0 is not counting periodically, take
the capture anchor and host identities, and record the declared rates, CPU
Expand All @@ -213,8 +220,22 @@ The VMM performs the capture as one bounded transaction:
8. Publish the directory with one no-replace rename.
9. Terminate the source worker and process after publication.

A failure in step 5 is a rollback-safe capture failure. Deriving `Q` never
fails a capture; when it cannot be derived, it is omitted with a warning.
A failure in steps 5 through 8, before the rename, rolls the capture back:
the VMM starts the stopped state units again in dependency order, resumes host
input, and releases the boundary, so the guest continues. The source
terminates instead when a unit does not start again, or when the VMM cannot
remove the staging directory's alias of guest RAM. Deriving `Q` never fails a
capture; when it cannot be derived, it is omitted with a warning.

The transaction has no deadline of its own. Gating host input in step 1 and
the quiesce in step 4 wait for every device, and delivering the console output
in step 3 waits for its endpoint to read it. A device that never finishes, or
an endpoint that stops reading, stalls the capture, with nothing published and
the source never resumed, until the process that launched the VMM ends it; the
VMM logs each state unit that has not answered after five seconds, so that the
log of such a capture names it. Every launcher therefore bounds a capture with
a deadline that ends the VMM, as the nvx crate's template store does with the
deadline that creating or refreshing a template requires.

## 5.3.4 State-Unit Ordering

Expand Down Expand Up @@ -261,16 +282,16 @@ flowchart LR
## 5.3.5 Commit Point

The directory rename is the commit point. Before it, a capture failure exposes
no final snapshot, and the guest resumes only when rollback is known to be
valid. After it, the source never resumes. This gives the guest an observable
boundary: code after the snapshot request runs once in each restored process
and never in the successfully captured source. A capture driven through the
management RPC likewise halts the source at the committed boundary and
terminates its process, so clients observe the transport closing.

Establishing the boundary is itself fallible. If host input cannot be gated,
the worker rolls the gate back and releases the write so that the guest
continues; if that rollback is uncertain, the worker terminates. Once
no final snapshot, and the guest resumes only once the VMM has started the
stopped state units again. After it, the source never resumes. This gives the
guest an observable boundary: code after the snapshot request runs once in
each restored process and never in the successfully captured source. A capture
driven through the management RPC likewise halts the source at the committed
boundary and terminates its process, so clients observe the transport closing.

Establishing the boundary is itself fallible. If a device fails to gate its
host input, the worker rolls the gate back and releases the write so that the
guest continues; if that rollback fails, the worker terminates. Once
processor stopping has begun, a boundary failure is terminal: host input stays
gated, and the VM is torn down rather than resumed. The post-restore
acknowledgement boundary follows the same rule.
Expand All @@ -284,9 +305,10 @@ As Table 5.3 shows, a published snapshot never coexists with a running source.
| Outcome | Source | Snapshot |
| --- | --- | --- |
| Committed capture | Terminated | Published |
| Missing destination or rejected preflight | Continues | None |
| Failure before the commit point, with rollback known to be valid | Continues | None |
| Failure before the commit point, with rollback not known to be valid | Terminated | None |
| Missing destination, rejected preflight, or failed input gate | Continues | None |
| Failure after the quiesce, before the commit point, that rolls back | Continues | None |
| Failure before the commit point that cannot roll back | Terminated | None |
| Input gate, console delivery, or quiesce that never completes | Ended by its launcher | None |

After a commit, execution continues only in virtual machines restored from the
snapshot, either as clones or as one resumed instance; see
Expand Down
Loading
Loading