Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions Guide/src/dev_guide/snapshot_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,12 @@ bound egress policy. Encoding version 2 includes the static prefix, guest MAC,
and canonical ARP next-hop set in addition to the policy rules. An absent
encoding-version field denotes version 1, allowing network snapshots produced
before endpoint next-hop binding to retain their original digest semantics.
A platform snapshot whose digest is that of the complete deny-all policy bound
to its identity, in encoding version 3, or version 4 with IPv6, is a NIC
template, whose restore may install another policy with
`--restore-network-policy`. Earlier encodings do not record the host-loopback
action or the proxy endpoint, so they never identify a template, and the
format records nothing else for it.
A platform snapshot, or a snapshot without a tier, whose digest is that of the
complete deny-all policy bound to its identity, in encoding version 3, or
version 4 with IPv6, is a NIC template, whose restore may install another
policy with `--restore-network-policy`. Earlier encodings do not record the
host-loopback action or the proxy endpoint, so they never identify a template,
and the format records nothing else for it.

## Device state (`state.bin`)

Expand Down
62 changes: 37 additions & 25 deletions Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,8 +246,9 @@ describes the source definitions.
endpoint-ready RX at the capture boundary, rewind unused guest RX
descriptors, and recreate a fresh Consomme endpoint generation on restore.
Restore of a networked snapshot requires `--network-profile portable` and,
unless `--restore-network-policy` replaces a platform snapshot's complete
deny-all, the same active egress policy rules. The policy digest binds the
unless `--restore-network-policy` replaces the complete deny-all of a
platform snapshot or of a snapshot without a tier, the same active egress
policy rules. The policy digest binds the
saved static identity, including the IPv6 identity, and its derived ARP
next hops, which are reconstructed before vCPUs start. A snapshot whose
network predates IPv6 restores IPv4-only, with its original digest. Native
Expand Down Expand Up @@ -464,7 +465,8 @@ describes the source definitions.
When the snapshot contains virtio-net, restore also requires
`--network-profile portable`; the snapshot's profile and canonical egress
policy must match the supplied portable configuration, unless
`--restore-network-policy` installs a new policy on a platform snapshot.
`--restore-network-policy` installs a new policy on a platform snapshot or
a snapshot without a tier.

For sandbox-block snapshots, restore repeats each read-only
`--microvm-sandbox-block` argument. Its role, access, geometry, and SHA-256
Expand Down Expand Up @@ -501,10 +503,12 @@ describes the source definitions.
in progress; Windows flush completion waits for the named-pipe peer to
consume the complete frame.
* `--restore-entropy`: Accepted for compatibility; it has no effect. Every
microVM restore makes restore packet version 5 available on the private
portb restore channel. The packet carries 64 bytes of fresh entropy, the
online-VP target, the memory-expansion ranges, the downtime, and the TSC
rate deviation; the guest must consume it and explicitly reseed its RNG.
microVM restore makes restore packet version 5, or version 6 at a restore
that replaces the network policy of a snapshot without a tier, available on
the private portb restore channel. The packet carries 64 bytes of fresh
entropy, the online-VP target, the memory-expansion ranges, the downtime,
the TSC rate deviation, and, in version 6, the network settings of the
restore's policy; the guest must consume it and explicitly reseed its RNG.
[Snapshots](../../../user_guide/openvmm/snapshots.md) documents its format,
the portb status bits, and the time sample.
Every microVM portb device also reports generation-ID support in status bit
Expand Down Expand Up @@ -536,26 +540,34 @@ describes the source definitions.
for a cold boot, for a paired snapshot or one without sandbox blocks, and
together with a scratch `--microvm-sandbox-block`. A scratch file, when given
instead, keeps the exact captured length.
* `--restore-network-policy`: Restore a platform snapshot's NIC with the
network policy of this restore's options rather than the captured one. The
snapshot must be a NIC template: a platform snapshot whose NIC was captured
with a complete deny-all policy, `--network-egress deny` and
`--host-loopback deny` without egress rules, `--network-proxy`, or
* `--restore-network-policy`: Restore the NIC of a platform snapshot, or of a
snapshot without a tier, with the network policy of this restore's options
rather than the captured one. The snapshot must be a NIC template: one whose
NIC was captured with a complete deny-all policy, `--network-egress deny`
and `--host-loopback deny` without egress rules, `--network-proxy`, or
ingress. Its guest learned no DNS server and announces no listeners, and its
endpoint held no flows, listeners, or forwards, so its clones may each
endpoint held no flows, listeners, or forwards, so its restores may each
install their own policy, including `--network-ingress allow`, explicit
`--host-loopback allow`, and `--host-loopback-forward`. OpenVMM validates the
policy as at a cold boot, binds it to the captured identity, which the NIC
keeps, and installs it, with its forwards bound, before any vCPU runs; a
policy that fails validation, or a forward that cannot bind its port, fails
the restore. A restore that wants no connectivity passes the template's own
deny-all. The restore cannot readdress the NIC, and a policy may name IPv6
destinations only when the captured identity has IPv6. OpenVMM rejects the
option before any vCPU starts for a cold boot, for a snapshot of another
tier, for a snapshot without a NIC, and for a NIC captured with any other
policy, including one whose digest encoding predates version 3. The guest
keeps its captured command line, so it learns its DNS server and whether
to announce its listeners from its host, after the restore.
`--host-loopback allow`, and `--host-loopback-forward`. A snapshot without a
tier may hold a workload that has already run, but under the complete
deny-all that workload had no connectivity, so nothing in the snapshot
depends on the policy, and such a restore grants the workload connectivity
that it never had. OpenVMM validates the policy as at a cold boot, binds it
to the captured identity, which the NIC keeps, and installs it, with its
forwards bound, before any vCPU runs; a policy that fails validation, or a
forward that cannot bind its port, fails the restore. A restore that wants
no connectivity passes the template's own deny-all. The restore cannot
readdress the NIC, and a policy may name IPv6 destinations only when the
captured identity has IPv6. OpenVMM rejects the option before any vCPU
starts for a cold boot, for a `workload-start` or `instance-checkpoint`
snapshot, for a snapshot without a NIC, and for a NIC captured with any
other policy, including one whose digest encoding predates version 3. The
guest keeps its captured command line, so it learns its DNS server and
whether to announce its listeners after the restore: a platform clone from
its host, and the guest of a snapshot without a tier from restore packet
version 6, which carries them, and whose restore gates host input until the
guest acknowledges that it applied them. A guest that reads restore packet
version 5 alone fails such a restore rather than ignore them.
* `--restore-gate-timeout-ms <MILLISECONDS>`: Bound microVM guest repair and
gate acknowledgement after restore. The default is 60000 milliseconds.
* `--cpu-profile <ID>`: Select the microVM's pinned CPU profile, `auto`
Expand Down
55 changes: 38 additions & 17 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,11 @@ releasing gated input. The peer must accept and read while resume is in
progress; on Windows, flush completion waits until the named-pipe peer consumes
the complete frame.

For a tiered microVM restore, OpenVMM starts device workers with network,
control, and filesystem input gated. The guest performs post-restore repair
and writes the existing snapshot port (`0x605`) to acknowledge completion.
For a tiered microVM restore, and for a restore that replaces the network
policy of a snapshot without a tier, OpenVMM starts device workers with
network, control, and filesystem input gated. The guest performs post-restore
repair and writes the existing snapshot port (`0x605`) to acknowledge
completion.
OpenVMM stops at that exact post-write boundary, completes the deferred write
while the vCPU is stopped, and then releases device input before resuming the
vCPU. Until then, the guest's virtio-fs requests wait on their virtqueues, so
Expand All @@ -199,14 +201,18 @@ block sizes. The device stays writable, as the guest negotiated it, but reads
return zeros and writes fail with an I/O error: OpenVMM never drops a write or
keeps one in host memory. Only a platform snapshot's fresh scratch may be
absent, and a scratch file, when given, keeps the exact captured length. A
platform snapshot whose NIC was captured with a complete deny-all policy,
`--network-egress deny` and `--host-loopback deny` without egress rules, a
proxy, or ingress, is a NIC template: with `--restore-network-policy`, each
clone installs the policy of its own restore options, which may publish its
listeners and forward host ports, instead of the captured one. OpenVMM
validates that policy as at a cold boot and installs it before any vCPU runs,
and the NIC keeps its captured identity. Every other networked restore
requires the captured policy. An instance-checkpoint restore attempt atomically creates
platform snapshot, or a snapshot without a tier, whose NIC was captured with a
complete deny-all policy, `--network-egress deny` and `--host-loopback deny`
without egress rules, a proxy, or ingress, is a NIC template: with
`--restore-network-policy`, each restore installs the policy of its own
restore options, which may publish its listeners and forward host ports,
instead of the captured one. OpenVMM validates that policy as at a cold boot
and installs it before any vCPU runs, and the NIC keeps its captured identity.
A snapshot without a tier may hold a workload that has already run, but under
the complete deny-all that workload had no connectivity, so nothing in the
snapshot depends on the policy, and such a restore grants the workload
connectivity that it never had. Every other networked restore requires the
captured policy. An instance-checkpoint restore attempt atomically creates
`resume.claim`; subsequent restores are rejected. The claim is committed after
artifact and configuration validation but before worker construction, so the
restore attempt remains consumed if later worker startup fails.
Expand All @@ -219,16 +225,18 @@ the ID from the first 16 bytes of the restore packet's fresh entropy, so the
guest can reject an unchanged clone identity, reseed Linux, refresh runtime
identifiers, and acknowledge the input gate without another PMIO transfer.

Every microVM restore exposes restore packet version 5. Status bit 1
advertises it, and writing `0xa5` to the status port selects it for reading
from the data port. Its 24-byte little-endian header holds:
Every microVM restore exposes a restore packet: version 5, or version 6 at a
restore whose `--restore-network-policy` replaces the network policy of a
snapshot without a tier. Status bit 1 advertises it, and writing `0xa5` to the
status port selects it for reading from the data port. Its 24-byte
little-endian header holds:

- the magic `OVR` and version 5;
- the magic `OVR` and the version;
- flags: bit 0, the downtime came from host UTC; bit 1, a memory target;
bit 2, the guest must acknowledge the restore through port `0x605`; bit 3,
a test hook is active;
- the online-VP target (0 for none), the memory-range count, and a reserved
zero byte;
- the online-VP target (0 for none), the memory-range count, and a byte that
is reserved and zero in version 5 and is the network byte in version 6;
- the generation counter (`u32`), which is 0 at cold boot and one more than
the snapshot's at each restore;
- the TSC rate deviation (`i32`, ppm scaled by 2^16); and
Expand All @@ -240,6 +248,19 @@ memory target, and one or more memory ranges, so a zero-range target can skip
post-restore repair. The packet carries no host UTC: the guest sets its clock
from a time sample.

Version 6 carries the network settings of the restore's policy, which a cold
boot names on the command line that the restore keeps, and which a NIC
template's guest never learned: its network byte's bits 0 and 1 give the
family of the guest's DNS server, 0 for none, 1 for IPv4, or 2 for IPv6, and
bit 2 asks the guest to announce its listeners, as `virtnet_dns=` and
`virtnet_ingress=` do at a cold boot; the other bits are zero. A 16-byte DNS
server follows the entropy: an IPv6 address, or an IPv4 address and 12 zero
bytes, or zeros without a server. Such a restore requires the guest's
acknowledgement, which the guest gives once it has applied the settings. A
guest that reads version 5 alone fails such a restore rather than ignore its
settings, and every other restore, a platform clone's included, keeps
version 5.

Status bit 6 advertises the time sample. Writing `0xa7` to the status port
latches a 48-byte little-endian sample, version 2:

Expand Down
9 changes: 5 additions & 4 deletions openvmm/openvmm_entry/src/cli_args/microvm.rs
Original file line number Diff line number Diff line change
Expand Up @@ -274,9 +274,9 @@ impl From<MachineProfileCli> for MachineProfile {
/// Options of the microVM machine profile.
#[derive(clap::Args)]
pub struct MicrovmCli {
/// Accepted for compatibility: every microVM restore exposes restore packet
/// version 4, which carries fresh entropy, through the private portb restore
/// channel.
/// Accepted for compatibility: every microVM restore exposes a restore
/// packet, version 5, or version 6 with network settings, which carries
/// fresh entropy, through the private portb restore channel.
#[clap(long, requires = "restore_snapshot")]
pub restore_entropy: bool,

Expand All @@ -297,7 +297,8 @@ pub struct MicrovmCli {
pub restore_absent_scratch: bool,

/// Install this restore's network policy in place of the complete
/// deny-all policy that a platform snapshot's NIC was captured with.
/// deny-all policy that the NIC of a platform snapshot, or of a snapshot
/// without a tier, was captured with.
///
/// The NIC keeps its captured identity. The network policy options of
/// this restore, which may include `--network-ingress allow`,
Expand Down
12 changes: 11 additions & 1 deletion openvmm/openvmm_entry/src/microvm/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ use super::network::EffectiveMicrovmNetwork;
use super::network::effective_microvm_network;
use super::network::microvm_ingress;
use super::network::microvm_network_endpoint;
use super::network::microvm_restore_network;
use super::output::MicrovmOutputDrain;
use super::output::OutputCompletion;
use super::output::spawn_output;
Expand Down Expand Up @@ -420,9 +421,18 @@ impl<'a> MicrovmConfigBuilder<'a> {
let (drain, output_drain) = MicrovmOutputDrain::new(Some(output_completion));
self.resources.output_drain = Some(drain);
let (generation_id, time_abi_restore) = if self.restore.machine_contract.is_some() {
let network = if self.restore.network_settings {
let network = self
.network
.as_ref()
.context("a restore that replaces the network policy requires a NIC")?;
Some(microvm_restore_network(&network.policy))
} else {
None
};
let (generation_id, base) = self
.restore
.time_abi_restore_packet(opt.microvm.restore_processors)?;
.time_abi_restore_packet(opt.microvm.restore_processors, network)?;
(
generation_id,
Some(self.resources.time_abi_restore_packet(base)),
Expand Down
Loading
Loading