Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -808,8 +808,9 @@ Each also runs the ignored `nvx-rs` test `tests/openvmm_vectors.rs`, which
compares the submodule's control-session protocol vectors, and the lines of its
platform allowlist of kernel command-line tokens, with the copies that the
protocol and template tests read in the `nvx-rs` job, which cannot check out
OpenVMM, and the versions of OpenVMM's snapshot format, saved-state schema,
time ABI, and restore packets with those of `nvx::contract`;
OpenVMM, the versions of OpenVMM's snapshot format, saved-state schema,
time ABI, and restore packets with those of `nvx::contract`, and the bound of
the restore packet's activation record with that of `nvx::activation`;
see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors).
The ignored test `tests/openvmm_cpu_profiles.rs` likewise compares the
submodule's built-in CPU profiles with the copy of their identities from which
Expand Down
2 changes: 1 addition & 1 deletion doc/design/1.5-compatibility-and-versioning-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ contract carries its own version or identity, independent of the others.
| Device-private schemas | Saved queue and device state that must be understood before a device can resume. |
| Snapshot format | Persisted machine contract, manifest, state inventory, memory geometry, attachment requirements, and artifact identity. |
| RAM expansion capability | Recorded RAM capacity, memory-block size, and expansion ranges that restore may activate. |
| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, and fresh entropy. |
| Restore packet | Layout and meaning of the one-time record that gives a restored guest its generation, downtime, rate deviation, activation targets, fresh entropy, and a platform clone's activation record. |
| Control protocol | Authenticated sessions between the host and the guest agents, request framing, operation semantics, and lifecycle epochs. |
| Management RPC machine profile | Identity of the microVM machine profile in the management RPC. |

Expand Down
32 changes: 27 additions & 5 deletions doc/design/5.5-restore-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,9 @@ flags. Every restore exposes one restore packet through the byte channel, for
every tier and for untiered snapshots. The packet is version 1. A restore that
installs a new network policy on a snapshot without a tier sets its `NETWORK`
flag, and its network byte and DNS server carry the policy's settings; every
other restore leaves both zero.
other restore leaves both zero. A platform restore that hands its clone an
activation record sets the `ACTIVATION` flag, and the packet ends with the
record; every other packet ends with the DNS server.

```text
offset size field
Expand All @@ -224,6 +226,8 @@ offset size field
24 16*n ranges n x { u64 gpa_start, u64 length }, LE
24+16n 64 entropy
88+16n 16 dns_server zero unless NETWORK
104+16n 4 activation_length u32, little-endian, 1 to 1024; only with ACTIVATION
108+16n L activation_record L = activation_length bytes; only with ACTIVATION
```

**Figure 5.7.** Restore packet layout.
Expand All @@ -237,12 +241,14 @@ offset size field
| 2 | `ACK_REQUIRED` | Host input is gated, and the guest must acknowledge through the snapshot port after repair |
| 3 | `TEST_HOOKS` | Host overrides of the downtime, UTC, rate deviation, host clock rate, time samples, or CPU profile are active |
| 4 | `NETWORK` | The network byte and the DNS server carry network settings |
| 5-7 | | Zero; the guest rejects a packet with any of them set |
| 5 | `ACTIVATION` | The packet ends with a platform clone's activation record |
| 6-7 | | Zero; the guest rejects a packet with any of them set |

Every field is sealed before the first restored processor runs. The packet
carries no host UTC: the guest takes a [time
sample](3.2-clock-and-time.md#329-time-samples) after it, which pairs host UTC
with the guest TSC on the host. A packet without ranges is 104 bytes. The packet
with the guest TSC on the host. A packet without ranges or an activation record
is 104 bytes, and an activation record of `L` bytes adds `4 + L`. The packet
is authoritative, and the status bits mirror it; the acknowledgement flag spares
the guest from inferring gating from the tier and targets. A VM process exposes
at most one packet, and reading its last byte consumes it, so a guest captured
Expand All @@ -265,6 +271,22 @@ ingress or explicit host-loopback access. Such a restore always sets
`ACK_REQUIRED`, so the guest applies the settings before host input reaches
it.

The `ACTIVATION` flag marks the activation record of a platform snapshot's
clone, which the launcher supplies with
`--restore-activation-record`: the settings of the clone's own launch, in the
canonical encoding of an activation record, which the host and the guest share
through `nvx::activation`. OpenVMM reads the record before any processor runs
and passes its bytes through without interpreting them, so the record carries
no secret: secrets and host mapping tables stay on the authenticated control
channel. Its length, 1 to 1,024 bytes, precedes it as a `u32`, which keeps the
record aligned to the guest's four-byte reads, and the guest rejects a length
outside those bounds. Each four bytes of the record cost the guest one more
port access, as every byte of the packet does. The flag was a reserved bit,
which a guest that predates it rejects, so such a guest fails a restore that
hands it a record rather than ignore the record, and restores without a record
are unchanged. Such a restore always sets `ACK_REQUIRED`, so the guest has read
the record before host input reaches it.

## 5.5.6 Generations

- The generation counter `g` is zero in a cold-booted process and the captured
Expand All @@ -288,8 +310,8 @@ runs before the acknowledgement:
1. Read the byte-channel status, whose restore-packet bit is always set after
a restore.
2. Select the packet, read it four bytes per access, and validate its magic,
version, reserved bits, counts, network settings, generation, and
generation ID.
version, reserved bits, counts, network settings, activation record's
length, generation, and generation ID.
3. Set the wall clock: in the same port transaction as the packet, take up to
three [time samples](3.2-clock-and-time.md#329-time-samples) of the
packet's generation, until one pairs host UTC with the guest TSC within
Expand Down
41 changes: 31 additions & 10 deletions doc/openvmm-upstream-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@
reorganization kept it there; see [Upstream status](#upstream-status).
- Mapped fork tip: `7677ad8d56a59864475012b63b6f01a22575c30d`, position 131. The graph, PR
summary, and commit mapping below cover positions 001-131.
- NVX's pinned OpenVMM revision: `6cb6ba87cfb6707078d19f09ec81d9a78fcd1ede`, position 232, the
head of microsoft/openvmm#4995, whose commit follows position 231, `98e5b3ca5`, the head of
`feature/microvm/dev`, which microsoft/openvmm#4994 merged with position 230 after position 229,
`ed8fc0713`, which microsoft/openvmm#4993 merged after position 228, `d6addf8c1`, which
- NVX's pinned OpenVMM revision: `d501e50660136000ec93412aea78fdafb9cf391c`, position 233, the
head of microsoft/openvmm#4996, whose commit follows position 232, `6cb6ba87c`, the head of
`feature/microvm/dev`, which microsoft/openvmm#4995 merged after position 231, `98e5b3ca5`,
which microsoft/openvmm#4994 merged with position 230 after position 229, `ed8fc0713`, which
microsoft/openvmm#4993 merged after position 228, `d6addf8c1`, which
microsoft/openvmm#4992 merged after position 227, `154548a0e`, which
microsoft/openvmm#4991 merged after position 226, `881619911`, which
microsoft/openvmm#4989 merged after position 225, `99aac19b3`, which
Expand All @@ -25,8 +26,8 @@
position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits
of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1
positions 149-220 are positions 148-219, and the tree of position 220 is the tree of
`3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 101 commits past
the mapped tip on the same first-parent chain. Positions 132-232 are not yet assigned to PRs; see
`3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 102 commits past
the mapped tip on the same first-parent chain. Positions 132-233 are not yet assigned to PRs; see
[Commits after the mapped tip](#commits-after-the-mapped-tip).
- Upstream main: `ba105dd7f00499acc1862456c179c0930cb2c8f7` (2026-11-06), the fork base itself;
see [Upstream main past the fork base](#upstream-main-past-the-fork-base).
Expand Down Expand Up @@ -986,7 +987,8 @@ ba105dd7f
-> earlier snapshot formats refused before decoding [230]
-> manifest fields reserved for upstream [231]
-> unknown paired-scratch restore mode refused [232]
-> 6cb6ba87c (NVX pin)
-> clone's activation record carried in the restore packet [233]
-> d501e5066 (NVX pin)
```

For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main,
Expand All @@ -1002,13 +1004,13 @@ reorganization folded the PR20 and PR12 fixups into the commits that they fix; s

## Commits after the mapped tip

The NVX pin extends the chain past position 131 without merges. None of the 101 later commits is a
The NVX pin extends the chain past position 131 without merges. None of the 102 later commits is a
fixup of a mapped PR, and none is yet classified or assigned to PRs. Each of positions 132-186
combines the v2.4 commits that the table lists, as v3.0 positions 145-160 and 162-200 did, which
developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191
followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0
reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0
reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-232 followed the
reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-233 followed the
v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions
222 and 223. Each commit has the position that it
had in v5.1 through position 147, and one less from v5.1 position 149 on; see
Expand Down Expand Up @@ -1118,6 +1120,7 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4
| 230 | `8a994544a` | - | - | - | - | openvmm_helpers: refuse earlier snapshot formats before decoding them (microsoft/openvmm#4994) |
| 231 | `98e5b3ca5` | - | - | - | - | openvmm_helpers: reserve the manifest fields that upstream assigns (microsoft/openvmm#4994) |
| 232 | `6cb6ba87c` | - | - | - | - | openvmm_helpers: refuse an unknown paired-scratch restore mode (microsoft/openvmm#4995) |
| 233 | `d501e5066` | - | - | - | - | microvm: carry a clone's activation record in the restore packet (microsoft/openvmm#4996) |

The remarks that say whether commits apply to upstream main record what the v3.0 reorganization
found on its base, `b2676222b`, at the positions that the commits have now.
Expand Down Expand Up @@ -1222,7 +1225,14 @@ fields 18-20 for upstream's SaveVM and RestoreVM, microsoft/openvmm#4553. Positi
restore-mode match that position 166 adds to the capture-tier check of PR19's position 093:
restore runs that check before the machine contract's shape check, so the match now refuses a mode
that it does not know with the shape check's error, where it panicked on the assumption that the
shape check had refused the mode.
shape check had refused the mode. Position 233 lets a platform restore end the restore packet with
its clone's activation record: it defines bit 5 of the packet's flags, which position 193 reserved
when it restarted the packet at version 1, beside the network settings of position 181; adds
`--restore-activation-record`, which the restore preparation of PR17's position 079 reads before
any vCPU runs, only for the platform tier of PR19's position 093, and with the post-restore gate of
PR20's position 095; adds the field that the management RPC's conversion of position 228 turns into
that option, in the protocol that PR25's position 124 extends; and adds a test of the restore
packet that the portb device of PR15's position 060 serves since position 137.
Assigning the later commits therefore means revisiting those PRs, not only appending new ones.

Position 177, by contrast, changes only `disk_blob`, which upstream main and the fork base
Expand Down Expand Up @@ -1429,6 +1439,17 @@ claim of position 093. Replaying it on upstream main conflicts in both of its fi
lacks the microVM snapshot module of position 074 and the tiers and paired scratch that positions
092, 093, and 166 add to the restore, so it follows PR17 and PR19 upstream.

Position 233 changes the restore packet of the time ABI series in `chipset_resources`, which
position 132 adds and positions 160, 181, and 193 change, and adds a test to the portb device that
serves it, which PR15's positions 060 and 061, PR17's position 071, PR19's position 092, PR20's
positions 094, 095, 097, and 099, PR24's position 119, and positions 137, 139, 147, 160, 181, 193,
195, 208, and 226 also change. It changes the microVM CLI of PR15's position 064, the restore
preparation of PR17's position 079, the management RPC's microVM conversion of position 228 and
the protocol of PR25's position 124, and their Guide pages. Replaying it on upstream main
conflicts in all nine of its files: upstream main lacks the time ABI series, the microVM CLI and
restore of PR15 and PR17, and the conversion of position 228, so it follows that series, PR15,
PR17, PR19, PR20, and PR25 upstream.

## PR summary

Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0
Expand Down
8 changes: 5 additions & 3 deletions nvx-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2862,10 +2862,12 @@ NVX_BUNDLE=skip cargo test --locked --all-features --lib -- \

`tests/openvmm_vectors.rs` is an ignored test that compares the copies of
OpenVMM's vectors in `tests/vectors/openvmm/` with the files of the OpenVMM
submodule, which the `nvx-rs` job cannot check out, and the versions of
submodule, which the `nvx-rs` job cannot check out, the versions of
OpenVMM's snapshot format, saved-state schema, time ABI, and restore packets in
`nvx::contract` with the submodule's. The NVX CLI jobs run it once they check
the submodule out, and so does an OpenVMM pin promotion; see
`nvx::contract` with the submodule's, and the bound of an activation record,
`nvx::activation::MAX_RECORD_BYTES`, with that of the restore packet's. The NVX
CLI jobs run it once they check the submodule out, and so does an OpenVMM pin
promotion; see
[OpenVMM's vectors](tests/vectors/README.md#openvmms-vectors).
`tests/openvmm_cpu_profiles.rs` is an ignored test that compares
`src/host/cli/cpu_generations.json`, the identities of the built-in CPU
Expand Down
1 change: 1 addition & 0 deletions nvx-rs/public-api/linux.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1600,6 +1600,7 @@ method nvx::guest::time::MemoryRange::length
method nvx::guest::time::MemoryRange::start
struct nvx::guest::time::Restore
method nvx::guest::time::Restore::acknowledgement_required
method nvx::guest::time::Restore::activation_record
method nvx::guest::time::Restore::finished
method nvx::guest::time::Restore::generation_id
method nvx::guest::time::Restore::memory_ranges
Expand Down
6 changes: 5 additions & 1 deletion nvx-rs/src/contract.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,11 @@ pub const TIME_ABI_VERSION: u64 = 1;
///
/// Version 1 carries no host UTC: the guest takes a time sample to set its clock. A restore that
/// replaces the network policy of a snapshot without a tier flags the packet's network: the new
/// policy's DNS server, and whether the guest announces its listeners under it.
/// policy's DNS server, and whether the guest announces its listeners under it. A platform
/// restore may flag an activation record, which then ends the packet: the settings of the clone's
/// own launch, which its launcher supplied. The flag was a reserved bit, which a guest that
/// predates it refuses, so a packet that sets it fails on such a guest, and a packet that does not
/// keeps the version's layout and meaning.
pub const RESTORE_PACKET_VERSION: u64 = 1;

/// Returns how the `openvmm` section of an NVX `SOURCE-MANIFEST.json` departs from the contracts
Expand Down
21 changes: 20 additions & 1 deletion nvx-rs/src/guest/time/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,8 @@ pub struct Restore {
memory_ranges: Vec<MemoryRange>,
/// The network settings that the restore packet carries, if any.
network: Option<RestoreNetwork>,
/// The activation record that the restore packet ends with, if any, undecoded.
activation: Option<Vec<u8>>,
/// Whether the call finished the restore.
finished: bool,
/// The restore's fresh entropy, in the in-memory mode, until the caller takes it.
Expand Down Expand Up @@ -657,6 +659,15 @@ impl Restore {
pub fn network(&self) -> Option<RestoreNetwork> {
self.network
}

/// Returns the activation record that the restore packet ends with: the bytes, 1 to
/// [`MAX_RECORD_BYTES`](crate::activation::MAX_RECORD_BYTES) of them, that the host handed a
/// platform clone with its restore, which the time ABI does not decode, or `None` for a
/// restore that handed over none. A clone decodes them with
/// [`ActivationRecord::decode`](crate::activation::ActivationRecord::decode).
pub fn activation_record(&self) -> Option<&[u8]> {
self.activation.as_deref()
}
}

/// The network settings of a restore that replaced the network policy of a snapshot without a
Expand Down Expand Up @@ -869,7 +880,13 @@ fn read_packet(runtime: &Runtime<'_>, ports: &PortAccess<'_>, recorded: u32) ->
}
let mut body = vec![0_u8; header.body_size()];
portb_read(ports, Port::PortbData, &mut body);
let packet = RestorePacket::parse(header, &body);
// The activation record follows the body, which ends with its length, a multiple of four
// bytes after the header, so its reads start on a four-byte boundary of the packet.
let packet = RestorePacket::parse(header, &body, |size| {
let mut record = vec![0_u8; size];
portb_read(ports, Port::PortbData, &mut record);
record
});
// The packet owns its entropy now, in the seed's allocation.
sys::zeroize(&mut body);
PacketRead::Packet(packet.map(|packet| {
Expand Down Expand Up @@ -940,6 +957,7 @@ fn repair_restore(
ranges,
entropy,
network,
activation,
} = packet;
let generation_id = entropy.generation_id();
if generation_id == target.previous_id {
Expand Down Expand Up @@ -1022,6 +1040,7 @@ fn repair_restore(
.map(|&(start, length)| MemoryRange { start, length })
.collect(),
network,
activation,
finished,
seed,
}))
Expand Down
Loading
Loading