Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Guide/src/reference/openvmm/management/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ describes the source definitions.
in progress; Windows flush completion waits for the named-pipe peer to
consume the complete frame.
* `--restore-entropy`: Accepted for compatibility; it has no effect. Every
microVM restore makes restore packet version 4 available on the private
microVM restore makes restore packet version 5 available on the private
portb restore channel. The packet carries 64 bytes of fresh entropy, the
online-VP target, the memory-expansion ranges, the downtime, and the TSC
rate deviation; the guest must consume it and explicitly reseed its RNG.
Expand Down
4 changes: 2 additions & 2 deletions Guide/src/reference/openvmm/management/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ restore flow over both transports:
client requires the matching path configuration.
* `restore_entropy` is accepted for compatibility and is valid only with
`restore_path`; it has no effect, because every restore delivers restore
packet version 4 with fresh entropy. Every microVM process also receives a
packet version 5 with fresh entropy. Every microVM process also receives a
fresh, non-serialized 16-byte generation ID through the fixed portb selector.
On restore, the ID is the first 16 bytes of the restore packet's entropy.
* `restore_processor_count` requests restore-time activation of the contiguous
Expand All @@ -198,7 +198,7 @@ restore flow over both transports:
snapshot base and immutable capacity. Zero selects the base. Expansion uses
fresh private zeroed backing, implies the post-restore repair gate, and is
rejected for snapshots without a RAM capacity contract. A nonzero value sets
the memory-target flag of restore packet version 4, with the expansion
the memory-target flag of restore packet version 5, with the expansion
ranges (none at the base size); zero leaves the flag clear.
* `restore_gate_timeout_ms` bounds gated guest repair. Zero selects the
60-second default; a nonzero value is valid only with `restore_path`.
Expand Down
63 changes: 48 additions & 15 deletions Guide/src/user_guide/openvmm/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,34 +219,50 @@ the ID from the first 16 bytes of the restore packet's fresh entropy, so the
guest can reject an unchanged clone identity, reseed Linux, refresh runtime
identifiers, and acknowledge the input gate without another PMIO transfer.

Every microVM restore exposes restore packet version 4. Status bit 1
Every microVM restore exposes restore packet version 5. Status bit 1
advertises it, and writing `0xa5` to the status port selects it for reading
from the data port. Its 32-byte little-endian header holds:
from the data port. Its 24-byte little-endian header holds:

- the magic `OVR` and version 4;
- the magic `OVR` and version 5;
- flags: bit 0, the downtime came from host UTC; bit 1, a memory target;
bit 2, the guest must acknowledge the restore through port `0x605`; bit 3,
a test hook is active;
- the online-VP target (0 for none), the memory-range count, and a reserved
zero byte;
- the generation counter (`u32`), which is 0 at cold boot and one more than
the snapshot's at each restore;
- the TSC rate deviation (`i32`, ppm scaled by 2^16);
- the downtime in nanoseconds (`u64`); and
- host UTC in nanoseconds since the Unix epoch (`u64`), latched when the guest
first selects the packet.
- the TSC rate deviation (`i32`, ppm scaled by 2^16); and
- the downtime in nanoseconds (`u64`).

The `(u64 GPA start, u64 length)` memory ranges and 64 bytes of fresh entropy
follow the header. Status bits 2, 3, and 4 report an online-VP target, a
memory target, and one or more memory ranges, so a zero-range target can skip
post-restore repair.
post-restore repair. The packet carries no host UTC: the guest sets its clock
from a time sample.

Status bit 6 advertises the time sample. Writing `0xa7` to the status port
latches a 16-byte sample: version 1, flags (bit 3: a test hook is active), two
reserved zero bytes, the generation counter (`u32`), and host UTC in
nanoseconds (`u64`). The guest reads it from window port `0xeb`, so samples
never interleave with console input on the data port. The selected packet and
the window can be read one, two, or four bytes at a time.
latches a 48-byte little-endian sample, version 2:

- the version, flags (bit 3: a test hook is active), the status of host UTC
(0 unknown, 1 synchronized, 2 free-running, 3 unreliable), and a reserved
zero byte;
- the generation counter (`u32`);
- host UTC in nanoseconds since the Unix epoch (`u64`), and the guest TSC at
the same instant (`u64`), which OpenVMM pairs on the host;
- the most those two instants can differ, in nanoseconds (`u32`);
- a disruption counter (`u32`), which changes when host UTC steps, the host
resumes from a suspension, or the guest TSC's offset from the host TSC
changes;
- the host's clock correction (`i64`): how fast host UTC runs against the
guest TSC counted at the declared rate, minus one, in ppm scaled by 2^16;
and
- a bound on the error of that correction (`u32`, ppm scaled by 2^16) and
the time service's bound on the error of host UTC (`u32`, microseconds),
each `0xffffffff` when unknown.

The guest reads it from window port `0xeb`, so samples never interleave with
console input on the data port. The selected packet and the window can be
read one, two, or four bytes at a time.

A microVM template may opt into restore-time vCPU activation by booting with an
explicit canonical `maxcpus=1`, `2`, `4`, or `8` value below or equal to its
Expand Down Expand Up @@ -300,7 +316,7 @@ validation error and refuse to start.

## Time and CPU compatibility

Every microVM uses the NVX time ABI v1. The guest sees a minimal Hyper-V
Every microVM uses the NVX time ABI 2. The guest sees a minimal Hyper-V
identity: CPUID leaves `0x40000000` through `0x40000005`, explicit zero leaves
`0x40000006` through `0x4000000f` and `0x40000080` through `0x40000082`, and
identity MSRs that return the declared TSC rate (`0x40000022`) and LAPIC rate
Expand All @@ -314,6 +330,20 @@ HWCR (`0xc0010015`) only `TscFreqSel`, so Linux takes the TSC to count at the
P0 frequency, and DE_CFG (`0xc0011029`) only `LFENCE` serialization. Writing
the value read back changes nothing; any other write raises #GP.

Each time sample pairs host UTC with the guest TSC on the host, so the guest
needs no tight bracket of its own around the port write: the guest TSC is the
host TSC plus an offset, which KVM reports, and which MSHV and WHP measure
from VP 0's TSC at every start, before any VP runs. Each sample also
publishes the host's clock correction, which the guest applies instead of
estimating it. OpenVMM reads it from the host's time service: on Linux, the
frequency correction, tick, status, and maximum error that `adjtimex`
reports, and on Windows, the time adjustment that
`GetSystemTimeAdjustmentPrecise` reports, with an adjustment that Windows
does not apply reported as a free-running clock. The correction refers to
the declared rate, so on a restore it includes the deviation of the native
rate. A paravirtual clock is still not exposed: the samples only discipline
the guest's wall clock, and the TSC remains its clocksource.

The guest CPU is a pinned CPU profile: one per CPU generation, the same on
every backend. OpenVMM pins `intel.skylake-sp.v1` (Xeon Scalable, first
generation: family 6, model 85, steppings 0 to 4), `intel.icelake-sp.v1`
Expand Down Expand Up @@ -375,6 +405,8 @@ others.

Restore also requires:

- a snapshot of the same time ABI version, so a snapshot of time ABI 1 must
be recaptured (`E_MANIFEST_TIME`);
- the backend that captured the snapshot;
- a native TSC rate within 250 ppm of the snapshot's
(`E_TSC_RATE_TOLERANCE`), and the identical LAPIC rate
Expand All @@ -383,7 +415,8 @@ Restore also requires:
`E_DOWNTIME_EXCESSIVE`).

The declared rates never change across a restore; the restore packet reports
the rate deviation so the guest can compensate. The downtime is host monotonic
the rate deviation, and every time sample's correction includes it, so the
guest can compensate. The downtime is host monotonic
time when capture and restore run on the same host boot, and host UTC
otherwise. Capture and restore reject an armed periodic or TSC-deadline LAPIC
timer and a periodically counting PIT channel 0 (`E_LAPIC_PERIODIC`,
Expand Down
42 changes: 36 additions & 6 deletions openvmm/openvmm_core/src/worker/dispatch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2052,18 +2052,40 @@ impl InitializedVm {
}),
&parameters.hooks,
)?;
// The portb device takes the host fields of its time samples
// from the clock, which pairs host UTC with the guest TSC
// through the offset that the unit measures.
let clock = Arc::new(time_abi::TimeAbiClock::new(
msrs.clone(),
&report,
&parameters.hooks,
));
resolver.add_resolver(time_abi::TimeSampleSourceResolver(clock.clone()));
let measure_offset: time_abi::MeasureOffset = {
let partition = partition.clone();
let hypervisor = hypervisor.clone();
Box::new(move || time_abi::guest_tsc_offset(partition.as_ref(), &hypervisor))
};
let unit = state_units
.add(time_abi::TIME_ABI_UNIT)
.spawn(driver_source.simple(), {
let msrs = msrs.clone();
let report = report.clone();
move |recv| {
state_unit::run_unit(time_abi::TimeAbiUnit { msrs, report }, recv)
state_unit::run_unit(
time_abi::TimeAbiUnit {
msrs,
report,
clock,
measure_offset,
},
recv,
)
}
})
.unwrap();
Some(time_abi::TimeAbiState {
_unit: unit,
unit,
msrs,
profile,
effective_cpuid,
Expand Down Expand Up @@ -3376,12 +3398,20 @@ impl InitializedVm {
// create a new channel to intercept guest resets
let (halt_send, halt_recv) = mesh::channel();

let partition_unit_builder = state_units
.add("partition")
.depends_on(devices.chipset_unit())
.depends_on(vmtime.handle());
// The time ABI unit measures the guest TSC's offset for the time
// samples at every start, before any VP runs.
#[cfg(guest_arch = "x86_64")]
let partition_unit_builder = match &time_abi_state {
Some(time_abi) => partition_unit_builder.depends_on(time_abi.unit.handle()),
None => partition_unit_builder,
};
let (partition_unit, vp_runners) = PartitionUnit::new(
driver_source.simple(),
state_units
.add("partition")
.depends_on(devices.chipset_unit())
.depends_on(vmtime.handle()),
partition_unit_builder,
partition.clone().into_vm_partition(),
PartitionUnitParams {
processor_topology: &processor_topology,
Expand Down
Loading
Loading