Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,14 +152,15 @@ monotonic time advanced by the downtime. The captures share one downtime
window, so the scenario adds about a minute per backend.
Before each capture, the guest starts a CPU-bound workload in a cgroup of its
own and freezes it, as guest preparation freezes the workload for paired
scratch, arms a one-hour timer, and, once its time daemon's next discipline
poll has passed, records its clocks, its frequency correction, the workload's
CPU time and cgroup usage, and the length of its kernel log. The daemon changes
the correction only at its polls, which take the port lock that the capture
takes too, 16 s apart at first and 64 s apart once it has settled, so the
correction recorded is the one that the capture holds. Each capture comes
within seconds of its guest's boot, so the wait adds up to about 16 s to it,
and the capture allows this preparation 90 s beyond the scenario's timeout. The
scratch, arms a one-hour timer, records its clocks, its frequency correction,
the workload's CPU time and cgroup usage, and the length of its kernel log, and
leaves a slew of its clock in flight: an adjtime of 5 ms, which the kernel
slews at 500 ppm for 10 s, and whose rate it caches in its tick length from the
next second on, which a second's wait lets pass. Each capture's OpenVMM boots
with the `host-rate-offset-ppm` test hook at 100 ppm, so that the guest holds a
large frequency correction, which the host's own, at most 250 ppm on a
qualified host, composes with under Linux's 500 ppm limit, and the capture
allows this preparation 15 s beyond the scenario's timeout. The
scenario then restores each snapshot again after a simulated year: OpenVMM's
`downtime-add-s` test hook adds what brings the downtime to 60 s below the
time ABI's
Expand All @@ -172,8 +173,10 @@ downtime from the expected source. Against its realtime clock, which must
stand within 10 s of host UTC, its raw monotonic clock must have advanced by
the simulated downtime at the declared TSC rate, to the kHz, within 1 s plus
0.5 ppm for the kernel's rounded clocksource multiplier, and its monotonic and
boot-time clocks by that times one plus the frequency correction that it held
before the capture, within 5 ppm more for a phase slew then. The timer must
boot-time clocks by the simulated downtime itself, within 1 s plus 1 ppm,
whatever frequency correction and slew it held before the capture, since the
capture holds the declared rate: clocks that kept the 100 ppm and the slew,
about 555 ppm, would stand some 17,000 s beyond it. The timer must
fire within 10 s of the restore's clock step, before or after it, the frozen
workload must gain no CPU time or cgroup usage across the restore and about a
second's in the second that it then runs, and the restore must print its
Expand All @@ -191,7 +194,7 @@ arithmetic: about one restore in twenty then dereferenced NULL in
`pick_task_fair()` and left its processor spinning on the runqueue lock. Last,
OpenVMM must refuse a restore of the first snapshot with the bound plus one
second, with `E_DOWNTIME_EXCESSIVE`, before the guest runs. The simulated
restores and the captures' waits add about two minutes per backend.
restores add about a minute per backend.
The `time-abi-conformance` scenario boots the largest requested vCPU count and
runs the guest's exhaustive CI check, `/sbin/nvx-time exhaustive`, which the
boot check leaves to CI. On every online CPU it checks every leaf
Expand Down
48 changes: 36 additions & 12 deletions doc/design/3.2-clock-and-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,14 +320,26 @@ the TSC has advanced by `D` plus at most the restore latency `R`. The guest
observes no other discontinuity between the snapshot request and the first
restored instruction.

The guest kernel counts that advance as it counts any run of the TSC: its raw
monotonic clock at the declared rate, to the kHz, and its monotonic and
boot-time clocks at the frequency correction that the guest held at the
capture, which follows the host's
[clock correction](#329-time-samples) and stays within the kernel's ±500 ppm
limit. Those two clocks therefore advance by `D` times one plus that
correction, which over a year differs from `D` by up to 4.4 hours. Restore
steps the realtime clock to host UTC, so it keeps no such difference.
The guest kernel counts that advance as it counts any run of the TSC, whole,
at the rate of its clocks when it first reads the TSC after the restore, which
is their rate at the capture. Its raw monotonic clock counts it at the
declared rate, to the kHz. Its monotonic and boot-time clocks count it at the
rate that the kernel's frequency correction gives them, which the guest holds
at the host's [clock correction](#329-time-samples), within the kernel's
±500 ppm limit, and which would make them advance by `D` times one plus that
correction, up to 4.4 hours more or less than `D` over a year. The guest
therefore holds the correction that counts the TSC at exactly the declared
rate, `F_k / F - 1` for the kHz `F_k` to which the kernel rounds `F`, from its
capture request to the capture anchor, in place of the host's, and clears the
phase slew that its discipline may have under way, because the kernel keeps
adding that slew's rate to its clocks' for up to a second after the phase
offset is cleared. Its monotonic and boot-time clocks then advance by `D`, to
the resolution of the kernel's clocksource multiplier: within 1 ppm of `D`, 32 s
over a year. The guest holds the host's correction again when restore repair
steps the clock, or when a capture that continues without a restore returns,
as [guest preparation](5.3-snapshot-protocol.md#532-guest-preparation) and
[guest repair](5.5-restore-protocol.md#557-guest-repair) say. Restore steps
the realtime clock to host UTC, so it keeps no such difference.

The guest's scheduler clock reads the raw TSC, which advances by the downtime,
so the downtime would be runtime of the task that each processor ran at the
Expand Down Expand Up @@ -394,6 +406,15 @@ can oops before its guest rejects the packet, which hung about one such
restore in twenty in qualification. Snapshots captured before the change
therefore keep their 30-day window; only a recapture gets the longer one.

Holding the declared rate through the capture request changes no contract
version either: it belongs to the guest's own capture and repair steps, on
which no VMM or snapshot format depends, and a restore still advances the TSC
by `D` at the declared rate. It too holds only for a snapshot whose guest has
it: a snapshot of an earlier guest, which kept the host's correction through
its capture, counts the downtime in its monotonic and boot-time clocks as that
guest's kernel did, `D` times one plus that correction, until it is captured
again.

The restore anchor is the instant of the synchronized TSC set. Guest time runs
from that instant, or, on a backend that writes the TSCs with partition time
frozen, from the instant at which partition time resumes after the write, and
Expand Down Expand Up @@ -598,10 +619,13 @@ the kernel's limit is held at the limit and reported in the state file, so an
offset can grow until the discipline steps it. A sample whose generation
differs from the recorded one is discarded, because a
restore happened and restore repair resets the discipline. The discipline
follows host wall-clock steps and never powers off the guest. Restore repair
and the discipline step the clock through the kernel, so a workload can
observe every step through the kernel's clock-set notification, such as a
`timerfd` armed with `TFD_TIMER_CANCEL_ON_SET`.
follows host wall-clock steps and never powers off the guest. Restore repair,
the discipline, and a capture request step the clock through the kernel, so a
workload can observe every step through the kernel's clock-set notification,
such as a `timerfd` armed with `TFD_TIMER_CANCEL_ON_SET`; the capture
request's step is by zero, and keeps the clock as it is. The capture request
also holds the clock at the declared rate until the capture ends, as
[Downtime and Continuity](#328-downtime-and-continuity) describes.

The [snapshot protocol](5.3-snapshot-protocol.md) and
[guest repair](5.5-restore-protocol.md#557-guest-repair) define the snapshot
Expand Down
14 changes: 11 additions & 3 deletions doc/design/5.3-snapshot-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,12 @@ Before they write the snapshot port, the guest agents:
4. Write back the pages that shared mappings dirtied on the mounted
[host shares](3.6-host-filesystem.md), and wait for the writes.
5. Write the capture request, with its scratch policy, to the snapshot port.
Just before the write, the time tools hold the wall clock at the declared
TSC rate, with no phase slew in flight, under the byte channel's lock, which
keeps the discipline's polls from changing the rate before the capture
anchor, so that the kernel counts the restore's advance of the TSC by the
downtime in its monotonic and boot-time clocks as the downtime itself; see
[Downtime and Continuity](3.2-clock-and-time.md#328-downtime-and-continuity).

Step 4 keeps captured memory from holding file data that the host files lack,
which every restore would otherwise write back to the live host files. The
Expand Down Expand Up @@ -143,9 +149,11 @@ A committed capture terminates the source, so the write returns in the source
only when the capture had no destination or failed before its commit point.
The guest agents tell the cases apart by the restore-packet status bit, which
is clear in a resumed source and set in a restored process. In a resumed
source, they remove their barriers and restore the detector settings, and a
failure to do so powers off the guest. A rejected capture likewise thaws every
guest-owned barrier and then releases the stall-detector suppression.
source, the time tools hold the host's clock correction again before the
request returns, and the guest agents remove their barriers and restore the
detector settings, and a failure to do so, or to hold the correction, powers
off the guest. A rejected capture likewise thaws every guest-owned barrier and
then releases the stall-detector suppression.

## 5.3.3 Capture Transaction

Expand Down
3 changes: 2 additions & 1 deletion doc/design/5.5-restore-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,8 @@ runs before the acknowledgement:
packet's generation, until one pairs host UTC with the guest TSC within
100 µs, step the clock by its offset, and hold the frequency correction
that its host clock correction gives, which carries the rate deviation,
within ±500 ppm.
within ±500 ppm, in place of the declared-rate correction that the capture
request held.
4. Activate processors and memory as requested, then apply the network settings
of a packet that carries them as the guest's init applies a cold boot's: name
the DNS server in `/etc/resolv.conf`, and, when the packet asks for the
Expand Down
8 changes: 8 additions & 0 deletions nvx-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1384,6 +1384,14 @@ The request's mode selects how a restore hands over its fresh entropy: the file
mode, which `nvx-time` keeps, writes it to a file, and the in-memory mode
returns it as the `Restore`'s owned `Seed`, and writes no file.

Just before it writes the snapshot port, a `CaptureRequest` holds the wall
clock at the declared TSC rate, without the host's clock correction and without
a phase slew in flight, so that a restore's advance of the TSC by the downtime
counts in the monotonic and boot-time clocks as the downtime itself; the repair
of a restore holds the host's correction again, and so does a request that
returns in a VM that continues. A hold that fails is a violation of
conformance check C10, which fails the request before it writes the port.

A violation of the ABI stops the VM in enforcing mode, with the time ABI's
status, as `nvx-time` does; in strict mode, `TimeAbi::strict`, which a caller
that owns the VM's exit status and shutdown selects, the call fails with the
Expand Down
49 changes: 45 additions & 4 deletions nvx-rs/src/guest/sys/discipline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,21 @@ pub(crate) enum ClockAdjustment {
/// The frequency correction and the error estimates.
hold: FrequencyHold,
},
/// Sets the frequency correction alone, and keeps the status, the error estimates, and the
/// phase offset.
SetFrequency(i64),
/// Makes the wall clock run at the frequency correction of `hold` alone, from the moment of
/// the call: it steps the wall clock by zero, which clears the kernel's NTP state, the phase
/// offset and the tick length's cached correction of it included, which a clearing of the
/// phase offset alone leaves in force until the kernel's next second, and then sets the
/// frequency correction and the error estimates of `hold`, and `status`, which the step reset
/// to unsynchronized.
Pin {
/// The frequency correction and the error estimates.
hold: FrequencyHold,
/// The discipline's status, as the kernel reports it.
status: i32,
},
}

/// The frequency correction that the discipline holds, and the error estimates that it sets with
Expand All @@ -68,14 +83,17 @@ impl ClockAdjustment {
Self::Frequency(_) => HOLD,
Self::ClearPhase => libc::ADJ_OFFSET | libc::ADJ_NANO,
Self::Slew { .. } => HOLD | libc::ADJ_OFFSET | libc::ADJ_TIMECONST | libc::ADJ_NANO,
Self::SetFrequency(_) => libc::ADJ_FREQUENCY,
Self::Pin { .. } => HOLD | libc::ADJ_SETOFFSET | libc::ADJ_NANO,
}
}

/// Returns the discipline status that the adjustment sets, or zero for one that sets none.
pub(crate) fn status(self) -> i32 {
match self {
Self::Query | Self::Step { .. } | Self::ClearPhase => 0,
Self::Query | Self::Step { .. } | Self::ClearPhase | Self::SetFrequency(_) => 0,
Self::Frequency(_) | Self::Slew { .. } => PLL_NANOSECONDS,
Self::Pin { status, .. } => status,
}
}
}
Expand All @@ -85,6 +103,24 @@ impl ClockAdjustment {
pub(crate) struct ClockDiscipline {
/// Frequency correction in parts per million, scaled by 2^16.
pub(crate) scaled_ppm: i64,
/// Maximum error in microseconds.
pub(crate) maximum_error_us: i64,
/// Estimated error in microseconds.
pub(crate) estimated_error_us: i64,
/// The discipline's status bits, `STA_*`.
pub(crate) status: i32,
}

impl ClockDiscipline {
/// Returns the hold of the frequency correction `scaled_ppm` with this discipline's error
/// estimates.
pub(crate) fn hold(self, scaled_ppm: i64) -> FrequencyHold {
FrequencyHold {
scaled_ppm,
maximum_error_us: self.maximum_error_us,
estimated_error_us: self.estimated_error_us,
}
}
}

impl FrequencyHold {
Expand All @@ -104,7 +140,7 @@ pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result<ClockDisciplin
value.modes = adjustment.modes();
value.status = adjustment.status();
match adjustment {
ClockAdjustment::Query => {}
ClockAdjustment::Query | ClockAdjustment::ClearPhase => {}
ClockAdjustment::Step {
seconds,
nanoseconds,
Expand All @@ -115,8 +151,10 @@ pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result<ClockDisciplin
tv_usec: nanoseconds,
};
}
ClockAdjustment::Frequency(hold) => hold.apply(&mut value),
ClockAdjustment::ClearPhase => {}
ClockAdjustment::Frequency(hold) | ClockAdjustment::Pin { hold, .. } => {
hold.apply(&mut value);
}
ClockAdjustment::SetFrequency(scaled_ppm) => value.freq = scaled_ppm,
ClockAdjustment::Slew {
offset_ns,
time_constant,
Expand All @@ -131,6 +169,9 @@ pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result<ClockDisciplin
Errno::result(unsafe { libc::adjtimex(&raw mut value) })?;
Ok(ClockDiscipline {
scaled_ppm: value.freq,
maximum_error_us: value.maxerror,
estimated_error_us: value.esterror,
status: value.status,
})
}

Expand Down
28 changes: 28 additions & 0 deletions nvx-rs/src/guest/sys/discipline/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,22 @@ fn adjustments_select_the_kernel_modes_and_status() {
time_constant: 4,
hold,
};
let set_frequency = ClockAdjustment::SetFrequency(-7);
let pin = ClockAdjustment::Pin {
hold,
status: libc::STA_PLL | libc::STA_NANO,
};
assert_eq!((step.modes(), step.status()), (0x2100, 0));
// The status holds the frequency, `STA_FREQHOLD`, with the loop and nanoseconds.
assert_eq!((frequency.modes(), frequency.status()), (0x001e, 0x2081));
// Clearing the nanosecond offset changes neither status nor frequency.
assert_eq!((clear_phase.modes(), clear_phase.status()), (0x2001, 0));
assert_eq!((slew.modes(), slew.status()), (0x203f, 0x2081));
// Setting the frequency alone changes nothing else.
assert_eq!((set_frequency.modes(), set_frequency.status()), (0x0002, 0));
// The pin is a step by zero, which resets the status and the error estimates, with a hold
// of them, and of the frequency, whose status is the caller's.
assert_eq!((pin.modes(), pin.status()), (0x211e, 0x2001));
assert_eq!(
(
ClockAdjustment::Query.modes(),
Expand All @@ -35,6 +45,24 @@ fn adjustments_select_the_kernel_modes_and_status() {
);
}

#[test]
fn a_discipline_holds_a_frequency_with_its_own_error_estimates() {
let discipline = ClockDiscipline {
scaled_ppm: 1_000,
maximum_error_us: 70,
estimated_error_us: 3,
status: libc::STA_PLL,
};
assert_eq!(
discipline.hold(-3),
FrequencyHold {
scaled_ppm: -3,
maximum_error_us: 70,
estimated_error_us: 3,
}
);
}

#[test]
fn a_hold_writes_the_frequency_and_the_error_estimates() {
// SAFETY: all zeros is the kernel's baseline for every field of `timex`.
Expand Down
Loading
Loading