Repository navigation
guest: hold the declared TSC rate through a capture request - #1584
Conversation
This pull request changes existing testsIt modifies or removes 16 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests. Test code
End-to-end tests
Relocated unchanged: 0; added: 6. Compared the merge base |
There was a problem hiding this comment.
🟡 Changes recommended
The privilege documentation omits capture’s new CAP_SYS_TIME requirement, so embedders following the documented permissions can encounter C10 and VM shutdown in enforcing mode even without a restore.
1 open issue
What changed in this PR
This PR updates the guest capture path so monotonic and boot-time clocks count snapshot downtime at the declared TSC rate, independent of the host’s clock correction. It advances #516 while leaving template restore windows at 30 days.
Changes:
- Pin the clock rate and clear pending slew before capture; restore the host correction after restore or continuation.
- Strengthen simulated-year checks with an injected frequency correction, an active slew, and tighter clock tolerances.
- Document capture behavior, clock-set notifications, and compatibility with older snapshots.
| File | Description |
|---|---|
scripts/test_microvm_tests.py |
Update scenario tests and verify clock-tolerance boundaries. |
scripts/nvx_tools/microvm_tests.py |
Inject host correction and check declared-rate downtime advances. |
scripts/nvx_tools/microvm_test_scripts/restore-year-capture.sh |
Start a pending slew without waiting for a daemon poll. |
nvx-rs/src/guest/time/test_mode.rs |
Handle the new clock-adjustment variants. |
nvx-rs/src/guest/time/system/tests.rs |
Test rate holds, resumption, and adjustment failures. |
nvx-rs/src/guest/time/system.rs |
Implement declared-rate holds and frequency resumption. |
nvx-rs/src/guest/time/api/tests.rs |
Verify capture adjustment ordering and failure reporting. |
nvx-rs/src/guest/time/api.rs |
Integrate rate holds into capture and continuation. |
nvx-rs/src/guest/sys/discipline/tests.rs |
Verify new adjustment modes and preserved error estimates. |
nvx-rs/src/guest/sys/discipline.rs |
Add clock pinning and frequency-only adjustments. |
nvx-rs/README.md |
Describe capture rate holds and failure behavior. |
doc/design/5.5-restore-protocol.md |
Clarify restoration of the host correction. |
doc/design/5.3-snapshot-protocol.md |
Document capture pinning and continuation recovery. |
doc/design/3.2-clock-and-time.md |
Explain clock accuracy, compatibility, and notifications. |
doc/ci.md |
Update simulated-year preparation and acceptance criteria. |
Review effort: Balanced
A restore advanced the guest's monotonic and boot-time clocks by the downtime D times one plus the frequency correction that the guest held at its capture, not by D. The kernel counts a TSC advance whole, at the rate of its clocks' multiplier when it first reads the TSC afterwards, which is their rate at the capture, and nvx-time holds the host's clock correction, within 500 ppm, as that rate. Over the year that the time ABI now admits, the clocks stood up to 4.4 hours from the downtime. The restore-downtime scenario's simulated year ended short of it by 461 s on KVM, 1,390 s on MSHV, and 13 to 24 s on WHP, which hold -14.7, -44.3, and -0.6 ppm (#516). Make the capture request hold the declared rate. Just before it writes the snapshot port, under the port-B lock that keeps the discipline's polls out until the request returns, it queries the discipline and pins it with one adjtimex call: a step by zero, which clears the kernel's NTP state and the phase slew that the tick length caches until its next second, with the frequency correction F_k / F - 1 that counts the TSC at exactly the declared rate F, for the kHz F_k to which the kernel rounds it, and the status and error estimates that the query found. The monotonic and boot-time clocks then advance by D, within the 1 ppm of the kernel's clocksource multiplier. A restore's repair holds the host's correction again, as it did, and a request that returns in a VM that continues sets back the frequency that it found. The next poll slews the phase that the step cleared. A repair that fails before its step leaves the declared rate in force, with the clock unstepped, and enforcing mode stops the VM. A hold that fails fails the request before it writes the port, as a violation of the existing check C10, so that the request adds no API, event, or error code. The request now needs CAP_SYS_TIME in a VM that continues too, which the repair of a restore already needed to set the clock, and the rustdoc of TimeAbi lists it. The step by zero notifies a workload that watches the clock for steps with TFD_TIMER_CANCEL_ON_SET, as restore repair's step does; 3.2.10 says so. Clearing the phase alone, with ADJ_OFFSET, is not enough: the kernel keeps the cached slew, up to 500 ppm of a year, in force until its next second. No contract version changes. The hold belongs to the guest's capture and repair steps, on which no VMM or snapshot format depends, and a restore still advances the TSC by D at the declared rate. It holds only for a snapshot whose guest has it: a snapshot of an earlier guest keeps counting the downtime at its correction until it is captured again. 3.2.8 documents this, 5.3 and 5.5 the capture and repair steps. The restore-downtime scenario now checks the fix. Each capture's OpenVMM boots with the host-rate-offset-ppm hook at 100 ppm, so that the guest holds a large correction, and the guest leaves an adjtime slew of 500 ppm in flight at its request. The restored guest's monotonic and boot-time clocks must advance by the simulated downtime, within 1 s plus 1 ppm, whatever it held before; its raw clock keeps the declared-rate check. The wait for the daemon's next poll, which made the recorded correction the held one, goes, which makes the scenario about a minute shorter. With this harness, the unfixed guest ends 17,490 s beyond the downtime on MSHV, and a variant that holds the rate but leaves the slew in flight 15,770 s. With the fix, all four restores end 1.7 s short of it on MSHV, 2.5 s short on WHP, and 1.1 s beyond on KVM. Tests: the capture's adjustments in each mode; that a declared rate of 2,300,000,123 Hz holds -3,505 scaled ppm, a whole-kHz rate none, and that a hold that fails reports its cause, and touches nothing without a rate; that a kernel that refuses the hold fails the request before the snapshot port, with its event on the debug console; the discipline's modes and status; the harness's bounds on both clocks, and its pre-capture script's slew. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
b91fbda to
c0279c4
Compare

Part of #516
Task left open: the restore window of
nvx-rstemplates, which stays at 30days instead of one year. It needs a capability change, which this pull request
does not make; see "Why the template window stays open" below.
What and why
#1484 made OpenVMM and the guest accept a snapshot downtime of up to 365 days,
and the issue stayed open for two tasks, which its status comment lists. This
pull request does the first: the guest's monotonic and boot-time clocks advance
across the downtime by
D, not byDtimes one plus the guest's frequencycorrection.
Root cause: a restore advances the guest TSC by the downtime
Dat the declaredrate. The kernel counts a TSC advance whole, at the rate of its clocks' multiplier
when it first reads the TSC afterwards, which is their rate at the capture. The
time daemon holds the guest's frequency correction at the host's clock correction,
within the kernel's ±500 ppm limit, so the monotonic and boot-time clocks advanced
by
Dtimes one plus that correction: up to 4.4 hours off over a year. At thebase revision, the
restore-downtimescenario's simulated year ended short of thedowntime by 461 s on KVM (held correction −14.7 ppm), 1,390 s on MSHV (−44.3 ppm),
and 13 to 24 s on WHP (−0.6 ppm).
The fix is the one that the issue's status comment prescribes, in the capture path
of the guest alone:
which keeps the discipline's polls out until the request returns, the time tools
query the discipline and pin it with one
adjtimexcall. It steps the clock byzero, which makes the kernel clear its NTP state, and with it the phase slew whose
rate the kernel's tick length caches until its next second. It sets the frequency
correction
F_k / F - 1, which counts the TSC at exactly the declared rateFfor the kHz
F_kto which the kernel rounds it, and the status and errorestimates that the query found. The monotonic and boot-time clocks then advance
by
D, within the 1 ppm of the kernel's clocksource multiplier.returns in a VM that continues, because the capture had no destination or rolled
back, sets back the frequency that it found, and the discipline's next poll slews
the phase that the step cleared.
the existing check C10, with its event on OpenVMM's debug console. The change adds
no API, event, or error code. The request now needs
CAP_SYS_TIMEin a VM thatcontinues too, which the repair of a restore already needed to set the clock, and
the rustdoc of
TimeAbilists it.restore-downtimechecks it. Each capture's OpenVMM boots with thehost-rate-offset-ppmtest hook at 100 ppm, so that the guest holds a largecorrection, and the guest leaves an
adjtimeslew of 500 ppm in flight at itsrequest. The restored guest's monotonic and boot-time clocks must advance by the
simulated downtime, within 1 s plus 1 ppm, whatever it held; the raw monotonic
clock keeps its declared-rate check. The wait for the daemon's next poll, which
made the recorded correction the held one, goes: the scenario is about a minute
shorter.
The change is one commit:
guest: hold the declared TSC rate through a capture request. The guest and the scenario's expectation change together, since theunfixed guest fails the new check and the fixed guest fails the old one.
Decisions:
force until the kernel's next second. A variant that held the rate and cleared
the phase that way ended 15,770 s beyond the downtime on MSHV, 500 ppm of a year;
a step by zero flushes it. The cost is a clock-set notification at each capture,
as at each restore repair's step, which 3.2.10 of the architecture guide now says.
belongs to the guest's own capture and repair steps, on which no VMM or snapshot
format depends, and a restore still advances the TSC by
Dat the declared rate.It holds only for a snapshot whose guest has it: a snapshot of an earlier guest
keeps counting the downtime at its correction until it is captured again, which
3.2.8 documents with the release-level compatibility of the 365-day bound.
check, so the time ABI's events, codes, and versions stay as they are.
declared rate in force. Enforcing mode powers the VM off for the failure, and a
strict caller ends it, so only a report-only call, a diagnostic mode, continues;
the rustdoc of
TimeAbi::capturesays so.The kernel floors the multiplier of its frequency-corrected clock where its raw
clock rounds to the nearest, which leaves up to about 1.5 multiplier units: 0.2
ppm at 2.3 GHz, and about 0.9 ppm at 10 GHz. The measured residual is −0.08 to
+0.04 ppm.
Why the template window stays open
The second task,
nvx-rstemplates' restore window, stays open.MAX_DOWNTIME_NSin
nvx-rs/src/host/template/eligibility.rsis 30 days, the bound that everyruntime accepts. Widening it to a year is correct only for a capture whose
OpenVMM and guest accept a year, and nothing that a template records tells such a
capture from one of an older runtime:
TimeContract) holds the rates and the anchor, andno bound. The manifest's
openvmm_versionis OpenVMM's Cargo package version.compares, and its contract versions did not change with the bound.
library's window applies to every embedder's runtime, whose artifacts it never
reads or hashes.
refused by its OpenVMM with
E_DOWNTIME_EXCESSIVEor by its guest withG_REPAIR_PACKET, and neither refusal quarantines the generation, so the storewould keep selecting a generation that no clone can restore instead of capturing
a replacement. A guest from before the scheduler patch can also hang after about
104 days instead of refusing.
A correct longer window per generation needs the capturing runtime to state the
bound that it accepts: a capability change, which is a version. The change stays
for a decision that allows one.
Validation
Head submitted to CI:
c0279c42ce433a7ec344f62547a4ba5c64d2edd9, on4b2ad587527,the head of
dev. The rebase onto it from6e7a1320047, where the runs below weremade, moved only
data/*.csv, so the sources that they ran are this head's, butfor the rustdoc of
TimeAbithat Copilot's review changed, after which the Linuxchecks below, but for the macOS target check, ran again.
Local, on Windows:
python scripts/local_gates.py --base origin/dev --run, ingroups, passed for every gate:
test-changes(16 changes to existing tests,explained below),
markdown-links,test-layout,quality-code(ruff, pyright,ShellCheck, shfmt),
nvx-rs-format,nvx-rs-lint,nvx-rs-test,nvx-rs-bundling,nvx-rs-doc,nvx-rs-msrv,nvx-rs-adapter-protocol, andnvx-rs-adapter-host. Fornvx-cli, I ran its commands but for thepip installof its first line, which would change the Python environment:
compileall, everyunit-test module that it lists, 2,199 tests,
scripts/test_adversarial.py, 84tests, and the
--helpchecks, all passing.python scripts/public_api.pyandpython scripts/nvx.py verifypass.Linux, on
nvx-kvm-test(Rust 1.97.1 and 1.89):cargo test --lockedwith allfeatures, with the defaults, and with
--lib --no-default-features, 1,907 librarytests with all features; the guest tests, 1,020 with
--features guest, and theprivileged
capture_in_each_modetest, which needsCAP_SYS_ADMIN, inunshare -Urm; the fiveclippy -D warningsinvocations ofcheck-nvx-rs;cargo docwith
-D warnings;cargo +1.89 check --all-targets --all-features; the macOStarget check; and
nvx.py check-guest-binaries, the musl lint and tests, allpassing.
Code review: a subagent reviewed the diff. It found that a failed hold's event
missed OpenVMM's debug console, because the capture's port transaction was still
open, which this change fixes and tests, and that a failed repair in report-only
mode leaves the declared rate in force, which the rustdoc now says. Copilot's
review of the first head asked for the capture's
CAP_SYS_TIMErequirement to bedocumented, which the head does.
End to end, on the dedicated test VMs, all of host type
virtual-machine, withrestore-downtimeat 1 and 8 vCPUs, each with and withoutrcupdate.rcu_expedited=1, one simulated year from both downtime sources. Thevalue is how far the restored guest's monotonic clock stands from the simulated
downtime, with the tolerance at 1 s plus 1 ppm, about 32 s. "Before" is the
unmodified scenario with the release
v0.1.0-dev.6e7a13200472artifacts; "after"is this change's scenario with a guest built from its sources, which the kernel
leaves unchanged:
nvx-kvm-testnvx-mshv-testnvx-whp-testOn MSHV, the same new scenario fails the unfixed guest with +17,490 s, and a guest
that holds the rate but leaves the slew in flight with +15,770 s. A first WHP run,
while others' builds kept the host at 67 to 86% CPU, printed no restore marker for
60 s in the 8-vCPU expedited restore after a real downtime, and its guest console
stopped in a warp probe; two reruns on an idle host passed, one of them with the
final guest binary.
Other scenarios with the fix: on
nvx-mshv-test,snapshot-core,snapshot-rollback,snapshot-tiers,time-host-correction,time-abi-conformance,restore-processors,restore-memory,smp-snapshot,console-snapshot, andscratch-snapshot; onnvx-kvm-test,snapshot-coreandsnapshot-rollback, all passing.Left to CI:
NVX CLI / Linux, the microVM shards of every scenario on all threebackends, the WHP scenarios beyond
restore-downtime, guest and kernel builds andtheir determinism, debug-kernel jobs, platforms and packages, the runtime
adapter's guest-only configuration, and the performance gate. No OpenVMM change
is involved, so OpenVMM's CI does not apply.
CI run 43562650301 on
c0279c42ce4passed all of these, with the performance gate and the requiredstatus check. In it,
restore-downtimerestored each of its snapshots, with thischange's guest, after simulated downtimes of 31,535,843 to 31,535,863 s on KVM,
MSHV, and WHP, at 1 and 8 vCPUs, from both downtime sources, with held corrections
of 95 ppm (KVM), 79 to 90 ppm (MSHV), and 107 ppm (WHP). The monotonic and
boot-time clocks stood +2.61 s (KVM), −2.25 s (MSHV), and −2.15 s (WHP) from the
simulated downtime, within the 32 s tolerance, where those corrections alone would
have put them about 2,500 to 3,400 s beyond it. The expired timer fired from 1.2 ms
to 1.52 s before the clock step, and every other check of the scenario passed.
Changes to existing tests
python scripts/check_test_changes.py --base origin/devlists 16 changes toexisting tests, which the label
tests-modifiedmarks. Each follows from the newbehavior, and none weakens a check: the unchanged expectations of the old tests
hold wherever the behavior did not change.
Rust:
guest::time::api::tests::capture_in_each_modealso asserts the capture's clockadjustments in each mode: the hold, then the repair's step and held frequency
in a restore, and the frequency set back in a VM that continues. It also
captures with a kernel that refuses the hold, which fails before the snapshot
port, with its event on the debug console.
guest::time::system::tests::FakeOpsgainsdiscipline, the discipline thatevery adjustment reports, and
adjust_error, an error for every adjustment buta query.
FakeOps::newinitializes them.FakeOps::adjust_clockreportsdisciplinein place of a default, and refusesan adjustment other than a query with
adjust_error.guest::sys::discipline::tests::adjustments_select_the_kernel_modes_and_statusasserts the modes and the status of the two new adjustments,
SetFrequencyandPin.Python, in
scripts/test_microvm_tests.py:MicrovmTestsgains_YEAR_SHORT_NS, how far a fixed guest's monotonic clocksstand short of the downtime, and a held correction of +55.3 ppm in place of
−44.5 ppm, since the guest holds the declared rate through the capture.
MicrovmTests._year_reportbuilds the report of a fixed guest, whose monotonicand boot-time clocks advance by the downtime less that shortfall, in place of
the downtime at the held correction.
test_restore_downtime_shares_one_window_and_waits_for_the_restore_markerasserts that each capture boots with the
host-rate-offset-ppmhook.test_simulated_year_check_accepts_clocks_at_their_ratesexpects the newsummary, which reports the correction held before the capture, and the
monotonic and boot-time clocks at the downtime.
test_simulated_year_check_names_every_failure: its monotonic case nowmodels an unfixed guest, whose kernel counted the downtime at the held
correction, which the check must refuse; it modeled the downtime itself.
test_simulated_year_capture_records_the_correction_after_the_next_pollisrenamed
test_simulated_year_capture_leaves_a_slew_in_flight_after_it_records:the preparation records the clocks at once, without waiting for a poll, and
then leaves a slew in flight. A new test bounds the check's tolerance on both
clocks,
test_simulated_year_check_bounds_the_monotonic_clocks_by_the_multiplier.End to end:
restore-year-capture.shrecords without waiting for the daemon's next polland ends with an
adjtimeslew of 5 ms and a wait of 1.2 s.RESTORE_YEAR_SLEW_PPMbecomesRESTORE_YEAR_HOST_PPM, the hook's 100 ppm,and
RESTORE_YEAR_MONOTONIC_PPB, the monotonic clocks' tolerance, is new.RESTORE_YEAR_PREPARATION_SECONDSgoes from 90 to 15, since the preparation nolonger waits for a poll.
run_restore_downtimeboots each capture with the hook.check_simulated_yearexpects the monotonic and boot-time clocks to advance bythe simulated downtime, within 1 s plus 1 ppm, in place of by the downtime at
the held correction, within 1 s plus 5.5 ppm for the multiplier and a slew.
OpenVMM pin and landing order
The
openvmmgitlink does not move: it stays atefe6bc26734f, so this pullrequest has no OpenVMM pull request, roadmap mapping, or landing order.