Repository navigation
Description
Summary
The time-host-correction microVM scenario, which #1018 added, has failed twice in CI on Linux: on azure-kvm-1 in the merge group of #1189, which took #1189 out of the merge queue, and on azure-mshv-1 in a pull-request run of #1199. In both, the guest and OpenVMM did what time ABI 2 specifies. The host's own clock correction moved: chronyd slewed host UTC by about 120 ppm for a few seconds, while two of the scenario's checks assume that the host's correction moves by less than 75 ppm in that time.
- KVM, restore phase. A chronyd slew was under way when the guest repaired its clock after the restore, so the guest took the host's correction of that moment, −121.3 ppm, which OpenVMM published with the +151 ppm hook as +29.7 ppm. The slew had ended by the first sample, 2.1 s later, which published +147.7 ppm. The held-correction check, which allows 75 ppm between the two, reported that the guest "kept a stale correction".
- MSHV, cold-boot phase. chronyd updated the host clock between the measurement window's two samples, and host UTC ran 117.5 ppm slower than the correction that both samples published. The host-rate check allows 78.7 ppm.
The scenario retries a measurement window only when the guest's discipline, host UTC's disruption counter, or a sample's uncertainty shows a problem. A time service's update or slew shows in none of them, so neither failure was retried. This is a test defect: the checks' tolerances depend on a host time service that the test doesn't control.
Incidents
1. KVM, merge group of #1189 (2026-11-03)
- Workflow: CI run 41552686892, attempt 1,
merge_group. - Head:
23472fc259ebf4ff7bdf08af9ab8b306f85a5749, openvmm: drop access-platform and the console size in microVM ABI 3 #1189's674df2a87ondev's1a183b74f. OpenVMMb9b1f8f6d41bcb7cc9b2d1ec7c836b30f5dc7718. - Failed job:
NVX microVM tests / Linux / KVM / Shard 1 of 2onazure-kvm-1, stepRun NVX microVM tests on Linux. - Downstream:
Required status checkfailed withnvx-microvm-tests-kvm: expected success, got failure, and the queue removed openvmm: drop access-platform and the console size in microVM ABI 3 #1189 at 00:51Z. openvmm: drop access-platform and the console size in microVM ABI 3 #1189's next group, run 41556578835, passed the scenario on KVM (azure-kvm-1again), MSHV, and WHP. - openvmm: drop access-platform and the console size in microVM ABI 3 #1189 doesn't change the scenario or the time code. Its OpenVMM pin change leaves
vmm_core/virt/src/time_abi/host.rsunchanged, and its own pull-request run 41544610173 passed the same shard.
host clock correction probe: the host's own correction is -0.569 ppm
host clock correction cold boot (hook +201 ppm): host_correction_ppb=200431 frequency_ppb=200431 guest_ppm=+200.455 host_ppm=+200.069 published_ppm=+200.004 in 5.0 s
error: host clock correction restore (hook +151 ppm): the guest holds the host's correction of 29715 ppb, but its first stable sample publishes +147.729 ppm, more than 75 ppm away: its boot step or restore repair kept a stale correction
2. MSHV, pull-request run of #1199 (2026-11-02)
- Workflow: CI run 41414381235, attempt 1,
pull_request, heade864c4df9c4147f39cbb89b4e0dae39402e095a3, OpenVMM6ac7fa50bc209b07a4c46d12169e28ce458d11ce. The run endedcancelledafter the job failed. - Failed job:
NVX microVM tests / Linux / MSHV / Shard 1 of 2onazure-mshv-1. - openvmm: allow nested virtualization in KVM and WHP microVMs #1199's description notes this failure and that the scenario passed in the next run, and 6 of 6 times on another MSHV host. Its head has the same scenario code and the same OpenVMM
host.rsasdev's1a183b74f.
host clock correction probe: the host's own correction is -43.686 ppm
error: host clock correction cold boot (hook +244 ppm): host UTC ran at +82.554 ppm against the guest TSC in 5.0 s, not at its published correction of +200.097 ppm, within 78.678 ppm
Evidence
The guest reports in the jobs' artifacts, nvx-microvm-tests-kvm-shard-1 and nvx-microvm-tests-mshv-shard-1 (microvm-*/time-host-correction-*.log), give the correction that each sample published and the host's maximum error, max_error_us, which Linux grows by 500 µs a second and chronyd resets at each update. Dividing out the hook gives the host's own correction, as time_host_correction_hooks does: (1 + published) / (1 + hook) − 1.
KVM restore (hook +151 ppm)
| Point | Time (UTC) | Published | Host's own | max_error_us |
|---|---|---|---|---|
| Restore repair, held by the guest | 00:35:48.3 | +29.715 ppm | −121.267 ppm | 152 |
| Sample 1 | 00:35:50.4 | +147.729 ppm | −3.271 ppm | 1152 |
| Sample 2 | 00:35:55.4 | +147.729 ppm | −3.271 ppm | 3652 |
Earlier in the same job, the host's own correction was −0.569 ppm in the probe and −0.996 ppm in the cold boot. A maximum error of 152 µs at the repair shows that chronyd had just updated the clock. Its growth by 1000 µs in the next 2 s and by 2500 µs in the 5 s after shows that chronyd made no further update. In between, the host's correction returned from −121.3 to −3.3 ppm as the update's slew ended. The guest held exactly the correction that OpenVMM published at its repair, which is fresh, not stale.
MSHV cold boot (hook +244 ppm)
| Point | Published | Host's own | max_error_us |
|---|---|---|---|
| Boot step, held by the guest | +200.304 ppm | −43.685 ppm | 2083 |
| Sample 1, 08:49:37 UTC | +200.304 ppm | −43.686 ppm | 3583 |
| Sample 2, 5.0 s later | +199.891 ppm | −44.099 ppm | 2699 |
| Host UTC between the samples | +82.554 ppm | −161.407 ppm |
Without an update, sample 2's maximum error would have been about 6083 µs, so chronyd updated the clock between the samples. Both samples published a host correction of about −44 ppm, but host UTC averaged about −161 ppm between them and fell 0.59 ms behind in 5.0 s. A slew that starts and ends between two samples appears in neither. azure-mshv-1 is the runner whose host clock wanders (#768, #1008).
Root cause
Links point to dev's 1a183b74f and to OpenVMM's d891dfd9d, whose host.rs both runs share.
- On Linux, OpenVMM reads the host's correction from
adjtimexfor every sample:freqplus theticklength (host.rs#L285-L312). chronyd slews host UTC through those, so a sample publishes the slew in progress when it is taken, and nothing of a slew that starts and ends between samples. - The guest's boot step or restore repair holds the correction of the sample it takes (state.rs#L258-L272).
- The held-correction check (microvm_tests.py#L5546-L5557) requires that correction within
TIME_HOST_CORRECTION_HELD_PPM, 75 ppm (#L520-L524), of the first stable sample's, which comes seconds later. time ABI: publish the host's clock correction with each time sample #1018 set it to half of the smallest hook, 150 ppm, after the held correction stayed within 2.7 ppm of the first sample in 67 phase logs from the Azure test VMs (review reply). A slew of more than 75 ppm at the hold that ends before the first sample fails the check (incident 1). - The host-rate check (#L5645-L5657) requires host UTC to run within
TIME_HOST_CORRECTION_HOST_PPM, 75 ppm, of the two samples' average published correction, plus half their difference and their pairing bounds. Its comment (#L531-L538) assumes slews of up to 25 ppm between samples. A larger slew between them fails the check (incident 2). - Window selection (#L5374-L5386) retries a window, up to
TIME_HOST_CORRECTION_ATTEMPTS(3), only when the guest's discipline, host UTC's disruption counter, or a sample's uncertainty shows a problem. A time-service update or slew shows in none of them.
History
- As of 2026-11-03 01:30Z, 126 CI runs had been created since time ABI: publish the host's clock correction with each time sample #1018 merged at 2026-10-31 06:27Z. Across all their attempts, 459
NVX microVM testsshard jobs passed and 2 failed: these two. The scenario runs in shard 1 on KVM and MSHV and in shard 2 on WHP. - time ABI: publish the host's clock correction with each time sample #1018's pre-merge run 40953869905 failed the scenario's host-rate check on WHP. That failure was different: OpenVMM's pin at the time,
a417f946a, inverted the sign of the Windows time adjustment. The published host correction was about +48.5 ppm while host UTC ran at −48.6 ppm, and the fix landed before time ABI: publish the host's clock correction with each time sample #1018 merged.
Possible fixes
Any fix must still fail a guest that keeps a stale correction, and a hook that skews host UTC or its published correction alone.
- Treat a time-service update in a window like a disruption. On Linux, a sample whose
max_error_usis less than the previous sample's plus 500 µs per elapsed second shows that the time service updated the clock between them. Discarding such a window would have retried incident 2. It can't help incident 1, whose slew ended without an update, or WHP, whose host reports no maximum error. - Check the held correction's provenance rather than its value. If the guest's state also identified the sample its correction came from (for example, by its
utc_ns), the test could require that sample to be the boot step's or the repair's, rather than require its correction to be near a later sample's. That check would need no host tolerance. It would address incident 1. - Retry the phase. A guest that keeps a stale correction fails every attempt, while a slew at the moment of a repair rarely recurs. This covers both incidents, at the cost of another boot or restore per retry, and it could hide an intermittent product defect.
- Larger hooks. Published corrections of 400 and 300 ppm, still below Linux's 500 ppm limit, would allow 150 ppm for both checks. That only moves the threshold.
Done when the scenario tolerates host time-service slews like these two, still fails a stale correction and a one-sided hook, and has regression tests built from both incidents' guest reports.
Related
- time ABI: publish the host's clock correction with each time sample instead of having guests estimate it #1009 and time ABI: publish the host's clock correction with each time sample #1018: time ABI 2 publishes the host's clock correction with each sample, and time ABI: publish the host's clock correction with each time sample #1018 added this scenario. [DO NOT REVIEW] microVM: publish the host's clock correction with each time sample openvmm#4830 is the OpenVMM side.
- ci: track azure-mshv-1's intermittent H4 failures after a live migration to a host with an unstable clock #768 and doctor: H4 fails hosts whose reference clock wanders, because it gates their current clock correction at 100 ppm #1008:
azure-mshv-1's wandering host clock faileddoctor's H4 check, which ci: gate H4 on chronyd's frequency estimate and end leaked VMs #779 and doctor: gate H4 on the guest's 250 ppm budget for the host's clock correction #1027 fixed.
Activity
test-microvm: retry a window that the host's time service updated
test-microvm: check where the held host correction came from
test-microvm: retry a window that the host's time service slewed
test-microvm: check where the held host correction came from
test-microvm: retry a window that the host's time service slewed
test-microvm: check where the held host correction came from
guest: record whether host test hooks altered the held time sample