Skip to content

test-microvm: time-host-correction fails when the host's time service slews host UTC by more than 75 ppm during a phase (flaky NVX microVM tests / Linux) #1229

Closed

Description

Summary

The time-host-correction microVM scenario, which #1018 added, has failed twice in CI on Linux: on azure-kvm-1 in the merge group of #1189, which took #1189 out of the merge queue, and on azure-mshv-1 in a pull-request run of #1199. In both, the guest and OpenVMM did what time ABI 2 specifies. The host's own clock correction moved: chronyd slewed host UTC by about 120 ppm for a few seconds, while two of the scenario's checks assume that the host's correction moves by less than 75 ppm in that time.

  • KVM, restore phase. A chronyd slew was under way when the guest repaired its clock after the restore, so the guest took the host's correction of that moment, −121.3 ppm, which OpenVMM published with the +151 ppm hook as +29.7 ppm. The slew had ended by the first sample, 2.1 s later, which published +147.7 ppm. The held-correction check, which allows 75 ppm between the two, reported that the guest "kept a stale correction".
  • MSHV, cold-boot phase. chronyd updated the host clock between the measurement window's two samples, and host UTC ran 117.5 ppm slower than the correction that both samples published. The host-rate check allows 78.7 ppm.

The scenario retries a measurement window only when the guest's discipline, host UTC's disruption counter, or a sample's uncertainty shows a problem. A time service's update or slew shows in none of them, so neither failure was retried. This is a test defect: the checks' tolerances depend on a host time service that the test doesn't control.

Incidents

1. KVM, merge group of #1189 (2026-11-03)

host clock correction probe: the host's own correction is -0.569 ppm
host clock correction cold boot (hook +201 ppm): host_correction_ppb=200431 frequency_ppb=200431 guest_ppm=+200.455 host_ppm=+200.069 published_ppm=+200.004 in 5.0 s
error: host clock correction restore (hook +151 ppm): the guest holds the host's correction of 29715 ppb, but its first stable sample publishes +147.729 ppm, more than 75 ppm away: its boot step or restore repair kept a stale correction

2. MSHV, pull-request run of #1199 (2026-11-02)

host clock correction probe: the host's own correction is -43.686 ppm
error: host clock correction cold boot (hook +244 ppm): host UTC ran at +82.554 ppm against the guest TSC in 5.0 s, not at its published correction of +200.097 ppm, within 78.678 ppm

Evidence

The guest reports in the jobs' artifacts, nvx-microvm-tests-kvm-shard-1 and nvx-microvm-tests-mshv-shard-1 (microvm-*/time-host-correction-*.log), give the correction that each sample published and the host's maximum error, max_error_us, which Linux grows by 500 µs a second and chronyd resets at each update. Dividing out the hook gives the host's own correction, as time_host_correction_hooks does: (1 + published) / (1 + hook) − 1.

KVM restore (hook +151 ppm)

Point Time (UTC) Published Host's own max_error_us
Restore repair, held by the guest 00:35:48.3 +29.715 ppm −121.267 ppm 152
Sample 1 00:35:50.4 +147.729 ppm −3.271 ppm 1152
Sample 2 00:35:55.4 +147.729 ppm −3.271 ppm 3652

Earlier in the same job, the host's own correction was −0.569 ppm in the probe and −0.996 ppm in the cold boot. A maximum error of 152 µs at the repair shows that chronyd had just updated the clock. Its growth by 1000 µs in the next 2 s and by 2500 µs in the 5 s after shows that chronyd made no further update. In between, the host's correction returned from −121.3 to −3.3 ppm as the update's slew ended. The guest held exactly the correction that OpenVMM published at its repair, which is fresh, not stale.

MSHV cold boot (hook +244 ppm)

Point Published Host's own max_error_us
Boot step, held by the guest +200.304 ppm −43.685 ppm 2083
Sample 1, 08:49:37 UTC +200.304 ppm −43.686 ppm 3583
Sample 2, 5.0 s later +199.891 ppm −44.099 ppm 2699
Host UTC between the samples +82.554 ppm −161.407 ppm

Without an update, sample 2's maximum error would have been about 6083 µs, so chronyd updated the clock between the samples. Both samples published a host correction of about −44 ppm, but host UTC averaged about −161 ppm between them and fell 0.59 ms behind in 5.0 s. A slew that starts and ends between two samples appears in neither. azure-mshv-1 is the runner whose host clock wanders (#768, #1008).

Root cause

Links point to dev's 1a183b74f and to OpenVMM's d891dfd9d, whose host.rs both runs share.

  1. On Linux, OpenVMM reads the host's correction from adjtimex for every sample: freq plus the tick length (host.rs#L285-L312). chronyd slews host UTC through those, so a sample publishes the slew in progress when it is taken, and nothing of a slew that starts and ends between samples.
  2. The guest's boot step or restore repair holds the correction of the sample it takes (state.rs#L258-L272).
  3. The held-correction check (microvm_tests.py#L5546-L5557) requires that correction within TIME_HOST_CORRECTION_HELD_PPM, 75 ppm (#L520-L524), of the first stable sample's, which comes seconds later. time ABI: publish the host's clock correction with each time sample #1018 set it to half of the smallest hook, 150 ppm, after the held correction stayed within 2.7 ppm of the first sample in 67 phase logs from the Azure test VMs (review reply). A slew of more than 75 ppm at the hold that ends before the first sample fails the check (incident 1).
  4. The host-rate check (#L5645-L5657) requires host UTC to run within TIME_HOST_CORRECTION_HOST_PPM, 75 ppm, of the two samples' average published correction, plus half their difference and their pairing bounds. Its comment (#L531-L538) assumes slews of up to 25 ppm between samples. A larger slew between them fails the check (incident 2).
  5. Window selection (#L5374-L5386) retries a window, up to TIME_HOST_CORRECTION_ATTEMPTS (3), only when the guest's discipline, host UTC's disruption counter, or a sample's uncertainty shows a problem. A time-service update or slew shows in none of them.

History

Possible fixes

Any fix must still fail a guest that keeps a stale correction, and a hook that skews host UTC or its published correction alone.

  1. Treat a time-service update in a window like a disruption. On Linux, a sample whose max_error_us is less than the previous sample's plus 500 µs per elapsed second shows that the time service updated the clock between them. Discarding such a window would have retried incident 2. It can't help incident 1, whose slew ended without an update, or WHP, whose host reports no maximum error.
  2. Check the held correction's provenance rather than its value. If the guest's state also identified the sample its correction came from (for example, by its utc_ns), the test could require that sample to be the boot step's or the repair's, rather than require its correction to be near a later sample's. That check would need no host tolerance. It would address incident 1.
  3. Retry the phase. A guest that keeps a stale correction fails every attempt, while a slew at the moment of a repair rarely recurs. This covers both incidents, at the cost of another boot or restore per retry, and it could hide an intermittent product defect.
  4. Larger hooks. Published corrections of 400 and 300 ppm, still below Linux's 500 ppm limit, would allow 150 ppm for both checks. That only moves the threshold.

Done when the scenario tolerates host time-service slews like these two, still fails a stale correction and a one-sided hook, and has regression tests built from both incidents' guest reports.

Related

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions