Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions doc/design/3.2-clock-and-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,15 +241,34 @@ so the saved TSC dates the saved count. The restore runs no timer until it
advances the count, by `ticks` and by the ticks for which the TSC has run from
the target that the restore wrote to it.

A one-shot timer that expired before the capture keeps its counts. KVM's saved
LAPIC state does not show whether KVM had delivered the timer's interrupt, so
on KVM the restored timer fires once more: the interrupt is due if KVM had not
delivered it, and spurious if the guest had already taken it. KVM fires it at
once from Linux 6.3, and a full timer period later before. On KVM, an expired
timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector
that is pending in the interrupt request register (IRR), which already holds
the interrupt, as after a downtime that expires the timer, reads an initial
count of zero after the restore, so that KVM does not fire it again.
A one-shot timer that expired before the capture keeps its counts, except
that on KVM, capture saves one whose interrupt the processor took with an
initial count of zero, which the restored guest reads instead of the count
that it wrote; Linux never reads it. KVM's LAPIC state shows such a timer like
one whose interrupt is still due, since KVM holds the expiry of a timer that
expires while its processor is outside the guest until the processor next
runs, and KVM's LAPIC state write fires a timer whose current count is zero
again, at once from Linux 6.3 and a full timer period later before. So when
the KVM backend stops a processor, it reads the timer's count and then has KVM
deliver the interrupts that it holds, without running the processor. KVM
records an expiry only once its host timer fires, which a loaded host can
delay, so if the timer had already expired at that read, the backend has KVM
deliver again until the interrupt appears, for up to 20 ms. Of the expired
timers whose unmasked LVT entry delivers a fixed interrupt to a valid vector,
capture then saves one that expired after the read with its vector pending in
the interrupt request register (IRR), and takes one whose vector is still
clear then for one whose interrupt the processor took. On KVM, such a timer
whose vector is pending in IRR, which already holds the interrupt, as after a
downtime that expires the timer, reads an initial count of zero after the
restore, so that KVM does not fire it again.

Two limits remain on KVM. If KVM records an expiry more than 20 ms late,
capture saves its interrupt as taken, and the restore does not deliver it. A
KVM that runs the timer on the VMX preemption timer, as on bare-metal Intel
hosts, records no expiry while the processor is outside guest mode, so on such
a host capture saves a timer that had expired when its processor stopped as
KVM reports it, and the restored timer fires once more: the interrupt is due
if KVM had not delivered it, and spurious if the guest had already taken it.

## 3.2.6 Backend Obligations

Expand Down
39 changes: 31 additions & 8 deletions doc/openvmm-upstream-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@
summary, and commit mapping below cover positions 001-131, the PR09 fixups at positions
241-243, the PR13 fixup at position 247, the PR18 fixup at position 248, the PR20 fixup at
position 249, and the PR19 fixup at position 250.
- NVX's pinned OpenVMM revision: `024ecee62d7c9b70456b63a5b4f94a6488f835cd`, position 258, the
head of microsoft/openvmm#5053, whose three commits follow position 255, `90f87319d`, the head
of `feature/microvm/dev`, which microsoft/openvmm#5052 merged with position 254 after position
- NVX's pinned OpenVMM revision: `c72fc56a3716fafaf1a9b5473c2de646c2a74815`, position 261, the
head of microsoft/openvmm#5055, whose three commits follow position 258, `024ecee62`, the head of
`feature/microvm/dev`, which microsoft/openvmm#5053 merged with positions 256 and 257 after
position 255, `90f87319d`, which microsoft/openvmm#5052 merged with position 254 after position
253, `3cd9f149b`, which microsoft/openvmm#5046 merged after position 252, `9f10ce7a8`,
which microsoft/openvmm#5044 merged after position 251, `174da3a0d`,
which microsoft/openvmm#5042 merged after position 250, `23136b647`,
Expand Down Expand Up @@ -45,8 +46,8 @@
position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits
of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1
positions 149-220 are positions 148-219, and the tree of position 220 is the tree of
`3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 127 commits past
the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-258 are not
`3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 130 commits past
the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-261 are not
yet assigned to PRs, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248
is a PR18 fixup, position 249 is a PR20 fixup, and position 250 is a PR19 fixup; see
[Commits after the mapped tip](#commits-after-the-mapped-tip).
Expand Down Expand Up @@ -1029,7 +1030,8 @@ ba105dd7f
-> 365-day maximum snapshot downtime [253]
-> restored VM time run from the TSCs' restart [254-255]
-> LAPIC timer restores without drift or a downtime refire [256-258]
-> 024ecee62 (NVX pin)
-> LAPIC timer captures that keep a taken timer from refiring [259-261]
-> c72fc56a3 (NVX pin)
```

For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main,
Expand Down Expand Up @@ -1066,15 +1068,15 @@ helper that the case uses.

## Commits after the mapped tip

The NVX pin extends the chain past position 131 without merges. Of the 127 later commits,
The NVX pin extends the chain past position 131 without merges. Of the 130 later commits,
positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup,
position 249 is a PR20 fixup, position 250 is a PR19 fixup, and the others are not yet classified
or assigned to PRs.
Each of positions 132-186 combines the v2.4 commits that the table lists, as v3.0 positions 145-160
and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across
commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart
of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0
reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-258 followed the
reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-261 followed the
v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions
222 and 223. Each commit has the position that it
had in v5.1 through position 147, and one less from v5.1 position 149 on; see
Expand Down Expand Up @@ -1210,6 +1212,9 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4
| 256 | `2a0401c43` | - | - | - | - | virt_kvm: share the guest RAM of the real-mode KVM tests (microsoft/openvmm#5053) |
| 257 | `21a76f878` | - | - | - | - | virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending (microsoft/openvmm#5053) |
| 258 | `024ecee62` | - | - | - | - | vmm_core, virt: run a restored LAPIC timer from the time ABI anchors (microsoft/openvmm#5053) |
| 259 | `727cad63f` | - | - | - | - | kvm: deliver the interrupts that KVM holds without running the VP (microsoft/openvmm#5055) |
| 260 | `9f14b54fd` | - | - | - | - | virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again (microsoft/openvmm#5055) |
| 261 | `c72fc56a3` | - | - | - | - | virt_kvm: build the LAPIC timer KVM tests for musl (microsoft/openvmm#5055); fold into 260 |

The remarks that say whether commits apply to upstream main record what the v3.0 reorganization
found on its base, `b2676222b`, at the positions that the commits have now.
Expand Down Expand Up @@ -1847,6 +1852,24 @@ files, seven of which upstream main lacks, those that PR10's positions 024 and 0
position 036, and positions 132, 135, 139, and 256 add, so it follows the time ABI series
upstream.

Positions 259-261 have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took
apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped PR
adds. Position 259 adds a call to the `kvm` crate's VP runner that has KVM deliver the interrupts
that it holds without entering the guest, in a file that upstream main has and that PR10's
position 026 and positions 133, 138, 147, and 148 also change, none of them in the runner's
`KVM_RUN` and exit decoding, which it changes and upstream main has. Replaying it on upstream main
applies without conflicts, and the `kvm` crate type-checks there, so it can go upstream on its own.

Position 260 adds a LAPIC timer module to `virt_kvm`'s x86-64 backend, which the VP run loop, the
partition, and the VP state's `apic` and `set_apic` use, in files that upstream main has and that
PR10's positions 023 and 026, PR17's position 077, and positions 133, 138, 144, 147, 149, 150,
255, 256, 257, and 258 also change, and rewrites the paragraph of the Guide's timekeeping reference
that position 257 adds. Replaying it on upstream main conflicts in that paragraph, in the module
list of the x86-64 module, and in the VP state module, whose KVM LAPIC page from position 257 its
KVM tests use, with position 256's guest RAM and position 259's delivery, so it follows positions
256, 257, and 259 upstream. Position 261 changes only position 260's KVM tests, which it builds for
musl, for which the `libc` crate defines no `SIGEV_THREAD_ID`, so it folds into position 260.

## PR summary

Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0
Expand Down
Loading