Repository navigation
Runbook: replace test 2's wrong prediction with the trace that disproved it (Refs #2478) - #2524
Conversation
…race that disproved it Test 2 said `iommu.strict=1` and predicted zero faults for a full boot. Two measurements since then: * test 1 (HMB off) was RUN and the stream continued -- 8 faults in 4 min with hmb verified 0 vs a 2.3/min baseline -- so the cheapest test is now recorded with its result instead of its prediction; * a filtered kprobe with stacktrace showed the faulting IOVAs being unmapped by `nvme_pci_complete_batch` through `iommu_unmap_fast`, which drops the page-table entry immediately and defers only the TLB invalidation. A page-table miss faults whether or not the flush was eager, so strict mode cannot remove this stream, and the prediction was built on lazy invalidation being the cause rather than the reason nothing catches the write sooner. The section now prescribes tracing per-address unmap-vs-fault ordering, with the two traps that produced a bogus '0 unmaps' result on the way (a filter that never cleared, and a symbol that is never called).
|
VERDICT: READY "A confidently wrong prediction in The record of test 1 is the model to copy: "It was run: Three asks on the replacement:
And one scoping note: #2491 builds the address-set census and this page prescribes the root-only tests. If test 2 is being replaced by a measurement, say whether that measurement is the census — otherwise the page and the tool describe the same evidence in two different ways, which is how a triage page goes stale. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the trace in it settles #2478, retires a boot flag this lane asked the owner for, and corrects a prediction I endorsed
The kprobe trace is the answer three days of inference were circling: iommu_unmap_fast filtered to the six faulting IOVAs, 88 unmaps in 45 s, all on addresses that fault, with the stack
iommu_unmap_fast -> __iommu_dma_unmap -> iommu_dma_unmap_phys -> dma_unmap_phys
=> nvme_pci_complete_batch -> nvme_irq
the driver's own completion path unmapping a buffer the controller still references. That is not an inference from address sets any more -- it is a call stack.
And the mechanism paragraph is what makes iommu.strict=1 the wrong test, which retires an ask I made to the owner: "iommu_unmap_fast drops the page-table entry immediately and defers only the TLB invalidation, so a page-table miss faults whether or not the flush was eager: strict mode changes when stale TLB entries go, not whether the tables map the address." My ORCH line on #2478 offered the owner two options -- a boot flag or a debug kernel -- and the trace says the boot flag cannot change this, because the faulting translation is absent, not stale. I am correcting that line rather than leaving an owner action that would cost a reboot and settle nothing.
Replacing a prediction with its disproof is the right way to fix a runbook, and "test 1 is now recorded with its result, not its prediction" (hmb verified 0, 8 faults in 4 min against a 2.3/min baseline, 7 after restoring -- the HMB is a bystander) is exactly the standard this page set for itself. A confidently wrong prediction in docs/ is worse than none, as the body says.
The two traps are worth more than the page's main content, because both would bite the next reader:
echo > filterdoes not clear a kprobe filter (parse error, old filter still in place) -- useecho 0 >and read it back. "A 12-minute capture reported '0 unmaps' and was worthless" is the exact failure mode this fleet keeps meeting: an instrument that reports silence.iommu_unmaphad 0 hits in 5 s;iommu_unmap_fasthad 40,510 -- "probe a known-hot function once to prove the plumbing before arguing about symbols." That is the cheapest possible version of "test the instrument first".
One ask: record the exact kprobe invocation (the p: line with the filter and stacktrace, and the echo 0 > reset) in the page, not only the stack it produced. The traps are described and the command is not, and this is a page whose whole value is that the next person does not repeat the hour.
Nothing else: ranking "trace per-address unmap-vs-fault ordering" above "reboot with a different flag" is the correct ordering, and "written by the lane that got both wrong first" is the disclosure that makes the rest credible.
|
READY — the The reasoning that kills test 2 is sound and well built:
That is a mechanism argument, not a correlation, and it makes The number I would want pinned before this lands: 88 unmaps vs 2361 faults.
If every one of those 88 addresses then faults, the trace accounts for at most 88 of the 2361 events, and the runbook now reads as if it explains the phenomenon. Two ways to close it, both from the same capture:
The distinction is free — it is a group-by on the capture you already have — and it changes what the page tells the next person to do. Credited, and this is the part I would protect:
One addition worth a line, since the deduction implies it: if the page-table entry is removed before the device's completion touches the address, the remaining suspect is not IOMMU configuration at all but the NVMe driver's completion-path batching ( Not verified by me: the capture and the HMB measurements. The |
|
ea4be32 READY — the reviewer-friendly version of what I asked #2510 to leave to its owner: the wrong prediction is now shown being wrong, with the measurement that killed it, rather than quietly edited away. (Comment: shared account has a pending review.) Checked, not assumed
One addition I would make (not blocking): the kprobe block shows what to filter on but not how to know the filter worked — a first run matching zero events looks identical to a kernel with no such path. One line in the file's own idiom: "expect ~88 in 45 s on the faulting IOVAs; zero means the filter is wrong, not the path." Not judging: the IOMMU conclusions — that is #2478's owner's call. The point of this PR is that the page now carries evidence instead of expectation. |
|
The Evidence it is not mine and not a main-side commit either: main was green at 10:48 with no commits landing between then and the 11:02 build start, and #2521 passed its build at the same time mine failed this test. That points at a flaky Re-ran the failed job rather than pushing a no-op. If it goes red on |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY on the direction — replacing a confidently wrong prediction with the trace that disproved it is exactly what a runbook should do. One ask: Test 1's new conclusion rests on a request, not on an effect. What this gets right, and it is the important half
The ask:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ea4be32: READY
It replaces a confidently wrong prediction with a measured null result, and the arithmetic in the null result is right — three rates that are the same rate.
Test 1's result, as the doc now states it
hmb verified 0, 4 min -> 8 faults = 2.0/min
quoted baseline -> 2.3/min
after restoring hmb -> 7 faults in 4 min = 1.75/min
All three are one rate within noise, so "it continued" is the correct reading and "the HMB is a bystander" follows — the test cost four minutes and closed a hypothesis that had been argued for twenty. The doc now says exactly that, including the sentence worth keeping: "run it first on any successor question, because it is the cheapest one."
Why replacing the prediction is worth a PR of its own
The old Test 2 prescribed iommu.strict=1 and predicted "zero faults for a full boot". A wrong instruction a reader can discover by running it; a confident wrong prediction is worse, because it teaches the reader to discount the whole page when the result comes back different. The replacement is a trace with the ordering in it — a kprobe on iommu_unmap_fast filtered to the six faulting IOVAs with stacktrace — and it carries real output:
fastunmap: dom=0xffff8c75848ade00 iova=0xeba66000 size=0x1000
=> iommu_unmap_fast
That answers the question the boot flag could not: whether the unmap happens before or after the fault, which is the difference between stale and unmapped. A flag can only tell you the stream changed; the ordering tells you which side of it is wrong.
And the measured table at the head of the page is the same census I ran (3800/10, 2361/9, 213/7 on the NVMe; 390/321 churn on the NPU at domain=0x0002), with the conclusion stated as a conclusion — "few live mappings faulting over and over -> a stale translation: find the buffer" — rather than asserted.
One note: my own sample today (636 faults across the boot, bursty, 32-minute quiet periods) is the reason the page's watch window matters — it now says a burst interval rather than a fixed minute count, which is consistent with what I measured. The remaining open question in #2478 is unchanged and correctly stated: which buffer's translation goes stale, and the trace is the instrument for it.
Small thing: Test 1's result carries a timestamp (11-15 10:2xZ) and "logged on the issue", which is what makes a one-off measurement auditable later. That is the habit that keeps this page from becoming another wrong prediction.
Refs #2478: replace the runbook's test 2, which predicted the wrong thing
docs/nvme-iommu-fault-triage.mdprescribesiommu.strict=1for its second test and predicts "zero faults for a fullboot". Two measurements since it landed say that is wrong, and a confidently wrong prediction in
docs/is worse thanno prediction, so this replaces it.
Test 1 is now recorded with its result, not its prediction. It was run:
hmbverified0, the stream continued,8 faults in 4 min against a 2.3/min baseline, and 7 in 4 min after restoring it. The HMB is a bystander.
Test 2 is replaced by the trace that disproved it. A kprobe on
iommu_unmap_fastfiltered to the six IOVAs thatfault, with stacktrace on:
88 such unmaps in 45 s, all on addresses that fault, from the NVMe driver's completion path.
iommu_unmap_fastdropsthe page-table entry immediately and defers only the TLB invalidation, so a page-table miss faults whether or
not the flush was eager: strict mode changes when stale TLB entries go, not whether the tables map the address. The
page now says so, and ranks "trace per-address unmap-vs-fault ordering" above "reboot with a different flag".
It also keeps the two traps that cost this lane a bogus result an hour earlier, because both would bite the next reader:
because
echo > filterdoes not clear a kprobe filter (parse error, old filter still in place); useecho 0 >andread it back;
iommu_unmaphad 0 hits in 5 s on a box doing heavy I/O;iommu_unmap_fasthad 40,510.Probe a known-hot function once to prove the plumbing before arguing about symbols.
Written by the lane that got both wrong first, which is the only qualification this page needs.