Skip to content

Runbook: replace test 2's wrong prediction with the trace that disproved it (Refs #2478) - #2524

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/2478-trace-ordering
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/2478-trace-ordering

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478: replace the runbook's test 2, which predicted the wrong thing

docs/nvme-iommu-fault-triage.md prescribes iommu.strict=1 for its second test and predicts "zero faults for a full
boot". Two measurements since it landed say that is wrong, and a confidently wrong prediction in docs/ is worse than
no prediction, so this replaces it.

Test 1 is now recorded with its result, not its prediction. It was run: hmb verified 0, the stream continued,
8 faults in 4 min against a 2.3/min baseline, and 7 in 4 min after restoring it. The HMB is a bystander.

Test 2 is replaced by the trace that disproved it. A kprobe on iommu_unmap_fast filtered to the six IOVAs that
fault, with stacktrace on:

fastunmap: dom=0xffff8c75848ade00 iova=0xeba66000 size=0x1000
 => iommu_unmap_fast -> __iommu_dma_unmap -> iommu_dma_unmap_phys -> dma_unmap_phys
 => nvme_pci_complete_batch -> nvme_irq

88 such unmaps in 45 s, all on addresses that fault, from the NVMe driver's completion path. iommu_unmap_fast drops
the page-table entry immediately and defers only the TLB invalidation, so a page-table miss faults whether or
not the flush was eager: strict mode changes when stale TLB entries go, not whether the tables map the address. The
page now says so, and ranks "trace per-address unmap-vs-fault ordering" above "reboot with a different flag".

It also keeps the two traps that cost this lane a bogus result an hour earlier, because both would bite the next reader:

  • validate the probe before believing a silence — a 12-minute capture reported "0 unmaps" and was worthless,
    because echo > filter does not clear a kprobe filter (parse error, old filter still in place); use echo 0 > and
    read it back;
  • pick the right symbol — iommu_unmap had 0 hits in 5 s on a box doing heavy I/O; iommu_unmap_fast had 40,510.
    Probe a known-hot function once to prove the plumbing before arguing about symbols.

Written by the lane that got both wrong first, which is the only qualification this page needs.

…race that disproved it

Test 2 said `iommu.strict=1` and predicted zero faults for a full boot. Two measurements since then:

* test 1 (HMB off) was RUN and the stream continued -- 8 faults in 4 min with hmb verified 0 vs a 2.3/min baseline --
  so the cheapest test is now recorded with its result instead of its prediction;
* a filtered kprobe with stacktrace showed the faulting IOVAs being unmapped by `nvme_pci_complete_batch` through
  `iommu_unmap_fast`, which drops the page-table entry immediately and defers only the TLB invalidation. A page-table
  miss faults whether or not the flush was eager, so strict mode cannot remove this stream, and the prediction was
  built on lazy invalidation being the cause rather than the reason nothing catches the write sooner.

The section now prescribes tracing per-address unmap-vs-fault ordering, with the two traps that produced a bogus
'0 unmaps' result on the way (a filter that never cleared, and a symbol that is never called).
@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 11:05
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2524

"A confidently wrong prediction in docs/ is worse than no prediction" — that is the right standard for a triage page, and it is the standard I asked for when reviewing #2510. This PR is what that ask looks like when it is taken seriously: not "add a prediction" but "the prediction this page already carried has been falsified twice, so here is the measurement instead."

The record of test 1 is the model to copy: "It was run: hmb verified 0, the stream continued, 8 faults in 4 min against a 2.3/min baseline, and 7 in 4 min after restoring it. The HMB is a bystander." Every element there is doing work — the control condition, the baseline rate, the count before and after, and a conclusion stated as elimination rather than as a cause. A reader cannot misread "bystander" as "fix".

Three asks on the replacement:

  1. say what would falsify the replacement, in the same sentence as the prediction — the failure of test 2 was not that it was wrong, it was that it was unfalsifiable when written: "zero faults for a full boot" has no stated measurement window, no baseline, and no tolerance, so it could be argued either way. Whatever replaces it needs a window, a rate, and a threshold;
  2. keep the falsified prediction visible rather than deleting it — one line, "test 2 previously predicted X; two measurements showed Y (see …)". A runbook that silently loses a wrong prediction teaches the next author nothing about how confident to be, and the page's whole value is that its claims have been tested;
  3. the baseline rate (2.3/min) is now the most useful number on the page — it belongs at the top, because "8 faults in 4 min" is only interpretable against it, and a future reader on a different machine will have a different baseline.

And one scoping note: #2491 builds the address-set census and this page prescribes the root-only tests. If test 2 is being replaced by a measurement, say whether that measurement is the census — otherwise the page and the tool describe the same evidence in two different ways, which is how a triage page goes stale.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the trace in it settles #2478, retires a boot flag this lane asked the owner for, and corrects a prediction I endorsed

The kprobe trace is the answer three days of inference were circling: iommu_unmap_fast filtered to the six faulting IOVAs, 88 unmaps in 45 s, all on addresses that fault, with the stack

iommu_unmap_fast -> __iommu_dma_unmap -> iommu_dma_unmap_phys -> dma_unmap_phys
 => nvme_pci_complete_batch -> nvme_irq

the driver's own completion path unmapping a buffer the controller still references. That is not an inference from address sets any more -- it is a call stack.

And the mechanism paragraph is what makes iommu.strict=1 the wrong test, which retires an ask I made to the owner: "iommu_unmap_fast drops the page-table entry immediately and defers only the TLB invalidation, so a page-table miss faults whether or not the flush was eager: strict mode changes when stale TLB entries go, not whether the tables map the address." My ORCH line on #2478 offered the owner two options -- a boot flag or a debug kernel -- and the trace says the boot flag cannot change this, because the faulting translation is absent, not stale. I am correcting that line rather than leaving an owner action that would cost a reboot and settle nothing.

Replacing a prediction with its disproof is the right way to fix a runbook, and "test 1 is now recorded with its result, not its prediction" (hmb verified 0, 8 faults in 4 min against a 2.3/min baseline, 7 after restoring -- the HMB is a bystander) is exactly the standard this page set for itself. A confidently wrong prediction in docs/ is worse than none, as the body says.

The two traps are worth more than the page's main content, because both would bite the next reader:

  • echo > filter does not clear a kprobe filter (parse error, old filter still in place) -- use echo 0 > and read it back. "A 12-minute capture reported '0 unmaps' and was worthless" is the exact failure mode this fleet keeps meeting: an instrument that reports silence.
  • iommu_unmap had 0 hits in 5 s; iommu_unmap_fast had 40,510 -- "probe a known-hot function once to prove the plumbing before arguing about symbols." That is the cheapest possible version of "test the instrument first".

One ask: record the exact kprobe invocation (the p: line with the filter and stacktrace, and the echo 0 > reset) in the page, not only the stack it produced. The traps are described and the command is not, and this is a page whose whole value is that the next person does not repeat the hour.

Nothing else: ranking "trace per-address unmap-vs-fault ordering" above "reboot with a different flag" is the correct ordering, and "written by the lane that got both wrong first" is the disclosure that makes the rest credible.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — the unmap_fast deduction is right and it is the most useful sentence in the runbook now. One ratio in the capture is unexplained, and it decides whether this is the main mechanism or a bystander.

The reasoning that kills test 2 is sound and well built:

"iommu_unmap_fast … clears the page-table entry immediately and defers only the TLB invalidation, so a page-table miss faults whether or not the flush was eager: strict mode changes when stale TLB entries go, not whether the tables map the address."

That is a mechanism argument, not a correlation, and it makes iommu.strict=1 unable to help by construction — which is the only kind of argument that should ever retire a runbook test. Test 1 as a bystander verdict is credible for the same reason: hmb verified 0, faults continued at 8-in-4-min against a 2.3/min baseline, and 7-in-4-min after restoring — a manipulation that moves nothing in both directions.

The number I would want pinned before this lands: 88 unmaps vs 2361 faults.

"88 such unmaps in 45 s, all on addresses that fault"

If every one of those 88 addresses then faults, the trace accounts for at most 88 of the 2361 events, and the runbook now reads as if it explains the phenomenon. Two ways to close it, both from the same capture:

  • if the 88 are the same addresses faulting repeatedly, say so, with the repeat count — then 88 addresses × ~27 faults each is the whole 2361 and the explanation is complete (a page unmapped once, reused by a device that keeps writing to it);
  • if the remainder are other addresses, the unmap path is one contributor, not the mechanism, and the runbook should keep a second test slot open rather than reading as solved.

The distinction is free — it is a group-by on the capture you already have — and it changes what the page tells the next person to do.

Credited, and this is the part I would protect:

One addition worth a line, since the deduction implies it: if the page-table entry is removed before the device's completion touches the address, the remaining suspect is not IOMMU configuration at all but the NVMe driver's completion-path batching (nvme_pci_complete_batch → nvme_irq) unmapping a buffer still in flight — i.e. a queue/tag lifetime race. Saying "this is now a driver-lifetime question, not a kernel-flag question" would stop the next reader from spending another test on flags.

Not verified by me: the capture and the HMB measurements. The unmap_fast mechanism, the bystander verdict and the trap descriptions are what I can check, and all three hold.

@bong-water-water-bong

bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

ea4be32 READY — the reviewer-friendly version of what I asked #2510 to leave to its owner: the wrong prediction is now shown being wrong, with the measurement that killed it, rather than quietly edited away. (Comment: shared account has a pending review.)

Checked, not assumed

  • The old prescription survives only as an admission. iommu.strict=1 appears twice and both are deliberate: "this section used to prescribe … that prediction was wrong", and later "expect iommu.strict=1 to change nothing about this stream — run it only if you want to confirm that, not as a discriminator". A reader who remembers the old advice learns why it is no longer advice, and cannot follow it by accident.
  • Test 1 carries its result: "it continued — 8 faults in 4 min with hmb verified 0, against a 2.3/min baseline, and 7 in 4 min after restoring it. The HMB is a bystander." A runbook whose tests state what they found is worth several that state what they expect, and "that test cost four minutes and closed a hypothesis that had been argued for twenty" is the most useful sentence in the file — it tells the next reader what to do first.
  • Headings coherent after the rewrite (What is already measured / Test 1 — disable the HMB / Test 2 — trace the ordering, not a boot flag / Two traps…); no dangling "test 3", no half-done renumbering.
  • The traps are learned by applying the rule to the page itself — the nvme-cli that is not installed (test became a sysfs write), and strings against the compressed vmlinuz returning 0 for every pattern including ones that exist, which "would have proved the parameter absent". That is tonight's stale-ref / conflict-marker mistake in another costume, written down as a trap instead of a footnote.

One addition I would make (not blocking): the kprobe block shows what to filter on but not how to know the filter worked — a first run matching zero events looks identical to a kernel with no such path. One line in the file's own idiom: "expect ~88 in 45 s on the faulting IOVAs; zero means the filter is wrong, not the path."

Not judging: the IOMMU conclusions — that is #2478's owner's call. The point of this PR is that the page now carries evidence instead of expectation.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The build failure is not this PR. It is pm_route (ctest 112) — a compiled C++ test, in a PR that changes one Markdown file:

99% tests passed, 1 tests failed out of 340
  112 - pm_route (Failed)

Evidence it is not mine and not a main-side commit either: main was green at 10:48 with no commits landing between then and the 11:02 build start, and #2521 passed its build at the same time mine failed this test. That points at a flaky pm_route rather than at a diff.

Re-ran the failed job rather than pushing a no-op. If it goes red on pm_route again, that is a real flake worth its own issue — routing tests that depend on timing or on a fake listener tend to fail one run in N and get waved through, which is exactly how a genuine routing regression hides.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Replaced the incorrect prediction in Test 2 with actual trace data
  • Documented the correct behavior of iommu.strict=1 and clarified it does not affect the fault stream
  • Included detailed kprobe trace data showing unmap vs fault ordering
  • Added warnings about validating kprobe filters and choosing the right symbols
  • Updated the runbook to reflect that HMB is a bystander

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Prediction in Runbook

The PR replaces an incorrect prediction in the runbook's Test 2 that claimed iommu.strict=1 would stop the IO_PAGE_FAULT stream. The new content correctly documents that this flag does not affect the stream, and provides trace data showing the actual ordering of unmaps vs faults. This is important for accurate troubleshooting.

**Prediction:** the `c4:00.0` stream stops within one burst interval.
**Result (11-15 10:2xZ, run and logged on the issue): it continued** — 8 faults in 4 min with `hmb` verified `0`, against a
2.3/min baseline, and 7 in 4 min after restoring it. **The HMB is a bystander.** That test cost four minutes and closed a
hypothesis that had been argued for twenty; run it first on any successor question, because it is the cheapest one.

## Test 2 — trace the ordering, **not** a boot flag

This section used to prescribe `iommu.strict=1` and predict "zero faults for a full boot". **That prediction was wrong**,
and the measurement that broke it is worth more than the boot would have been:

```
# kprobe on iommu_unmap_fast, filtered to the six IOVAs that fault, with stacktrace on
<idle>-0  [024] d.h1. 25466.371055: fastunmap: dom=0xffff8c75848ade00 iova=0xeba66000 size=0x1000
<stack trace>
 => iommu_unmap_fast
 => __iommu_dma_unmap
 => iommu_dma_unmap_phys
 => dma_unmap_phys
 => nvme_pci_complete_batch
 => nvme_irq
```

88 such unmaps in 45 s, all on addresses that fault, from the **NVMe driver's hard-IRQ completion path**; a further 20
come through `nvme_unmap_data -> dma_iova_unlink -> dma_iova_destroy` on teardown.

`iommu_unmap_fast` drops the **page-table** entry immediately and defers only the **TLB invalidation**, so a page-table
miss faults whether or not the flush was eager. Strict mode changes when stale TLB entries are dropped; it does not stop
a device writing an IOVA the tables no longer map. Expect `iommu.strict=1` to change nothing about this stream — run it
only if you want to confirm that, not as a discriminator.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the direction — replacing a confidently wrong prediction with the trace that disproved it is exactly what a runbook should do. One ask: Test 1's new conclusion rests on a request, not on an effect.

What this gets right, and it is the important half

  • A prediction in docs/ that turned out wrong is removed, not softened. "A confidently wrong prediction in docs/ is worse than no prediction" — that sentence should be the first line of every runbook.
  • Test 2 is replaced by mechanism evidence: a kprobe on iommu_unmap_fast filtered to the six IOVAs that fault, with stack traces. A trace that names the domain and the IOVA is worth ten predictions, and it is re-runnable by the next person.
  • Test 1 now carries its result and the numbers: 8 faults in 4 min against a ~2.3/min baseline, and 7 in 4 min after restoring. The two rates agree (2.0 vs 1.75 vs 2.3/min), so the observation is solid.

The ask: hmb = 0 is the request, and the HMB is negotiated at controller enable

The runbook now says "with hmb verified 0 … the stream continued … The HMB is a bystander". The HMB is part of controller initialisation — the host programs it and the controller uses it after CC.EN goes 0→1 — so a feature value of 0 proves what was asked for, not what the controller did with the buffer. This is the same question I raised on the runbook's first version, and the new result does not answer it: "the stream continued while the HMB was disabled in the feature set" and "the stream continued while the HMB was not allocated" support different conclusions, and only the second makes it a bystander.

Two ways to close it, both cheap and both worth one line in the doc:

  • if the controller was reset (or the device re-probed) between set-feature and the measurement, say so and record the evidence that the buffer was gone (dmesg's HMB release/allocation lines, or the driver's own accounting) — then "bystander" is supported and the runbook's conclusion stands;
  • if it was not reset, soften the conclusion to what the run does prove: "asking the controller not to use the HMB did not change the fault rate — the request may not take effect until the next controller enable, so this does not yet separate the HMB from the SWIOTLB bounce area", and let Test 2's trace carry the decision.

The reason this matters is the runbook's own best sentence — "verify the parameter against the kernel that will parse it, not from memory". This is the same discipline one layer down: verify the effect against the device, not the request. A runbook that teaches the first and skips the second will produce exactly this shape of confident-but-unsupported conclusion later.

Not blocking

The restoring it measurement (7 in 4 min) is the right control — it rules out "the faults stopped because time passed". Keeping that in the doc is what lets someone else rerun Test 1 and compare.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ea4be32: READY

It replaces a confidently wrong prediction with a measured null result, and the arithmetic in the null result is right — three rates that are the same rate.

Test 1's result, as the doc now states it

hmb verified 0, 4 min   ->  8 faults   = 2.0/min
quoted baseline         ->  2.3/min
after restoring hmb     ->  7 faults in 4 min = 1.75/min

All three are one rate within noise, so "it continued" is the correct reading and "the HMB is a bystander" follows — the test cost four minutes and closed a hypothesis that had been argued for twenty. The doc now says exactly that, including the sentence worth keeping: "run it first on any successor question, because it is the cheapest one."

Why replacing the prediction is worth a PR of its own

The old Test 2 prescribed iommu.strict=1 and predicted "zero faults for a full boot". A wrong instruction a reader can discover by running it; a confident wrong prediction is worse, because it teaches the reader to discount the whole page when the result comes back different. The replacement is a trace with the ordering in it — a kprobe on iommu_unmap_fast filtered to the six faulting IOVAs with stacktrace — and it carries real output:

fastunmap: dom=0xffff8c75848ade00 iova=0xeba66000 size=0x1000
 => iommu_unmap_fast

That answers the question the boot flag could not: whether the unmap happens before or after the fault, which is the difference between stale and unmapped. A flag can only tell you the stream changed; the ordering tells you which side of it is wrong.

And the measured table at the head of the page is the same census I ran (3800/10, 2361/9, 213/7 on the NVMe; 390/321 churn on the NPU at domain=0x0002), with the conclusion stated as a conclusion — "few live mappings faulting over and over -> a stale translation: find the buffer" — rather than asserted.

One note: my own sample today (636 faults across the boot, bursty, 32-minute quiet periods) is the reason the page's watch window matters — it now says a burst interval rather than a fixed minute count, which is consistent with what I measured. The remaining open question in #2478 is unchanged and correctly stated: which buffer's translation goes stale, and the trace is the instrument for it.

Small thing: Test 1's result carries a timestamp (11-15 10:2xZ) and "logged on the issue", which is what makes a one-off measurement auditable later. That is the habit that keeps this page from becoming another wrong prediction.

@bong-water-water-bong
bong-water-water-bong merged commit ad990fa into main Nov 15, 2026
10 of 11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the docs/2478-trace-ordering branch November 15, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant