Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/nvme-iommu-fault-triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,36 @@ is not.**
(1,947 in 20 s, against 18 in 30 s for the working exact filter). **Read the filter file back after writing it**, and
keep in mind that an exact-iova filter cannot see a *larger* mapping that covers the page -- for this driver that is
tolerable because its unmaps are all 4 KiB (60 of 60 measured), but it is a measured caveat, not a proof.
4. **An event whose filter file is damaged can never be deleted, and the damage can come from a write you were told
failed.** Measured 08:15Z: an event created with a five-term filter (`iova==A || … || iova==E`) recorded and
reported correctly. Writing the filter a **second time** (`echo > filter` to clear it, then `printf '1'`) returned
`I/O error` and left the filter file holding its own error text:

```
$ cat .../events/kprobes/b2map/filter
1
^
parse_error: Field not found
```

From then on **every** write to that event -- clearing, enabling, and `echo '-:kprobes/b2map'` to delete it --
returns `I/O error`, so the probe is **permanently armed and disabled**: it records nothing (`enable=0`), it holds a
kprobe slot on `iommu_map`/`iommu_unmap_fast` until the next reboot, and there is no userspace way to remove it. A
fresh event created and deleted in the same minute works fine, so the subsystem is not wedged -- only that event is.

5. **Both the rate and the target go stale within minutes, so re-derive them immediately before arming.** Measured
on this box: an unfiltered `iommu_map`+`iommu_unmap_fast` pair ran at **~1,000 events/s** at 06:42Z and at
**14,449/s** (650,210 in 45 s) at 08:10Z, because another lane started a DMA-heavy build -- a 30-minute unfiltered
capture therefore keeps only its **last ~20 s** in the 1,410 KB per-CPU ring, which cannot contain a fault at one
per three minutes. And a filter built from the **last 25 faults** recorded 8 events in 30 s, then **0 in 30 s** five
minutes later: the faulting IOVAs move on faster than a capture can be planned. Consequence for a capture owner:
**refresh the address list and size the window from a positive control taken seconds, not minutes, earlier** -- and
let the positive control abort you, which is what it did three times today.

**Treat each event's `filter` as write-once.** To change a filter, delete the event and recreate it; do not clear or
edit one in place. And after deleting, **check `grep -c 'kprobes/' kprobe_events` back to its previous count** --
cleanup that fails silently is how a probe outlives the run that armed it, which is the rule this page already has
one paragraph up.

**Arm the map side too, unfiltered, and the argument for one address stops being needed (measured 11-16 06:42Z).**
A filtered capture on five addresses chosen from the fault census recorded **zero events in 30 s** and aborted on its
Expand Down
Loading