diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 38e201b1b..5548546cf 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -176,6 +176,36 @@ is not.** (1,947 in 20 s, against 18 in 30 s for the working exact filter). **Read the filter file back after writing it**, and keep in mind that an exact-iova filter cannot see a *larger* mapping that covers the page -- for this driver that is tolerable because its unmaps are all 4 KiB (60 of 60 measured), but it is a measured caveat, not a proof. +4. **An event whose filter file is damaged can never be deleted, and the damage can come from a write you were told + failed.** Measured 08:15Z: an event created with a five-term filter (`iova==A || … || iova==E`) recorded and + reported correctly. Writing the filter a **second time** (`echo > filter` to clear it, then `printf '1'`) returned + `I/O error` and left the filter file holding its own error text: + + ``` + $ cat .../events/kprobes/b2map/filter + 1 + ^ + parse_error: Field not found + ``` + + From then on **every** write to that event -- clearing, enabling, and `echo '-:kprobes/b2map'` to delete it -- + returns `I/O error`, so the probe is **permanently armed and disabled**: it records nothing (`enable=0`), it holds a + kprobe slot on `iommu_map`/`iommu_unmap_fast` until the next reboot, and there is no userspace way to remove it. A + fresh event created and deleted in the same minute works fine, so the subsystem is not wedged -- only that event is. + +5. **Both the rate and the target go stale within minutes, so re-derive them immediately before arming.** Measured + on this box: an unfiltered `iommu_map`+`iommu_unmap_fast` pair ran at **~1,000 events/s** at 06:42Z and at + **14,449/s** (650,210 in 45 s) at 08:10Z, because another lane started a DMA-heavy build -- a 30-minute unfiltered + capture therefore keeps only its **last ~20 s** in the 1,410 KB per-CPU ring, which cannot contain a fault at one + per three minutes. And a filter built from the **last 25 faults** recorded 8 events in 30 s, then **0 in 30 s** five + minutes later: the faulting IOVAs move on faster than a capture can be planned. Consequence for a capture owner: + **refresh the address list and size the window from a positive control taken seconds, not minutes, earlier** -- and + let the positive control abort you, which is what it did three times today. + + **Treat each event's `filter` as write-once.** To change a filter, delete the event and recreate it; do not clear or + edit one in place. And after deleting, **check `grep -c 'kprobes/' kprobe_events` back to its previous count** -- + cleanup that fails silently is how a probe outlives the run that armed it, which is the rule this page already has + one paragraph up. **Arm the map side too, unfiltered, and the argument for one address stops being needed (measured 11-16 06:42Z).** A filtered capture on five addresses chosen from the fault census recorded **zero events in 30 s** and aborted on its