From 58eae99be422d87750c6a85500d602f14ae22f0d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Mon, 16 Nov 2026 04:12:28 -0400 Subject: [PATCH 1/2] docs: a damaged filter file makes a kprobe undeletable, and a failed write can be what damages it Measured 08:15Z: an event created with a five-term filter works, reports correctly and records; writing its filter a SECOND time (first 'echo > filter' to clear it, then printf '1') returns I/O error and leaves the filter file holding its own error text: $ cat events/kprobes/b2map/filter 1 ^ parse_error: Field not found After that every write to that event -- clearing, enabling, and the deletion line -- returns I/O error, so the probe is permanently armed AND disabled: no events recorded, a kprobe slot held on iommu_map/iommu_unmap_fast until reboot, no userspace recovery. A fresh event created and deleted in the same minute works, so only that event is wedged. Rule for the page: treat each event's filter as write-once (delete and recreate rather than editing), and check the kprobe count back to its previous value after cleanup -- silent cleanup failure is how a probe outlives its run. --- docs/nvme-iommu-fault-triage.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 38e201b1b..b76fec5e2 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -176,6 +176,27 @@ is not.** (1,947 in 20 s, against 18 in 30 s for the working exact filter). **Read the filter file back after writing it**, and keep in mind that an exact-iova filter cannot see a *larger* mapping that covers the page -- for this driver that is tolerable because its unmaps are all 4 KiB (60 of 60 measured), but it is a measured caveat, not a proof. +4. **An event whose filter file is damaged can never be deleted, and the damage can come from a write you were told + failed.** Measured 08:15Z: an event created with a five-term filter (`iova==A || … || iova==E`) recorded and + reported correctly. Writing the filter a **second time** (`echo > filter` to clear it, then `printf '1'`) returned + `I/O error` and left the filter file holding its own error text: + + ``` + $ cat .../events/kprobes/b2map/filter + 1 + ^ + parse_error: Field not found + ``` + + From then on **every** write to that event -- clearing, enabling, and `echo '-:kprobes/b2map'` to delete it -- + returns `I/O error`, so the probe is **permanently armed and disabled**: it records nothing (`enable=0`), it holds a + kprobe slot on `iommu_map`/`iommu_unmap_fast` until the next reboot, and there is no userspace way to remove it. A + fresh event created and deleted in the same minute works fine, so the subsystem is not wedged -- only that event is. + + **Treat each event's `filter` as write-once.** To change a filter, delete the event and recreate it; do not clear or + edit one in place. And after deleting, **check `grep -c 'kprobes/' kprobe_events` back to its previous count** -- + cleanup that fails silently is how a probe outlives the run that armed it, which is the rule this page already has + one paragraph up. **Arm the map side too, unfiltered, and the argument for one address stops being needed (measured 11-16 06:42Z).** A filtered capture on five addresses chosen from the fault census recorded **zero events in 30 s** and aborted on its From 99508cbe9569160448d7a94edb1a4dce05a18bb8 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Mon, 16 Nov 2026 04:13:32 -0400 Subject: [PATCH 2/2] docs: the capture rate and the fault target both go stale within minutes Measured today on one box: the unfiltered map+unmap pair ran at ~1,000 events/s at 06:42Z and 14,449/s at 08:10Z (650,210 in 45 s) because another lane started a DMA-heavy build, so a 30-minute unfiltered capture would have kept only its last ~20 s in the 1,410 KB ring -- a window that cannot contain a fault arriving once per three minutes. And a filter built from the last 25 faults saw 8 events in 30 s, then 0 in 30 s five minutes later. So: re-derive the address list and the window size from a positive control taken seconds earlier, and let that control abort the run -- it did, three times today, and each abort saved a capture that would have reported nothing. --- docs/nvme-iommu-fault-triage.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index b76fec5e2..5548546cf 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -193,6 +193,15 @@ is not.** kprobe slot on `iommu_map`/`iommu_unmap_fast` until the next reboot, and there is no userspace way to remove it. A fresh event created and deleted in the same minute works fine, so the subsystem is not wedged -- only that event is. +5. **Both the rate and the target go stale within minutes, so re-derive them immediately before arming.** Measured + on this box: an unfiltered `iommu_map`+`iommu_unmap_fast` pair ran at **~1,000 events/s** at 06:42Z and at + **14,449/s** (650,210 in 45 s) at 08:10Z, because another lane started a DMA-heavy build -- a 30-minute unfiltered + capture therefore keeps only its **last ~20 s** in the 1,410 KB per-CPU ring, which cannot contain a fault at one + per three minutes. And a filter built from the **last 25 faults** recorded 8 events in 30 s, then **0 in 30 s** five + minutes later: the faulting IOVAs move on faster than a capture can be planned. Consequence for a capture owner: + **refresh the address list and size the window from a positive control taken seconds, not minutes, earlier** -- and + let the positive control abort you, which is what it did three times today. + **Treat each event's `filter` as write-once.** To change a filter, delete the event and recreate it; do not clear or edit one in place. And after deleting, **check `grep -c 'kprobes/' kprobe_events` back to its previous count** -- cleanup that fails silently is how a probe outlives the run that armed it, which is the rule this page already has