From f8e2663b9612c029a53c1d296ce4b3ea43c429fa Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 08:42:23 -0400 Subject: [PATCH 1/2] The four ways a capture lied here today, as a pre-flight pattern Writing the fault-side trace for #2478 I produced four invalid results and one correct one? No -- three invalid results, each of which looked exactly like a finding: a filter that never cleared, a symbol that is never called, a probe whose definition had been removed by my own cleanup, and an output file made unreadable by 'sudo cp' and then grepped with stderr suppressed. All four present as an empty result. The page gains the ten-line pattern that catches all four (prove the plumbing on a known-hot symbol, pre-flight inside the long run so it aborts instead of sleeping, chmod the output, read it back before analysing) and the table of what each mistake looked like. The lesson is one sentence: an empty result is indistinguishable from the finding you were looking for, so the only defence is to prove the instrument before trusting its silence. --- docs/nvme-iommu-fault-triage.md | 37 +++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index dded44bd0..36d2788f5 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -131,6 +131,43 @@ Two things to be careful about, both learned the hard way here: * **pick the right symbol.** `iommu_unmap` had zero hits in 5 s on a box doing heavy I/O; `iommu_unmap_fast` had 40,510. Probe a known-hot function (`ksys_write`) once to prove the plumbing, then argue about symbols. +## Validating a capture before you believe it + +Four failures in one afternoon on this row, and every one was method rather than kernel. The pattern that catches all +four costs about ten lines: + +```bash +# 1. the probe exists and fires -- verify with a known-hot symbol the first time you use a new tracefs +echo "p:hot ksys_write" > /sys/kernel/tracing/kprobe_events +echo 1 > /sys/kernel/tracing/events/kprobes/hot/enable; sleep 3 +grep -c 'hot:' /sys/kernel/tracing/trace # must be > 0 before you trust any other probe +echo > /sys/kernel/tracing/kprobe_events # cleaning up removes the DEFINITION, not just the enable + +# 2. pre-flight INSIDE the long run: abort instead of sleeping on a dead instrument +echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5 +H=$(grep -c 'unm:' /sys/kernel/tracing/trace) +[ "$H" -gt 0 ] || { echo "INVALID: probes silent, aborting"; exit 1; } + +# 3. copy the output, then make sure it is readable +sudo cp /sys/kernel/tracing/trace ./cap.txt && sudo chmod 644 ./cap.txt + +# 4. read it back before analysing: an unreadable file greps as "no events" +grep -c 'unm:' ./cap.txt || echo "GREP FAILED - check permissions, not the data" +``` + +The four ways this went wrong here, in the order they happened: + +| mistake | what it looked like | what it cost | +|---|---|---| +| `echo > .../filter` to clear a filter | `event filter parse error`; the old filter stayed and the probe matched nothing | a 12-minute capture reported "0 unmaps" | +| probing `iommu_unmap` because that is the name you remember | 0 hits in 5 s on a box doing heavy I/O | a wrong conclusion about which path unmaps take | +| enabling a probe whose definition a previous cleanup had removed | silence, reported as "no events" | a 20-minute capture measured nothing | +| `sudo cp` output, then `grep 2>/dev/null` | silence, reported as "no events" | the data was in the file; only the read failed | + +Every one of them produces the same symptom — **an empty result** — which is indistinguishable from the finding you +might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates +"nothing happened" from "nothing was watching". + ## Two traps this runbook exists to stop * **A count without a device split is not a measurement.** `distinct` exposes the mixing. From be8ca313a498df25359ba392e8693c67c6ff381d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 09:32:23 -0400 Subject: [PATCH 2/2] The pattern's own two lines reproduced two of the mistakes it warns about Both review points are right and both are measured in the doc rather than argued: 1. `grep -c 'unm:' ./cap.txt || echo FAILED` -- `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so a legitimately empty capture was reported as a permissions problem: the exact confusion the section exists to prevent, inverted. The count and the error are now separated (`n=...; rc=$?; [ rc -le 1 ]`, then report $n, zero included). 2. `echo > /sys/kernel/tracing/kprobe_events` opens with truncation, so it removes EVERY probe definition on the box, other lanes' included -- and a probe that is still enabled cannot be removed at all, so it can also fail outright. Narrowed to the named probe: disable it, then `echo '-:hot' >> kprobe_events`. Never truncate that file on a shared host, and step 1 now appends rather than opening it for write. Also: the block writes tracefs as root throughout ('run the block under sudo -s'), and the symptom table gains both new rows so the next reader recognises them. --- docs/nvme-iommu-fault-triage.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 36d2788f5..b86a1afce 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -136,12 +136,17 @@ Two things to be careful about, both learned the hard way here: Four failures in one afternoon on this row, and every one was method rather than kernel. The pattern that catches all four costs about ten lines: +All of this needs root on the tracing filesystem; run the block under `sudo -s`. + ```bash # 1. the probe exists and fires -- verify with a known-hot symbol the first time you use a new tracefs -echo "p:hot ksys_write" > /sys/kernel/tracing/kprobe_events +echo "p:hot ksys_write" >> /sys/kernel/tracing/kprobe_events echo 1 > /sys/kernel/tracing/events/kprobes/hot/enable; sleep 3 grep -c 'hot:' /sys/kernel/tracing/trace # must be > 0 before you trust any other probe -echo > /sys/kernel/tracing/kprobe_events # cleaning up removes the DEFINITION, not just the enable +# remove THIS probe only: disable it, then delete it by name. `echo > kprobe_events` would truncate the file and +# remove every lane's probe definitions on a shared box -- and a still-enabled probe cannot be removed at all. +echo 0 > /sys/kernel/tracing/events/kprobes/hot/enable +echo '-:hot' >> /sys/kernel/tracing/kprobe_events # 2. pre-flight INSIDE the long run: abort instead of sleeping on a dead instrument echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5 @@ -152,7 +157,10 @@ H=$(grep -c 'unm:' /sys/kernel/tracing/trace) sudo cp /sys/kernel/tracing/trace ./cap.txt && sudo chmod 644 ./cap.txt # 4. read it back before analysing: an unreadable file greps as "no events" -grep -c 'unm:' ./cap.txt || echo "GREP FAILED - check permissions, not the data" +# `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so `|| echo FAILED` reports a legitimate empty capture as a +# permissions problem -- the exact confusion this section exists to prevent. Separate the count from the error. +n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data" +echo "events: $n" # zero is a result; rc 2 is a broken read ``` The four ways this went wrong here, in the order they happened: @@ -163,6 +171,8 @@ The four ways this went wrong here, in the order they happened: | probing `iommu_unmap` because that is the name you remember | 0 hits in 5 s on a box doing heavy I/O | a wrong conclusion about which path unmaps take | | enabling a probe whose definition a previous cleanup had removed | silence, reported as "no events" | a 20-minute capture measured nothing | | `sudo cp` output, then `grep 2>/dev/null` | silence, reported as "no events" | the data was in the file; only the read failed | +| `grep -c ... || echo FAILED` on a legitimately empty file | "FAILED", though the read was fine | a real zero called a permissions problem | +| `echo > kprobe_events` to clean up | every lane's probe definitions gone (and a still-enabled probe cannot be removed) | other people's instruments, silently | Every one of them produces the same symptom — **an empty result** — which is indistinguishable from the finding you might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates