diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index dded44bd0..b86a1afce 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -131,6 +131,53 @@ Two things to be careful about, both learned the hard way here: * **pick the right symbol.** `iommu_unmap` had zero hits in 5 s on a box doing heavy I/O; `iommu_unmap_fast` had 40,510. Probe a known-hot function (`ksys_write`) once to prove the plumbing, then argue about symbols. +## Validating a capture before you believe it + +Four failures in one afternoon on this row, and every one was method rather than kernel. The pattern that catches all +four costs about ten lines: + +All of this needs root on the tracing filesystem; run the block under `sudo -s`. + +```bash +# 1. the probe exists and fires -- verify with a known-hot symbol the first time you use a new tracefs +echo "p:hot ksys_write" >> /sys/kernel/tracing/kprobe_events +echo 1 > /sys/kernel/tracing/events/kprobes/hot/enable; sleep 3 +grep -c 'hot:' /sys/kernel/tracing/trace # must be > 0 before you trust any other probe +# remove THIS probe only: disable it, then delete it by name. `echo > kprobe_events` would truncate the file and +# remove every lane's probe definitions on a shared box -- and a still-enabled probe cannot be removed at all. +echo 0 > /sys/kernel/tracing/events/kprobes/hot/enable +echo '-:hot' >> /sys/kernel/tracing/kprobe_events + +# 2. pre-flight INSIDE the long run: abort instead of sleeping on a dead instrument +echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5 +H=$(grep -c 'unm:' /sys/kernel/tracing/trace) +[ "$H" -gt 0 ] || { echo "INVALID: probes silent, aborting"; exit 1; } + +# 3. copy the output, then make sure it is readable +sudo cp /sys/kernel/tracing/trace ./cap.txt && sudo chmod 644 ./cap.txt + +# 4. read it back before analysing: an unreadable file greps as "no events" +# `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so `|| echo FAILED` reports a legitimate empty capture as a +# permissions problem -- the exact confusion this section exists to prevent. Separate the count from the error. +n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data" +echo "events: $n" # zero is a result; rc 2 is a broken read +``` + +The four ways this went wrong here, in the order they happened: + +| mistake | what it looked like | what it cost | +|---|---|---| +| `echo > .../filter` to clear a filter | `event filter parse error`; the old filter stayed and the probe matched nothing | a 12-minute capture reported "0 unmaps" | +| probing `iommu_unmap` because that is the name you remember | 0 hits in 5 s on a box doing heavy I/O | a wrong conclusion about which path unmaps take | +| enabling a probe whose definition a previous cleanup had removed | silence, reported as "no events" | a 20-minute capture measured nothing | +| `sudo cp` output, then `grep 2>/dev/null` | silence, reported as "no events" | the data was in the file; only the read failed | +| `grep -c ... || echo FAILED` on a legitimately empty file | "FAILED", though the read was fine | a real zero called a permissions problem | +| `echo > kprobe_events` to clean up | every lane's probe definitions gone (and a still-enabled probe cannot be removed) | other people's instruments, silently | + +Every one of them produces the same symptom — **an empty result** — which is indistinguishable from the finding you +might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates +"nothing happened" from "nothing was watching". + ## Two traps this runbook exists to stop * **A count without a device split is not a measurement.** `distinct` exposes the mixing.