From 8801260a524ced892d932b68746c78e315fe7046 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 10:32:18 -0400 Subject: [PATCH] The runbook is a method, not an NVMe story: add the NPU as the second worked example engine#2542 applied the same instrument to a different device on the same host (NPU, domain 0x0003) and got a diagnosis in two measurements without touching it. Four things generalise and each was a decision: split by (device, domain) before counting; decode flags (0x0007 = write to a non-present page, so not a permission fault on live memory); read the walk rather than the count (9 contiguous pages plus one 16 MiB above = two ranges in one stride, i.e. a descriptor list, not a scatter); and eliminate the spaces a number could belong to (op 18 is outside both the ioctl ids 0-9 and the command types 0-2, and the engine cannot name it because its NPU path goes through xrt::run). It also records what the pre-flight bought: the live probe on those ten IOVAs found nothing in 8 s, which correctly means 'these buffers are idle now', not 'no unmaps' -- hours-old bursts are not observable live, and that is a property of the instrument. --- docs/nvme-iommu-fault-triage.md | 34 +++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index b86a1afce..501d1ff9b 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -178,6 +178,40 @@ Every one of them produces the same symptom — **an empty result** — which is might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates "nothing happened" from "nothing was watching". +## The same method on a second device (the NPU) + +The title says NVMe because that is where the fault stream was found, but nothing in the method is NVMe-specific: +engine#2542 is the same instrument applied to the NPU (`0000:c6:00.1`, domain `0x0003`) on the same host, and it +produced a diagnosis in two measurements without touching the device. + +``` +20 events this boot, two bursts: 10 @ 08:43, 10 @ 08:53 +walk: 0x4128000 0x4129000 … 0x4130000 (9 contiguous pages) + 0x5128000 (1 page, 16 MiB above) + 0x4128000 … (restarts) +flags=0x0007 on all 20 +``` + +Four things generalise, and each one was a decision rather than a discovery: + +1. **Split by `(device, domain)` before counting.** The NVMe stream is domain `0x0015`, the NPU's is `0x0003`; a raw + `grep -c IO_PAGE_FAULT` mixes them and inflates whichever you were looking for. `tools/diag/iommu_fault_census.py` + exists for exactly this. +2. **Decode `flags`.** `0x0007` is present=0, write=1, user=1 — a **write to a non-present page**, which rules out a + permission fault on live memory and points at an address that was never (or no longer) mapped. `flags=0x0000` on the + NVMe stream means the same for a read. +3. **Read the walk, not the count.** Nine contiguous pages followed by one page far above is *two ranges in one stride* + — a descriptor list whose tenth entry lives elsewhere — not a scatter of unrelated pointers. The stride tells you + which structure to look for; the count tells you nothing. +4. **Eliminate the spaces a number could belong to.** `Unsupported op 18` is outside the DRM ioctl ids (0–9) and the + command types (0–2) in the shipped UAPI, so it is an opcode in the command stream, and the engine repo cannot name it + either (its NPU path calls `xrt::run`; there is no opcode table in-tree). Three candidates eliminated in one pass. + +The pre-flight pattern above is what made the NPU run cheap: the live probe on those ten IOVAs found nothing in 8 s, +which correctly said "these buffers are idle now" instead of being mistaken for the absence of unmaps. Hours-old bursts +are not observable live — that is a property of the instrument, and knowing it saved a twenty-minute capture that could +only have reported "idle". + ## Two traps this runbook exists to stop * **A count without a device split is not a measurement.** `distinct` exposes the mixing.