From 4994b1e7e1514abf221fe01f0114eb7a1c9b5739 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Mon, 16 Nov 2026 02:57:38 -0400 Subject: [PATCH] The runbook ends with the resolution now, not with method The page documented how to measure this stream and retired one hypothesis; it did not say what the answer was. It does now: the faults outlive the process that caused them (43 after the last holder exited, on addresses never mapped or unmapped while watched), which only a teardown-side explanation survives. names the path -- npu/generate.cpp's overlapping loop calls prepare() (which throws past kMaxContext) BEFORE wait(cur), so the catch-all reaches end(), which frees runlists, kernels and hw_context while the runlist may still run -- plus the trigger, the prediction for the confirming run, and three fixes in cost order. The section closes with the sentence this page earned: a stream that continues after its only holder has exited is not a concurrency problem but something the holder left behind, and watching the device's mappings during the fault cannot find it because the mapping was torn down before the observation began. --- docs/nvme-iommu-fault-triage.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 95ada4752..d49a97fdc 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -173,6 +173,35 @@ Two things to be careful about, both learned the hard way here: * **pick the right symbol.** `iommu_unmap` had zero hits in 5 s on a box doing heavy I/O; `iommu_unmap_fast` had 40,510. Probe a known-hot function (`ksys_write`) once to prove the plumbing, then argue about symbols. +## Resolution: the fault stream is a residue of a teardown that does not drain + +The investigation ends in `npu/` code, not in the IOMMU, and the evidence for that is the one thing no IOMMU-side hypothesis +could explain: **the faults outlive the process that caused them** (43 faults after the last holder of `/dev/accel/accel0` had +exited, on addresses never mapped and never unmapped for the 45 minutes they were watched). + +```cpp +// npu/generate.cpp: the decode loop overlaps host work with the run in flight +if (more) lane.prepare(next, ctx + 1); // can throw +lane.wait(cur); + +// npu/lane.cpp: …and end() destroys without draining, on a documented assumption +// "Destroy the hw_context while it is warm (every run completed)" +void end() { slots[0].rl.reset(); slots[0].runs.clear(); … ctx.reset(); } +``` + +`prepare()` → `layer_kernel()` throws when `ctx + 1 > kMaxContext` — *before* `wait(cur)`. The catch-all in `generate()` then +calls `end()`, which frees the runlists, the kernels and the hw_context **while the current runlist may still be executing**. +Every BO under that context is released while the controller is still writing it, and the device keeps DMA-ing into IOVAs whose +mappings were just torn down. + +Trigger: **a generation that rolls past `kMaxContext`.** Prediction for the confirming run: the stream appears after the throw +and not after a clean run. Fix, cheapest first: drain in `end()`; or guard `ctx + 1 > kMaxContext` *before* launching; reordering +`wait` before `prepare` is correct but costs the overlap that is deliberate. + +The lesson this page earned, in one line: **a stream that continues after its only holder has exited is not a concurrency +problem — it is something the holder left behind**, and no amount of watching the device's mappings during the fault will find +it, because the mapping was torn down before the observation began. + ## Validating a capture before you believe it Four failures in one afternoon on this row, and every one was method rather than kernel. The pattern that catches all