From 5bed50c6fa46000204f379e6df9cc5f6c7ebf2ca Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 16:40:12 -0400 Subject: [PATCH 1/2] Add the fifth ruled-out shape, and the shared-host caveat it taught The reproduction table listed four shapes; a fifth has since been tried (8 GiB held during 30 direct reads) and it holds the same result: zero faults. A table that claims to be the reproduction status and omits a tried shape is worse than no table, because the next person re-runs it. It also demotes the memory-pressure candidate properly: what is ruled out is MILD pressure on a 122 GiB box, not the pressure a cmake build applies -- and that distinction is now in the page instead of implicit in a comment. And it carries the general lesson the test produced: MemAvailable moved 58 -> 88 GiB during my own thirty-second window from other lanes' work, so a memory experiment on a shared host must record the baseline either side or it cannot tell its own effect from a neighbour's. --- docs/nvme-iommu-fault-triage.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 225ab6de6..b5dfd9ef7 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -190,17 +190,24 @@ attempts have already been made and failed. All four against a fixed baseline (7 | random direct reads | 400 × 64 KiB at random offsets | **0** | | concurrent direct reads | 24 threads, ~3.8 GB | **0** | | direct writes + fsync | 1 GiB written, read back, unlinked | **0** | +| memory pressure + reads | 8 GiB held and touched during 30 direct reads | **0** | So **the stream is not reproducible from userspace I/O** — read or write, sequential or random, one thread or twenty-four -— and a reproduction has to come from something else. In order of what the evidence still allows: +— and neither mild memory pressure nor that pressure combined with light I/O brings it back. A reproduction has to come +from something else. In order of what the evidence still allows: -1. **memory pressure / allocator state** — the bursts coincided with builds, which allocate and free heavily; a fault - needs the allocator to hand back an IOVA whose TLB entry has not been invalidated while a command is still - outstanding, and the reuse is real (494 reuses of one address in 15 minutes); +1. **memory pressure far heavier than 8 GiB** — the fifth test only covers *mild* pressure on a 122 GiB box; a `cmake` + build applies more, and the bursts did coincide with builds, which allocate and free heavily. The mechanism it would + have to reach is still real: the allocator hands the same IOVA straight back (494 reuses of one address in 15 minutes), + and a fault needs that hand-back to happen while a command is still outstanding; 2. **uptime** — allocator history may be the precondition, which no short test creates; 3. **a caller outside the block path** — a minority of traced unmaps arrived via `nvme_unmap_data -> dma_iova_unlink -> dma_iova_destroy`, i.e. teardown rather than completion. +One measurement note from the fifth test that applies to any experiment on this host: `MemAvailable` moved from 58 to +88 GiB **during my own thirty-second window**, purely from other work. A memory-pressure experiment on a shared box has to +record the baseline either side of the window, or it cannot separate its own effect from a neighbour's. + Two lessons from that list. The first three shapes were chosen because of **my own story** about the trigger, and the story was wrong — cheap tests turned a plausible narrative into four ruled-out candidates. And the fourth is the one I nearly did not run, because I had convinced myself it needed someone's permission; it needed a bounded write to this lane's own From 06ce73f6ad557736159c1e96e318c36179b4ecee Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 17:11:01 -0400 Subject: [PATCH 2/2] The five zeros answered a weaker question than the table claimed Every one of the five attempts ran from ~16:00 onward; the last NVMe fault was 08:53:17. So faults in the ten minutes before each attempt were 0, 0, 0, 0, 0, and each '0' answers 'did this load restart a stopped stream' -- not 'does this load trigger the stream'. Five zeros do not strengthen that answer, they repeat it. The table gains a 'faults in the 10 min before' column and a 'what it shows' column, and the summary now says what the data supports: five shapes tried while the stream was already stopped, none restarted it. That is still useful (the obvious ways to provoke it do not work) but it does not rule those loads out as triggers, because none was run against a live stream. The earlier 'four shapes tried, none reproduced it' claimed more; the fifth row only made the pattern obvious. Reproducing it needs a window while faults are arriving, with the rate recorded before and during -- otherwise a run can only report what these five did. --- docs/nvme-iommu-fault-triage.md | 58 ++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index b5dfd9ef7..e1a6129d8 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -178,41 +178,47 @@ Every one of them produces the same symptom — **an empty result** — which is might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates "nothing happened" from "nothing was watching". -## Reproduction status: four shapes tried, none reproduced it +## Reproduction status: five shapes tried while the stream was already stopped -Recorded before the theory, because the first thing anyone will try is to make the stream happen again, and the obvious -attempts have already been made and failed. All four against a fixed baseline (740 faults), all under `thermal-run.sh` -(nice 19, ionice idle), all through `O_DIRECT` — i.e. the per-request DMA mapping path the stack named: +Recorded before the theory, because the first thing anyone will try is to make the stream happen again. **But read the +"before" column first** — it is the difference between this table being evidence and being five ways of saying nothing. -| attempt | load | faults | -|---|---|---| -| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | **0** | -| random direct reads | 400 × 64 KiB at random offsets | **0** | -| concurrent direct reads | 24 threads, ~3.8 GB | **0** | -| direct writes + fsync | 1 GiB written, read back, unlinked | **0** | -| memory pressure + reads | 8 GiB held and touched during 30 direct reads | **0** | - -So **the stream is not reproducible from userspace I/O** — read or write, sequential or random, one thread or twenty-four -— and neither mild memory pressure nor that pressure combined with light I/O brings it back. A reproduction has to come -from something else. In order of what the evidence still allows: - -1. **memory pressure far heavier than 8 GiB** — the fifth test only covers *mild* pressure on a 122 GiB box; a `cmake` - build applies more, and the bursts did coincide with builds, which allocate and free heavily. The mechanism it would - have to reach is still real: the allocator hands the same IOVA straight back (494 reuses of one address in 15 minutes), - and a fault needs that hand-back to happen while a command is still outstanding; +``` +last NVMe fault this boot: 08:53:17 the five attempts ran from ~16:00 onward +faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0 +``` + +| attempt | load | faults in the 10 min before | faults during | what it shows | +|---|---|---|---|---| +| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | 0 | **0** | does not restart a stopped stream | +| random direct reads | 400 × 64 KiB at random offsets | 0 | **0** | same | +| concurrent direct reads | 24 threads, ~3.8 GB | 0 | **0** | same | +| direct writes + fsync | 1 GiB written, read back, unlinked | 0 | **0** | same | +| memory pressure + reads | 8 GiB held/touched during 30 direct reads | 0 | **0** | same | + +**So the honest summary is narrower than "reproduction ruled out": five shapes were tried while the stream was already +stopped, and none of them restarted it.** That is worth knowing — a stopped stream is not restarted by load, and the +obvious ways to provoke one do not work — but it does **not** show that those loads cannot *trigger* the stream when it is +running, because none of them was ever run against a live one. An earlier version of this section said "four shapes tried, +none reproduced it", which claimed more than the data supports; the fifth row made the pattern obvious but did not cause it. + +Reproducing it therefore needs a window when the stream is live, which means either waiting for the box to produce one +(it happened during other lanes' builds) or running the same shapes *while faults are arriving* and recording the rate +before and during. If a run starts from a quiet stream, it can only report what these five did: no restart. + +What the evidence still leaves as candidate *triggers*, unchanged by the five zeros: + +1. **memory pressure far heavier than 8 GiB** — a `cmake` build on a 122 GiB box applies more, and the bursts coincided + with builds. The mechanism it would reach is real: the allocator hands the same IOVA straight back (494 reuses of one + address in 15 minutes), and a fault needs that hand-back to happen while a command is still outstanding; 2. **uptime** — allocator history may be the precondition, which no short test creates; 3. **a caller outside the block path** — a minority of traced unmaps arrived via `nvme_unmap_data -> dma_iova_unlink -> dma_iova_destroy`, i.e. teardown rather than completion. -One measurement note from the fifth test that applies to any experiment on this host: `MemAvailable` moved from 58 to +One measurement note the fifth test produced, which applies to any experiment on this host: `MemAvailable` moved from 58 to 88 GiB **during my own thirty-second window**, purely from other work. A memory-pressure experiment on a shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's. -Two lessons from that list. The first three shapes were chosen because of **my own story** about the trigger, and the story -was wrong — cheap tests turned a plausible narrative into four ruled-out candidates. And the fourth is the one I nearly -did not run, because I had convinced myself it needed someone's permission; it needed a bounded write to this lane's own -scratch directory. - ## The same method on a second device (the NPU) The title says NVMe because that is where the fault stream was found, but nothing in the method is NVMe-specific: