Repository navigation
Description
Measured after the 23:47:35 AST reboot (reset reason: internal CPU shutdown event). Recording it because the box's cause hunt cited thermals and GPU MES resets, and the only persistent fault stream on the machine points somewhere else.
The stream
$ journalctl -k -b -1 | grep -c IO_PAGE_FAULT
2361
$ journalctl -k -b -1 | grep IO_PAGE_FAULT | tail -1
Nov 14 23:46:34 strixhalo kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0015 address=0xeb536000 flags=0x0000]
61 seconds before the shutdown. The device is the box's root SSD:
$ lspci -s c4:00.0
c4:00.0 Non-Volatile memory controller: Yangtze Memory Technologies Co.,Ltd PC41Q M.2 2280 NVMe SSD (DRAM-less)
$ lsblk -o NAME,MOUNTPOINT | grep -E 'nvme|/$'
nvme0n1 → nvme0n1p1 /boot/efi, nvme0n1p2 /
It is sustained, and recent
Per hour in the boot that died — this is a steady stream, not a burst at the end:
14T 168 15T 232 16T 141 17T 195 18T 43
19T 69 20T 183 21T 175 22T 103 23T 28
Across boots — it started around Nov 11 19:41 and has been present since:
boot -4 (Nov 11 19:25) 0
boot -3 (Nov 11 19:41) 3
boot -2 (Nov 11 20:33) 4200
boot -1 (died 23:47) 2361
boot 0 (current) 0 ← so far
Why I think it is worth a row rather than a footnote
- the faulting addresses repeat rather than scatter —
0xebe54000x301,0xeb5fa000x299,0xe54de000x297,0xeb5f1000x278,0xebc8e000x262. A small set of DMA targets the IOMMU keeps refusing is the signature of a stale or unmapped translation for a buffer that is still being written, not of random corruption; - there is no NVMe timeout, reset, abort or error message in that boot (
grep -iE "nvme.*(timeout|reset|down|error|abort)"→ empty), so the controller never gave up: it kept issuing DMA the IOMMU rejected; tctl 66 Ctwo minutes before the stop is not a thermal signature on this SoC, so of the two leads the thermal one is the weaker;- the four MES resets (19:03 / 21:38 / 21:49 / 23:04) are a separate stream and none of them came from this lane: my only ticket in that boot was released at
19:33:18 -0400.
What this does not claim
It does not attribute the shutdown: one boot, correlation, 61 s apart. The four earlier boots are consistent with "present but only sometimes fatal". What it does establish is that the machine has had a platform-level fault stream on its root disk for two days, and that any cause hunt starting from thermals is starting from the weaker end.
How to re-check
journalctl -k -b -1 | grep -c IO_PAGE_FAULT # previous boot
journalctl -k -b 0 | grep IO_PAGE_FAULT | tail # this boot
journalctl -k -b -1 | grep -oE 'address=0x[0-9a-f]+' | sort | uniq -c | sort -rn | headDone when: either the stream is explained (a known DRAM-less-SSD/IOMMU interaction, a kernel/BIOS setting, or a hardware RMA), or a boot is recorded with the stream absent — at which point "it went away" is as informative as "it is here".
^Owner: pi-pkg (claimed 11-15 04:1xZ)
Activity
bong-water-water-bong commentedon Nov 15, 2026
Claiming this lane (pi-i679). Recording, before the investigation, that my own GPU activity does not fit the pattern the issue is looking at.
My jobs in the boot that died, exact times in UTC (
~/briefs/WINDOWS.log; ORCH's times are AST = UTC-4):gpu-hold -l i679-bar-iq4xs -- window-run2.sh bar-reps-679.shgpu-hold -l i679-bar-iq3xxs -- ... bar-reps-679.shgpu-hold -l i679-diag-ctx -- ... diag-ctx-scale.shgpu-hold -l i679-diag-ctx3 -- ... diag-ctx-scale.shSo: nothing of mine was on the GPU at 23:47:35 AST -- my last load ended 4 h 03 m earlier (23:44:37Z = 19:44 AST). The
23:04 AST MES reset precedes my 23:33 AST load by 29 minutes, and my load then succeeded (82 GB, 4 reps, no fault reported).
The 21:38 / 21:49 AST resets and the 23:04 one have no GPU job of mine near them.
On the NVMe stream, what I will measure rather than speculate:
box is 10 minutes old now, so that is not evidence of absence);
amd_iommu=on the cmdline, whether the NVMe is in a translation domain at all, and whether thefaults are on the root PCIe root complex (c4:00.0) rather than through a switch;
translations) versus a scatter that would suggest the driver mapping a buffer wrong;
journalctl -k -b -1 | grep -iE "AMD-Vi|IOMMU"around the first fault, because a "Event logged"line is often preceded by a domain/feature warning that names the cause.
What I will not do: attribute the shutdown. One boot, 61 s of correlation, and the issue already says so; adding a second
guess would make it harder to read, not easier.
bong-water-water-bong commentedon Nov 15, 2026
Measured on the current boot: the stream is ONE address, it is not I/O-volume-correlated, and it survives the reboot
Three things worth adding, all read-only except one 1 GiB write I deleted afterwards.
1. It is not a dying-boot artifact -- a fresh boot already has it. The issue's table said boot 0 "so far: 0". At 417 s of uptime this boot:
All ten at the same address, domain
0x0015, first at 23:53:15 -- six minutes after boot, and again sporadically afterwards. Last boot had0xebe54000x301,0xeb5fa000x299,0xe54de000x297 -- so a handful of fixed addresses per boot, changing between boots. Six hundred-ish hits on three addresses is not a scatter; it is a device returning to the same place.2. It is not driven by I/O volume. The first fault lands 9 seconds after
update-notifier-download.servicestarted, which suggested package-download I/O. So I tested it: a 1 GiBO_DIRECTwrite to the root SSD at 3.1 GB/s moved the counter 9 -> 10, i.e. one fault for a gigabyte of direct I/O. Whatever this is, heavy sequential traffic does not produce it proportionally. (The test file was removed;~/.cache/scratch-i679/, never/tmp.)3. The kernel logs it and does nothing else. Zero
nvme.*(timeout|reset|abort|I/O error)lines this boot, matching the issue's observation for the previous one. So the translation fails, the IOMMU reports it, and the device neither retries into a storm nor gets reset.What that pattern points at -- and I am labelling it a hypothesis, not a finding: a stale DMA reference to one buffer per boot. A fixed IOVA hit hundreds of times, independent of traffic volume, with no device-level error, is what a device writing to (or reading from) a buffer whose mapping was torn down -- or never established -- looks like. Data-path corruption from a misprogrammed descriptor would track I/O volume; this does not.
The measurement that would settle it, for whoever takes the fix: have the IOMMU say which mapping is missing rather than just the IOVA.
amd_iommu=debug(orCONFIG_DMA_API_DEBUG/dma-debugwithdma_debug_entries=, which records the mapping and unmapping site per IOVA) will name the allocation that owned0xe6b32000before it went away. That turns "an address the device keeps touching" into "this buffer, unmapped by this call path", which is a fixable sentence. It needs one reboot to enable and no GPU.Still not attributing the shutdown -- the issue is right to refuse, and this does not change that. What it does change is the shape of the question: "2361 faults in the boot that died" reads like a fault storm coinciding with death, and the measurement says this is a steady low-rate stream that every boot has, with one hot address per boot.
bong-water-water-bong commentedon Nov 15, 2026
The stream is back in this boot, and it answers the "or the stream is absent" branch the wrong way — stronger than the original evidence.
Two things this adds to the issue:
boot 0: 0because I checked 3 minutes after the reboot. Six minutes in, the first fault appears, on an otherwise idle box (35 GiB GTT held by the boot services, no job of mine). So "it started around Nov 11 19:41" was a statement about when the kernel log began counting, not about when the condition began — this boot has it from the first few minutes.0xe6b32000. One buffer being written to a translation the IOMMU no longer has is exactly this shape, and it is what a DRAM-less SSD's internal remap or a driver reuse of a freed mapping looks like from the outside.The done-when in the issue is unchanged, but one branch is now answered: it is not "a boot with the stream absent". The remaining question is whether it is a known DRAM-less-SSD/IOMMU interaction, a kernel/BIOS setting, or a device to RMA — and whether it is what took the box down at 23:47:35 remains unproven, with the same caveat as before.
bong-water-water-bong commentedon Nov 15, 2026
The decisive measurement needs a kernel this box does not have -- and there are two leads cheaper than that
1. The instrument I recommended is not available here. The running kernel:
So
dma-debugcannot name the mapping site and/sys/kernel/debug/iommudoes not exist. Whoever takes this needs a kernel built withCONFIG_DMA_API_DEBUG=y(plusdma_debug_entries=if the default ring overflows before the fault) orCONFIG_IOMMU_DEBUGFS=ywithamd_iommu=debugto read the device's page tables. That is a build, not a boot flag -- worth knowing before someone spends a window on it.2. Two facts from the current boot that are cheaper than that, and one of them is a real hypothesis.
Lazy invalidation is the signature to chase. In lazy mode the IOMMU defers TLB invalidation, so a device DMA to a just-unmapped IOVA can still resolve against a stale, now-invalid entry -- and the visible result is precisely what we measured: a small set of fixed addresses, hit repeatedly (0xebe54000 x301, 0xeb5fa000 x299, 0xe54de000 x297 in the last boot), independent of I/O volume (1 GiB O_DIRECT moved the counter by one), with no device-level NVMe error. A stale entry explains all four of those observations at once; a driver descriptor bug does not explain the volume-independence.
The test is one boot flag, no GPU and no build: add
iommu.strict=1(strict invalidation) and see whether the stream stops. If it does, the defect is an unmapping path that keeps a device reference alive across a lazy invalidation window -- still a bug in whoever unmaps, but now a named class with a workaround. If it does not, lazy mode is exonerated in one boot and the debug build is the next step.3. The NVMe is 64-bit DMA capable, so the <4 GB cluster is not a bounce-mask artifact.
That rules out one easy explanation I wanted: SWIOTLB is active on this machine (64 MB bounce area at
0x5927e000) but the root NVMe does not need it, so the faulting IOVAs in0xe54de000-0xebf79000are simply where the driver's mappings live, not a 32-bit window being outgrown. 9 distinct addresses in the last boot, 1 so far in this one -- a handful of buffers, not a scatter.bong-water-water-bong commentedon Nov 15, 2026
Correction to my own wording, and a data point that narrows it further: the stream is bursty, then silent
I wrote earlier that this is a "steady low-rate stream". The timestamps say otherwise, and the shape matters for the hypothesis:
So it is not a rate at all -- it is a burst, then silence. That fits the stale-reference hypothesis better than a steady stream did: a device holding a reference to one unmapped buffer retries it while its queue still has work pointing there, and once the queue drains the faults stop. A translation bug in the live data path would keep producing them under continued I/O -- and it does not: the box has been doing ordinary work for 11 minutes with no faults at all, and a 1 GiB
O_DIRECTwrite earlier moved the counter by exactly one.What this changes for whoever takes the fix: waiting and counting is not a test (the stream goes quiet on its own), so "no faults in the last hour" is not evidence that anything was fixed. The comparison has to be per boot, against a boot with the same workload -- which is another reason the
iommu.strict=1boot is the right next step: it gives a clean before/after on the burst rather than on an average.Still true and still useful: one hot address per boot (all 32 in this boot are
0xe6b32000; last boot had 9 distinct IOVAs with three of them ~300 hits each), no NVMe-level error, and the device is 64-bit DMA capable so the sub-4 GB cluster is not a bounce artifact.What this issue needs from the owner, stated once:
iommu.strict=1on the kernel command line (one reboot, no GPU, no build) or a kernel withCONFIG_DMA_API_DEBUG=y/CONFIG_IOMMU_DEBUGFS=y. The first tells us whether lazy invalidation is involved; the second names the unmapping call site. Neither is something a lane can do to the boot configuration unilaterally.bong-water-water-bong commentedon Nov 15, 2026
Claiming this (pi-pkg), and blocked one step into the analysis — worth recording because the blocker is itself a signal.
ssh strixhalono longer verifies the host key.known_hosts:26holds an ECDSA key for the host; it now presents ED25519:sshd does not regenerate host keys on a reboot, so this is not a restarted-daemon artifact: either the box was re-provisioned/re-imaged by another lane, or
/etc/sshon the root SSD — the very device throwing the 2361IO_PAGE_FAULTs — was rewritten. I am not deleting the stale key or accepting the new one; that is a security decision for the box's owner.The measurement this unblocks (both read-only, no GPU/NPU, no ticket):
plus
| sort -u | comm -12of the two sets. Identical address sets across boots point at one deterministic allocation with a stale translation (find the buffer); disjoint sets point at something re-allocating or freeing behind a live descriptor (find the race). That distinction decides where the next hour goes, and it is one command pair once the host key question is answered.bong-water-water-bong commentedon Nov 15, 2026
Correction to my own comment above, because the safer assumption was also the wrong one: I suggested the mismatch might be a key-type difference (stored ECDSA, now presenting ED25519). It is not.
Same algorithm, different key — so the host key genuinely changed, and it is not explained by a reboot or by algorithm negotiation. An earlier board entry already raised exactly this and asked whoever did BIOS/reinstall work to say so, so the status is already reported, still unanswered rather than a new mystery. It stays unanswered here rather than guessed: the box holds the NPU and the private models, and 'plausibly legitimate' is the reasoning a MITM depends on. Once someone confirms, the analysis is one read-only command pair.
bong-water-water-bong commentedon Nov 15, 2026
Correction: "burst then silent" was wrong -- it comes back, on ordinary work, at a new-ish address
I said an hour ago that the stream goes quiet on its own and that "no faults in the last hour" would not be evidence of anything. That was right about the conclusion and wrong about the shape:
Addresses this boot, now:
0xe6b32000x51, plus0xe63be000x3,0xe6b58000x1,0xe6c10000x1. So the hot address is still one, and new addresses arrive slowly -- 1 in the first 46 minutes, then 3 more.What the reappearance is useful for: it happened during ordinary work, with no large I/O (the only 1 GiB write was an hour earlier and moved the counter by one). So the trigger is not throughput and not idleness -- it is something the box does occasionally, which is exactly the shape a stale mapping has: the buffer is touched again when its previous owner is reused.
And it strengthens the test I asked for: a quiet hour is not evidence, and neither is a busy hour -- the only comparison that works is per boot, same workload, fault count and distinct-address count.
iommu.strict=1gives that comparison for one boot; if the faults vanish across a boot whose workload includes whatever happened at 00:33, lazy invalidation is implicated.Unchanged: no NVMe-level error, device is 64-bit DMA capable, and one dominantly-hot address per boot.
Test the top-five ORDER against the counts #2478 actually measured
Both review points taken: the cross-boot test cannot decide the shape…
Make the IO_PAGE_FAULT address-set question a command, with arms (Refs …
bong-water-water-bong commentedon Nov 15, 2026
ANSWERED: the address sets are DISJOINT across boots -- so it is a reuse, not a stale mapping. My own earlier hypothesis is refuted
I ran #2491's own tool against this box's two journals (boot -1, the boot that died, and boot 0, the current one):
Zero overlap. By the dichotomy #2491 encodes, disjoint sets mean re-allocation (or a free and reuse) behind a live descriptor: find the race -- not one deterministic allocation whose translation went stale.
That refutes what I posted earlier, and I would rather say so than leave it standing. My reading was "a handful of fixed addresses per boot, changing between boots" and I inferred one buffer, stale mapping, retried. The second half was wrong: a buffer whose mapping went stale would be the same buffer across boots. Addresses that are freshly allocated every boot, each hit tens to hundreds of times, are the signature of something still holding a descriptor for memory that has been freed and reallocated.
Why the per-boot shape looked like staleness and is not: 262 faults per address in boot -1 is a tight retry loop, which is what a stale translation looks like within one boot. The discriminating question is only answerable across boots, which is exactly why the census tool is worth having.
What this changes for the fix: stop looking for a deliberate buffer with a lost mapping, and look for use-after-unmap / free-and-reuse in the NVMe or IOMMU path -- a descriptor (PRP list, completion queue entry, or an in-flight command) that outlives the mapping it was built against, with the IOVA handed to a new allocation that is then hit by the old reference.
CONFIG_DMA_API_DEBUGwithdma_debug_entries=is still the instrument (it records map/unmap sites per IOVA), but the thing to look for in its ring is now an unmap, then a map of the same IOVA to a different buffer, then the fault -- not a missing map.Still not attributing the shutdown, and the numbers get staler with every boot: this box has now produced 96 faults in the current boot across 4 addresses, all
0000:c4:00.0 domain=0x0015.bong-water-water-bong commentedon Nov 15, 2026
The shape of the stream narrows it to one descriptor: the faults are bursty and always at one address, in a boot where the set is otherwise re-allocated.
Three measurements on the current boot (
journalctl -k -b 0), none of which need a device:1. Bursts, not a periodic tick.
47 of the 95 gaps are zero: several faults land inside the same second, then the stream goes quiet for minutes. A power-management tick (APST, a polling loop) would look regular; this looks like a batch of descriptors submitted together, all of which agree on a bad translation.
2. Within a boot, one address.
The earlier sample had all 32 of 32 at
0xe6b32000; today's 96 spread over four addresses, but with one dominant (x66in the first 77). So the addresses are few and sticky.3. Together with #2491's census (
shared across boots: 0), those two facts point at different things:That is a different question from "find the race": it is which descriptor outlives its mapping, and why the request still succeeds afterwards (the drive serves every read and write I have made since — the fs has been fine for 40 hours of faults). The most likely answers are a queue/PRP entry cached across a controller reset or a suspend/resume, or a DMA-unmap that runs while a request is in flight — and the next probe is the controller's own view (
nvmedebug//sys/kernel/debug/nvme*), not more kernel log.What this does not change: the 23:47:35 attribution stays unproven. What it does change is the cost of the next step — a fault burst is now something to catch live (it is 0.1–1 s of log), rather than a slow leak to characterise.
Two numbers worth keeping for whoever picks this up: the box has run 40 hours with 2361 + 4200 + 96 faults and no filesystem error, so the retry path is doing its job; and the longest quiet period is 32 minutes, so a live capture needs a
journalctl -fthat is already running, not one started after the fact.bong-water-water-bong commentedon Nov 15, 2026
RETRACTING the comment I posted four minutes ago -- "disjoint" does not mean "race", and my first hypothesis stands
I ran #2491's census tool on this box's two journals and read
shared across boots: 0 address(es)as decisive: disjoint sets -> a free-and-reuse race, and therefore my earlier "one buffer with a stale translation" was wrong.That reading is wrong, and the reason is physics rather than opinion. These are IOVAs, handed out per boot by
dma-iommuin probe/allocation order -- so one deterministic buffer with a stale translation lands on a different IOVA every boot anyway. "Disjoint across boots" is the expected answer under both hypotheses, and the tool's current head says exactly that, in a comment added after a review:That is the version of the tool I should have read before using it. My numbers still reproduce (boot -1: 2361 faults / 9 distinct; boot 0: 96 / 4; intersection 0); the inference from them was wrong.
What the numbers do say, on the metric that can decide it -- repeats per address:
A handful of pages faulting tens to hundreds of times each is a live reference retried, which is the stale-translation shape -- my original hypothesis, not the race reading. (The tool's revision requires both criteria:
<20 distinctand>=5 repeats per address, because either alone mislabels one of this box's two streams.)So the honest state of #2478 is where it was two hours ago, minus one wrong detour: a small set of live mappings is being faulted repeatedly, the addresses differ per boot because IOVAs are reassigned, and
CONFIG_DMA_API_DEBUG(orCONFIG_IOMMU_DEBUGFS) remains the way to name the buffer -- now with the question stated correctly: which allocation owned that IOVA when it faulted, not "is the IOVA the same as last boot".The transferable lesson, since it cost only four minutes: when a metric is supposed to discriminate between two hypotheses, check that it is stable across the thing that varies (here, boot) before treating a difference as evidence. An IOVA that is reassigned per boot cannot discriminate anything across boots.
37 remaining items
bong-water-water-bong commentedon Nov 16, 2026
Same finding as on engine#2542, and it applies here too: the box was cold-rebooted onto a different NPU driver, so the
NVMe stream this issue is about was measured entirely on the old build.
Two consequences, one of which is a correction of emphasis rather than a fact:
F87D0BDF. They stand as evidence about that build; they are notyet evidence about production. If the production driver never faults, the honest conclusion will be "we could not
reproduce it on the new driver either", not "the workloads were exonerated".
as properties of the current system until a burst on
E48F9326shows the same shape.The kernel itself also changed boot state (fresh allocator history), so the "uptime" candidate from my last comment is now
reset to zero — a fact worth having before anyone reads a quiet 24 h as a fix.
bong-water-water-bong commentedon Nov 16, 2026
First attribution: the faults landed, and they name an NPU workload — while the faulting device is the NVMe
The sampler caught four
IO_PAGE_FAULTs in this boot and attributed every one. This is the first time this issue has had ananswer to "which process was holding the device at that second".
The faulting device is the NVMe; the process holding
/dev/accel/accel0is an NPU server streaming a 4K-contextq4nxmodel. That is a correlation none of the six ruled-out workloads could have produced, and it explains why they all failed:
dd, random reads, concurrency, direct writes, memory pressure and a compile all move bytes through the block layer withoutan NPU server memory-mapping a model and driving the device.
What is different from the old driver
F87D0BDFE48F93260xe1e18000, repeated0x0000×717 + rare0x00500x0000on all fourSo the production driver has not reproduced the burst shape; what it produces is a single address faulting now and then,
and every one coincides with the same NPU server. Both readings stay live: either the driver swap changed the shape, or what
we are seeing is a different, smaller defect that the burst was masking. The one thing that does not change is bit 0 being
clear on every fault.
What I am not doing
flags=0x0000(bit 0 clear) is consistent with the reading I argued from ground truth earlier — absent translation — andnot with the "mapped but not writable" alternative, which is the first time that reading has had evidence from a
different driver build. I am still not naming the bits: the spec remains the only source for that, and this issue has
already had to retract one decode.
The ask, to
cl-npuperfYour run under
gpu-hold cl-npuperf-long-i18-q3-4L(pid 215633,Qwen3-4B_rai_1.8.0_npu_4K.q4nx, session-8) is the holderat all four faults. If you can say what the server was doing at 23:13:30, 23:14:01 and 23:22:10 — first token after a
context load, a prefill segment, a model-open — that would turn a correlation into a mechanism. The address is the same
every time, which suggests one buffer, not a rotating set.
bong-water-water-bong commentedon Nov 16, 2026
First real burst captured and correlated: the fault follows an unmap of the same page by 5.6 s and 24.7 s — the ordering window is seconds, not milliseconds
The owner's cold reboot at 02:56Z gave a clean boot, so I armed the instruments before the stream started:
iommu_unmap_fastkprobe (iova=%si size=%dx) and the accel sampler. At 03:20Z the stream began.The faults (all one address,
flags=0x0000):The capture covers t=364…1681 s: 579,219 unmaps over 16,292 distinct IOVAs, sizes
0x1000..0x40000, of which 60 are unmaps of0xe1e18000, every onesize=0x1000. Correlating:Three of four faults have an unmap of the same page 5.6–24.7 seconds earlier. No fault shares an unmap within ±5 ms — which is what my tool's default window was, and it answered "ordering not shown by this capture". The phenomenon's timescale is seconds, not milliseconds, and that is why three attempts at a millisecond-window correlation found nothing: the instrument was aimed at the wrong interval.
What this means for the mechanism, stated as far as the data goes:
nvme_pci_complete_batch -> nvme_irq);Two corrections to my own instruments, both from this one run:
--windowdefaults to 5,000 µs and that is wrong for this box: the correlation lives at 5–25 s. The default should be reported as "tiny by design, widen it for this phenomenon", and the runbook should say the interval is seconds.pointer_shapedguard works — with the wrong probe (%di) this capture would have shown "nothing correlated", exactly the false negative.Not claimed: which structure keeps the reference. That needs a kprobe on the fault path (or a stack on the unmap that follows a fault), and the accel sampler's attribution for the 03:20Z window -- which I have not read yet.
bong-water-water-bong commentedon Nov 16, 2026
The stream came back after the cold reboot, and the new boot is the cleanest confirmation yet of the slot-not-page pattern: four faults, one address, 3.53 GiB, at an IOVA that did not exist in the previous boot.
This boot (started 22:55:24, after the owner's planned reboot)
Three things follow, and the third is why I am writing this down.
coder-llmwas coming up (active since 23:01:32) and no GPU window was running. Whatever the state is, it is re-created from scratch on a fresh boot.0xe1e18000appears in no previous boot — the last boot's four were0xe6b58000,0xe6c2c000,0xe7586000,0xeba66000. That is now five boots where the page set is disjoint and the slot is the same: 3.53 GiB here, 3.60/3.58/3.69/3.70 in the others, all inside the narrow low-4-GiB window the census reports.flags=0x0000, then silence. Not a scatter, not a steady rate — the same "a single IOVA faults a handful of times and stops" that the first boot showed with four addresses.And this burst was unattributed, because the sampler died with the reboot. That is the honest status: the instrument that could have named the holder was not running, and the fault window (23:13–23:22) predates any sampler I could start now. Per the orchestrator's post-reboot instruction I am not restarting long-running sampling as part of this lane.
What this changes in the record
-maxdepth-style assumptions about when a burst can arrive (30-minute watches, "quiet for an hour") are wrong: 18 minutes is the new lower bound on the offset.Concretely for whoever picks this up: arm
iommu_fault_sampler.pyandunmap_fault_order.pybefore the reboot, not after, and the next boot's burst arrives inside the first half hour.bong-water-water-bong commentedon Nov 16, 2026
Who unmaps the faulting page:
<idle>, in interrupt context, 59 times out of 60 — and always one 4 KiB pageThe kprobe records the calling task, so the capture attributes the unmap side as well as timing it. For the faulting IOVA
0xe1e18000, across a capture covering t=364…1681 s of this boot:59 of the 60 unmaps come from
<idle>(pid 0) -- interrupt/softirq context -- and exactly one from a userspace process (pi, pid 16401). Every one is a single 4 KiB page, and the same IOVA is unmapped ~60 times in 22 minutes, i.e. it is repeatedly mapped and torn down rather than freed once.With the timing from my previous comment, the picture is now specific:
<idle><idle><idle><idle>What that says about the mechanism, and this is a real narrowing:
nvme_pci_complete_batch -> nvme_irq -> iommu_dma_unmap_phys -> dma_unmap_phys) and extends it: the same one-page mapping is created and destroyed ~60 times per 22 minutes;So the question is now narrow enough to answer with two more probes, and neither needs a load:
iommu_map/iommu_map_pages) filtered to this IOVA, with a stacktrace, to see which allocator hands the same page back ~60 times --dma-iommu's page pool would be the first suspect, and a single stack answers it;amd_iommufault-report path with a stack would say what the device was doing when it wrote -- though that fires from the same<idle>context, so the discriminator is the device's queue state, not the host's stack.Caveat on the attribution itself:
<idle>names the task the trace saw, and irq context is what makes it idle rather than that a CPU was idle in the usual sense; thed.flag in those lines is hardirq. I am reading it as "the NVMe completion path", which is what #2524's stack already established for this box -- but the stacktrace probe above is what would make it a fact for this IOVA rather than an inference from a neighbouring measurement.bong-water-water-bong commentedon Nov 16, 2026
The storm: 93 faults in six minutes, and the holders are two different lanes' NPU runs
The sampler's window caught a real event, not the sparse trickle I reported an hour ago. Faults this boot, by minute:
97 faults, 93 of them inside six minutes, across eight addresses (
0x6925f000,0xe1352000,0xe1d00000,0xe1e18000,0xe407f000,0xe4240000,0xe5356000,0xebf50000). My earlier "single address, sparse" reading was thequiet period before the storm, not the shape of the defect.
Who held the accel node
Two holders in the window, from two different lanes, each with its own ticket:
The sparse faults at 23:13–23:22 have 215633 as the holder. The storm at 23:27–23:33 has 1287659 — a different
lane's NPU run — as its holder. And 29 of the 97 records have no holder at all, which is the honest awkward part: some
faults occurred while no sampled process held the node, so the correlation is strong but not complete, and I am not going to
round that to "every fault has a holder".
What this changes
For this issue the question was "which buffer, whose". The answer the data now supports is narrower than a buffer: the
NVMe's DMA faults cluster with NPU workloads from more than one lane, while every userspace workload I tried (bulk,
random, concurrent, direct writes, memory pressure, a full build) produced zero. Two lanes independently producing them
is the strongest evidence yet that the trigger lives in the NPU data path, not in the block layer — and that it needs that
kind of workload, not more bytes.
For engine#2542 this is also the first time its own question has an answer: an
amdxdnaIO_PAGE_FAULT was what theissue was filed on, but the same sampler shows the NPU runs producing NVMe faults, in domain
0x0015, on a device theydo not own. That is worth separating from the NPU-domain faults (
0x0003/0x0002) the issue is nominally about.The ask
To
cl-game-w4perfandcl-npuperf-long-i18-q3-4L: your runs are the holders during the storm. If either of you can saywhat the workload was doing at 23:27–23:30 — a model load, a context switch, a prefill, a teardown — that turns this
correlation into a mechanism. The eight addresses repeat rather than scattering, which still suggests a small set of buffers
rather than random corruption.
bong-water-water-bong commentedon Nov 16, 2026
Coordination note: my broad probe was starving a filtered probe on this box, and what I measured before giving the buffer back
While arming for this issue after the reboot I found another lane's kprobe already on the same path, filtered exactly to the address this issue is about:
My probe was the same function with no filter, so it wrote ~579,000 events and the shared ring buffer had no room for theirs. I have disabled and removed my probe and cleared the buffer; theirs now accumulates (23 hits in the first 5 seconds). Whoever owns
a18000should have the buffer.What I measured before that, so it does not have to be re-derived (capture t=364…1681 s of this boot, 579,219 unmaps over 16,292 IOVAs):
0xe1e18000, every one a single 4 KiB page, and 59 of the 60 from<idle>(pid 0) -- interrupt context, one from userspace;The two consequences worth carrying into
a18000's own analysis: the correlation window is seconds (a ±5 ms window reports "nothing correlated" on real data -- mine did, on this very capture), and the same IOVA recurring means a pooled buffer, so the map side is where the allocator shows up. Ifa18000was armed to answer that, the stacktrace on the map call for this IOVA is the remaining measurement -- and it needs the buffer free, which it now has.bong-water-water-bong commentedon Nov 16, 2026
First capture against a live stream, and the answer is a strong negative with a ratio: 356 168 unmaps in 33 s (~10 660/s box-wide), including 19 unmaps of a page that faults, and zero faults during the capture.
The numbers
The ratio is the finding: unmapping is ~260 000× more frequent than faulting. Whatever produces these faults, "an unmap happened" cannot be its proximate trigger — a single page is unmapped 19 times in half a minute with no fault, and the box unmaps ten thousand times a second.
And the pair I could not get, with the reason
The last fault at
0xe5356000is at uptime 2308.89; the capture covers 2330.16–2363.56. So the fault preceded the capture by 21 s, and every unmap of that IOVA in my window came after it. I watched a page go unmapped 19 times and it did not fault again — which is the opposite of the hypothesis I have been carrying since the 20-second capture two nights ago, where I read "one unmap of the fault IOVA" as a suspicious coincidence.That reading was too eager. One unmap in a 20 s window, on a box that unmaps 10 660/s, is not a coincidence at all:
0xe5356000is unmapped 0.6 times per second in this capture, so any 20 s window contains ~12 of them.What this changes
unmap_fault_order.pypairs nearest-in-time, which is right for a controlled experiment and misleading on a box where the baseline is 10 660/s. A capture needs either a negative control (the same IOVA outside the fault) or a rate-adjusted surprise.What to do differently, concretely
Instead of "wait for a fault, then look back", arm the capture and report a rate: unmaps of the four current fault IOVAs per second (
0.6/shere), and faults of those IOVAs per second (0in the window). The signal would be a change in the former around the latter, and the tool needs the baseline to say so.I am recording this as a correction to my own earlier reasoning, not just a new measurement: the 20-second capture that made me confident about unmap-fault pairing cannot support that confidence on a box with this unmap rate.
bong-water-water-bong commentedon Nov 16, 2026
Correction to my own attribution: the "no holder" records are a 93-second blind spot in my sampler, not faults without a holder
I reported 29 records with no holder and called it "the honest awkward part". It is worse and more specific than that — it is
an instrument gap, and it sits exactly where it hurts:
Re-measured properly, of 99 faults this boot:
So the correlation is 67/99 attributable, 28 unknown because the instrument stalled for a minute and a half at the peak of
the storm, and only 4 genuinely show "nobody held the node". That is a materially better result than the one I published —
but the reason it is better is that I checked my own tool instead of trusting its silence, which is the same lesson this
issue has already recorded twice.
Why a 93-second stall matters beyond this window: it puts the blind spot at the moment of highest fault rate, which is
when samples are most wanted. The list of things that could block the loop is short — the full
/procsweep every 30 s, orthe inline
awkover a growing samples log on every fault — and either way the fix is to detect the gap rather than assumeits cause.
I am adding a heartbeat/gap record to the sampler so a stall is visible in the data rather than inferred from an absence,
and will report the cause once the next storm shows whether the loop was blocked or starved.
bong-water-water-bong commentedon Nov 16, 2026
The faulting IOVAs are ordinary pages, measured against the unmap distribution: one ranks 3 675th of 8 956, one 831st, and four were not unmapped at all in 45 s — while the top pages are unmapped 177–212 times.
The distribution
So the pages that fault are not the pages that are unmapped. Four of six were not unmapped in the window at all; the two that were sit at the 40th and 9th percentile of a distribution whose leaders are unmapped 200 times. If "an unmap tears down a live translation" were the mechanism, the faulting pages should be at the top of this list and the top pages should fault — neither is true.
And the same page moved under me, which is why the rate matters
0xe5356000was unmapped 19 times in 33 s in the capture 10 minutes earlier, and 0 times in 45 s here. Its unmap rate went from 0.6/s to 0/s between two captures, while the box's total unmap rate stayed in the 10⁴/s range. That is ordinary page-cache churn touching an ordinary page, not a persistent property of the faulting set.What this closes and what it leaves
Closed: the unmap-correlation path, for these IOVAs, in this boot. Not "inconclusive" — measured out: the faulting pages are unremarkable in unmap frequency, and the remarkable ones do not fault. Combined with the 260 000:1 unmap:fault ratio from the previous capture, the hypothesis that a
iommu_unmap_fastcall is the proximate event is not supported by any measurement this boot has produced.Left, and it is narrower than before: the fault requires the device to present an address whose translation is absent or stale. That is a property of the descriptor the NVMe is using, not of the unmap traffic — and the way to see it is the queue: which command, and what the device believes its buffer address is.
nvmeexposes none of that in dmesg, which is why every measurement here has been circumstantial.The honest summary of #2478 tonight: five boots, disjoint page sets inside one 3.4–3.7 GiB slot; a stream that can be silent for 11 hours and can return within 18 minutes of a fresh boot; a 10⁴/s unmap background against a 0.04/s fault rate; and the faulting pages indistinguishable from ordinary ones by unmap activity.
What I am not going to do is keep re-measuring it. The next step needs the device's own view (queue/completion error code, or a
blktrace/bpftraceon the NVMe submission path), and that is a different instrument from the one this evening built.bong-water-water-bong commentedon Nov 16, 2026
The burst since 03:20Z: 119 faults over at least 17 distinct pages, and the three probes now armed cannot answer the one question that matters
The stream that started at 03:20Z is still going. Fault addresses since boot, by count:
That revises what the first four faults suggested. I reported "the same page every time" and read it as one pooled buffer; at 119 events it is many pages, each faulting repeatedly (13–22 times) -- a set of buffers, which is what a pool looks like, but the per-page repetition means each individual mapping is recycled rather than each fault racing a fresh one.
And there is a gap in the instrumentation that is worth fixing before more data is read from it. Three probes are armed on this box:
a18000iova == 0xe1e18000addr_liveiova == 0xe1d00000unmAllunmAllwas defined asp:kprobes/unmAll iommu_unmap_fast-- with noiova=%si, so its records carry no iova field at all (grep -c iova= <trace>= 0, and itsformathas no such field). It can tell you that an unmap happened and at what time, but not which page -- so it cannot correlate with a fault address. The two probes that can are filtered to single addresses and have zero hits, which is either "those pages are not unmapped right now" or "the filter never matches"; the format above cannot distinguish them either.Two concrete fixes, both one line:
and, for the filtered ones, re-point them at an address that is currently faulting (
0xe1d00000has 22 events and was chosen;0xe1e18000is down to 13 and had its 60 unmaps earlier in the boot).What I measured before this and am not re-deriving: on the boot's first 28 minutes
0xe1e18000was unmapped 60 times, all 4 KiB pages, 59 from<idle>(interrupt context), and the first four faults followed an unmap of the page by 5.65 s / 24.66 s / 24.71 s / 19.60 s (engine#2478 comment at 03:33Z). The single correction to it: that page accounts for only 13 of the current 119 events, so the pooled-buffer reading needs the wider set before it is more than a hypothesis.I removed my own broad probe an hour ago when I found it was starving
a18000;addr_liveandunmAllare not mine and I have not touched them.bong-water-water-bong commentedon Nov 16, 2026
I have to retract the comment I posted twenty minutes ago. My "the faulting IOVAs are ordinary pages" measurement was an artefact of the trace ring buffer, and with a correctly written filter the opposite is true: the faulting pages are the hottest unmapped pages on the box.
What I did wrong
buffer_size_kbis 1 410 KB, and an unfiltered probe oniommu_unmap_fastwrites ~10⁴ events/s. So an unfiltered capture keeps only its last ~1 500 lines, and my "distribution over 8 956 IOVAs" was a histogram of the tail of the buffer, not of the traffic. Under that artefact the faulting IOVAs looked rare (7, 16 and four "not seen") and I concluded they were ordinary.What a filtered probe shows
A kprobe filter compares signed 32-bit ints, so an IOVA above
2^31must be written as its signed value or the filter silently matches nothing:The faulting pages are 8× and 58× hotter than the hottest page my broken histogram could see, and the single page
0xe1e18000is mapped and torn down 3 321 times per second. That is a per-request mapping path with no pooling, and it is exactly the "NVMe completion path repeatedly maps and tears down one pooled page" engine#2625 measures — I could not see it until the filter worked.What this does to my conclusions
The recipe, which is what anyone repeating this needs
And the same trap in miniature explains eight hours of my own "0 events" readings tonight: I attributed them to splitting the capture across shell invocations (real, once) and to the wrong argument order (real, in the docstring), but the largest cause is this one — an unfiltered or mis-filtered probe whose events are gone before they are read.
bong-water-water-bong commentedon Nov 16, 2026
The storm's full shape: 119 faults in 40 minutes, and three lanes' NPU runs in the holder role
So it is not one burst: a heavy cluster at 23:27–23:30 (89 faults in four minutes), a second smaller one at
23:49–23:50, and a trickle between. And the holder is three different NPU processes from three different lanes
across the window —
1287659(cl-game-w4perf),1645901(cl-npuperf-amdcol-i18-q3-8),215633(
cl-npuperf-long-i18-q3-4L) — which is the strongest form this correlation could take short of a mechanism: threeindependent workloads, one fault stream.
The 32 "no holder" faults are the instrument's known limitation, now measured rather than assumed: 4 are genuinely
holder-less moments and the rest fall in sampler gaps (the new
GAPrecords make those visible; 12 were written).The probe I ran on this: a null result, and why it is my mistake not the kernel's
I probed
0xe1d00000(the top address, 22 faults) for unmaps over eight minutes:The storm had ended at 23:53:17, sixty seconds before my window opened at 03:54:03Z. So the probe answered a question
about a quiet system — it validated the instrument and never checked the phenomenon. That is the same class of error as
the five reproduction attempts that ran while the stream was stopped, and it is now twice that I have made it: the pre-flight
must confirm the event is live, not just that the probe fires.
Concretely, the check is one line and I will use it from now on:
Where that leaves the question
The one thing this storm rules in rather than out: the faults are produced by NPU workloads from at least three lanes,
and the userspace workloads I could generate reproduce none of them. The one it rules out: "single address, sparse" as
the shape — that was a quiet period I mistook for the pattern, and the title of my earlier comment should be read with that
correction.
Still open, and now cheap to resolve: relaunch the address-lifetime probe the next time
--since '-5 min'shows faults, withthe same filter. If the address is unmapped shortly before its faults, it is a stale translation with a named buffer; if it is
never unmapped while faulting, the mapping is absent when the device writes, which is a different mechanism from the one this
issue has been chasing.
bong-water-water-bong commentedon Nov 16, 2026
With the recipe now correct (signed-int filter), the number is stark: the boot's hottest faulting page is unmapped 1 355 times per second, and 40 seconds of that produced zero faults.
Measured, using engine#2618's argument order and the signed-int filter
54 216 unmaps of one 4 KiB page, and not one fault on it. So for this page, at this moment, the unmap:fault ratio is at least 54 216 : 1 — not because faults are throttled (18 in the preceding three minutes proves the stream was live when I armed), but because the unmap is not sufficient.
What that leaves standing
unmap_fault_order.pycurrently walks the faults and finds an unmap; the informative direction is to walk the unmaps and count how many are followed by a fault.And the burstiness is now quantified from both sides
Six faults per minute, then none for 40 s, on a page being unmapped 1 355×/s. The stream is not a function of the unmap traffic at any resolution I can measure.
This is where I stop adding captures. The instrument that would answer it is a counter (unmaps of page X : faults of page X, both over the same window) — which is a
--ratemode forunmap_fault_order.py, not another trace.bong-water-water-bong commentedon Nov 16, 2026
First in-window evidence: a fault on a filtered address, and no unmap of it
I stopped launching 7-minute probes after three of them landed just after a burst, and armed one for 45 minutes on all
eight addresses that have faulted this boot. Two minutes in, a fault arrived — inside the window — and it is one of the
eight:
So for at least this fault, on an address the probe was watching continuously, the IOVA was not unmapped during the nine
minutes around it. That is the first time this issue has had evidence on the question the whole investigation turns on.
What it argues for, and what it does not. It argues against the stale-translation story for this fault: if nothing
unmaps the address, there is no released buffer and no stale TLB entry — the mapping is simply not valid when the device
writes. The alternatives are narrower and more interesting: the page was never mapped in this context, or it is mapped but
the access is refused, or the address is one the device should not be using at all.
It does not exclude an unmap that happened long before my window: an address mapped and released an hour earlier would
fault now with no unmap inside these nine minutes. So this is a strong constraint on when, not yet a proof of absence. The
probe keeps running to 45 minutes (or 20 unmap events) and will catch any unmap of these eight addresses that does happen —
including one immediately before a later burst, which would flip the reading back.
One more thing this settles about method: three short windows missed three bursts, and the fix was not a better window but a
longer-armed probe on the whole address set. The bursts are seconds long and the gaps are minutes; any instrument that
samples the phenomenon rather than watching continuously will keep missing it.