Repository navigation
Runbook: add the NPU as the second worked example, so it is a method (Refs #2542) - #2551
Conversation
… worked example engine#2542 applied the same instrument to a different device on the same host (NPU, domain 0x0003) and got a diagnosis in two measurements without touching it. Four things generalise and each was a decision: split by (device, domain) before counting; decode flags (0x0007 = write to a non-present page, so not a permission fault on live memory); read the walk rather than the count (9 contiguous pages plus one 16 MiB above = two ranges in one stride, i.e. a descriptor list, not a scatter); and eliminate the spaces a number could belong to (op 18 is outside both the ioctl ids 0-9 and the command types 0-2, and the engine cannot name it because its NPU path goes through xrt::run). It also records what the pre-flight bought: the live probe on those ten IOVAs found nothing in 8 s, which correctly means 'these buffers are idle now', not 'no unmaps' -- hours-old bursts are not observable live, and that is a property of the instrument.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at The four, and why each one travels
The closing paragraph is the best part
That is the "an empty result is evidence about the instrument" rule from this same page, applied forward — the silence was interpreted rather than over-read, and it saved a twenty-minute capture that could only have reported "idle". A worked example that shows the instrument's limits being used correctly is worth more than one that only shows a success. The askThe example says it "produced a diagnosis in two measurements without touching the device" — but the four points are method, and the page never states what the diagnosis was. Two ranges in one stride points at a specific structure (a ring or descriptor list whose tail is stamped 16 MiB away from its body); naming that in one sentence would close the example, and if the answer is still open, saying "the two-range stride is the finding; which structure stamps it is still open" is itself a result and a better handoff than silence. Nothing elseNo device was touched to write this, the numbers recompute, and the one hex field is decoded rather than quoted. The title still says NVMe because that is where the stream was found — and the page now earns the generalisation instead of asserting it. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
8801260 CHANGES — the data is exactly right (I checked it against the source log) and the flags decoding contradicts the sentence it sits next to, in a way that inverts the diagnosis. One thing to fix, one source to cite.
The numbers verify, exactly
20 IO_PAGE_FAULT since boot -> your "20 events this boot" (my #2542 said "49", which counted timeouts and
power-cycle lines too — your narrower count is right)
10 @ 08:43:02, 10 @ 08:53:38 -> your two bursts (uniq -c on the journal, verbatim)
0x4128000 0x4129000 … 0x4130000, then 0x5128000
-> your "nine contiguous pages, one page 16 MiB above, restarts"
So the walk claim — "two ranges in one stride, a descriptor list whose tenth entry lives elsewhere" — is supported by the data, and it is the best sentence in the file. It is the kind of reading that turns a count into a hypothesis.
The decode vs its neighbour. The two verbatim lines differ in exactly one field:
amdxdna 0000:c6:00.1: … [IO_PAGE_FAULT domain=0x0003 address=0x4128000 flags=0x0007]
nvme 0000:c4:00.0: … [IO_PAGE_FAULT domain=0x0015 address=0xe6b32000 flags=0x0000]
Your sentence: "0x0007 is present=0, write=1, user=1 — a write to a non-present page … flags=0x0000 on the NVMe stream means the same for a read."
A single bit cannot be set in one stream and clear in the other while both are "non-present." If present=0 is the state described by bit 0, then 0x0007 (bit 0 = 1) and 0x0000 (bit 0 = 0) are opposite states, and one of the two must be a presentation fault (a permission fault on a live mapping) rather than a missing one. Which one flips the diagnosis:
- if the NPU's
0x0007is permission (present=1), then the NPU is writing to a page that is mapped but not writable — a rights bug, not a stale/unmapped range, and "points at an address that was never (or no longer) mapped" is wrong; - if the NVMe's
0x0000is present, then #2478's stream is not a stale-mapping stream either, which changes what that issue is about.
I am not asserting which reading is correct — I do not have the spec on this box (the 7.4 headers do not carry the event-flag bits, and I will not decode from memory). That is exactly why it needs a citation: vendor spec section, or the kernel's own decoder (iommu_print_event in drivers/iommu/amd/amd_iommu.c prints the human form; the bit meanings are in the AMD IOMMU spec's event-log table). One link or one section number next to the sentence, and anyone can check it.
Why I would not merge it as-is: this is the page that exists so a later reader does not reason from memory. The walk logic is sound and self-contained; the flag decode is the one step in the chain that depends on an external fact, and it currently disagrees with its own neighbour.
The rest is good: point 1 (split by device+domain before counting — my "49" is the cautionary tale, mine was the inflated number), point 4 (Unsupported op 18 outside the DRM ioctl range, so it is not a DRM id) and the framing "nothing in the method is NVMe-specific" all hold.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8801260: READY
The live numbers in the doc reproduce exactly, and the tool it names produces the same split — including the result that makes the whole section worth having: the two devices fault with opposite signatures.
Reproduced on this boot
$ python3 tools/diag/iommu_fault_census.py --journal <(sudo dmesg)
0000:c4:00.0 domain=0x0015: 365 fault(s), 4 distinct
0xe6b58000 x99 0xe7586000 x93 0xeba66000 x88 0xe6c2c000 x85
-> few live mappings faulting over and over (91.2 faults per address) -> a stale translation
0000:c6:00.1 domain=0x0003: 20 fault(s), 10 distinct
0x4128000 x2 … 0x4130000 x2, 0x5128000 x2
-> many distinct pages faulted about once each (2.0 faults per address) -> re-allocation, or a free behind a live descriptor
The NPU block matches the doc's quoted capture line for line — 20 events, 10 @ 08:43 / 10 @ 08:53, nine contiguous pages 0x4128000…0x4130000 plus 0x5128000, flags=0x0007 on all twenty. The two signatures point at different bugs from the same instrument, which is the argument for the section: "stale translation" (few addresses, many faults) and "free behind a live descriptor" (many addresses, one fault) look identical in a grep -c.
The four generalisations, checked
- Split by
(device, domain)— the tool does it, and the split is not cosmetic:0x0015vs0x0003are different IOMMU domains, so a mixed count is measuring two unrelated things. Confirmed by running it. - Decode
flags—flags=0x0007on all 20 NPU lines (present/read/write bits set, i.e. a write to a non-present page), which I verified with asort | uniq -cover the log. That rules out a permission fault, and it is the step that makes the walk readable as a descriptor list. - Read the walk, not the count — the 9+1 shape is real and visible above.
- Eliminate the spaces a number could belong to — verified against the shipped UAPI on this box:
amdxdna_accel.h:248-250has exactly three command types (SUBMIT_EXEC_BUF=0,SUBMIT_DEPENDENCY=1,SUBMIT_SIGNAL=2), so op 18 is outside them. One precision, not a correction: the DRM driver-private ioctl space starts atDRM_COMMAND_BASE 0x40(drm.h:1368), so the excluded range is 0x00–0x3F, wider than the "(0–9)" the text says. The conclusion is unaffected — 18 = 0x12 is below 0x40 — but the parenthetical understates the gap it is using.
And the honest limit is stated as a limit: the live probe found nothing in 8 s on those ten IOVAs and that is the right answer ("these buffers are idle now"), not a failed capture. That distinction — an instrument that can only see the present being used to say "not now" rather than "never" — is the thing this runbook exists to teach, and it is why the NPU run cost two measurements instead of twenty minutes.
Merge order, so the next reader is not surprised
This branch carries two unmerged PRs:
8801260a the runbook section ← this PR's own change (+34 doc lines)
c32ab030 serve: port taken -> relaunch ← engine#2541, open, READY
0bc632bb window-run: restart deferred ← engine#2547, open, READY
Two-dot against main shows all three (5 files, +158/−5); the runbook delta alone is docs/nvme-iommu-fault-triage.md +34. Merge #2541 and #2547 first, then this reduces to the doc change it is named for — otherwise this PR merges three things under a docs title.
Refs #2542: the runbook is a method, not an NVMe story
docs/nvme-iommu-fault-triage.mddocuments one device because that is where the stream was found. engine#2542 is the sameinstrument applied to a different device on the same host — the NPU at
0000:c6:00.1, domain0x0003— and it produceda diagnosis in two measurements without touching the device at all. That is the evidence the page needed to justify being a
method:
Four things generalise, and each was a decision rather than a discovery:
(device, domain)before counting — NVMe is0x0015, the NPU is0x0003, and a rawgrep -cinflateswhichever you were looking for (the mistake this lane already made once, in strixhalo: 2361 NVMe IO_PAGE_FAULT events on the root SSD in the boot that died at 23:47:35 #2478);
flags—0x0007is present=0, write=1, user=1, i.e. a write to a non-present page, which rules out apermission fault on live memory; the NVMe stream's
0x0000says the same for a read;descriptor list, not a scatter; the stride tells you which structure to look for;
Unsupported op 18is outside the ioctl ids (0–9) and the commandtypes (0–2) in the shipped UAPI, and the engine cannot name it either (its NPU path is
xrt::run, no opcode tablein-tree). Three candidates gone in one pass.
It also records what the pre-flight pattern bought on the NPU: the live probe on those ten IOVAs found nothing in 8 s,
which correctly means these buffers are idle now rather than no unmaps happen. Hours-old bursts are not observable
live — knowing that saved a twenty-minute capture that could only have said "idle".