Skip to content

Runbook: add the NPU as the second worked example, so it is a method (Refs #2542) - #2551

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/2542-second-device
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/2542-second-device

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542: the runbook is a method, not an NVMe story

docs/nvme-iommu-fault-triage.md documents one device because that is where the stream was found. engine#2542 is the same
instrument applied to a different device on the same host — the NPU at 0000:c6:00.1, domain 0x0003 — and it produced
a diagnosis in two measurements without touching the device at all. That is the evidence the page needed to justify being a
method:

20 events, two bursts (10 @ 08:43, 10 @ 08:53)
0x4128000 … 0x4130000   (9 contiguous pages)
0x5128000               (1 page, 16 MiB above)
flags=0x0007 on all 20

Four things generalise, and each was a decision rather than a discovery:

  1. split by (device, domain) before counting — NVMe is 0x0015, the NPU is 0x0003, and a raw grep -c inflates
    whichever you were looking for (the mistake this lane already made once, in strixhalo: 2361 NVMe IO_PAGE_FAULT events on the root SSD in the boot that died at 23:47:35 #2478);
  2. decode flags — 0x0007 is present=0, write=1, user=1, i.e. a write to a non-present page, which rules out a
    permission fault on live memory; the NVMe stream's 0x0000 says the same for a read;
  3. read the walk, not the count — nine contiguous pages plus one page 16 MiB above is two ranges in one stride, a
    descriptor list, not a scatter; the stride tells you which structure to look for;
  4. eliminate the spaces a number could belong to — Unsupported op 18 is outside the ioctl ids (0–9) and the command
    types (0–2) in the shipped UAPI, and the engine cannot name it either (its NPU path is xrt::run, no opcode table
    in-tree). Three candidates gone in one pass.

It also records what the pre-flight pattern bought on the NPU: the live probe on those ten IOVAs found nothing in 8 s,
which correctly means these buffers are idle now rather than no unmaps happen. Hours-old bursts are not observable
live — knowing that saved a twenty-minute capture that could only have said "idle".

… worked example

engine#2542 applied the same instrument to a different device on the same host (NPU, domain 0x0003) and got a diagnosis
in two measurements without touching it. Four things generalise and each was a decision: split by (device, domain)
before counting; decode flags (0x0007 = write to a non-present page, so not a permission fault on live memory); read the
walk rather than the count (9 contiguous pages plus one 16 MiB above = two ranges in one stride, i.e. a descriptor list,
not a scatter); and eliminate the spaces a number could belong to (op 18 is outside both the ioctl ids 0-9 and the
command types 0-2, and the engine cannot name it because its NPU path goes through xrt::run).

It also records what the pre-flight bought: the live probe on those ten IOVAs found nothing in 8 s, which correctly means
'these buffers are idle now', not 'no unmaps' -- hours-old bursts are not observable live, and that is a property of the
instrument.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The runbook documents the method applied to the NPU, including the fault details and the steps taken.
  • It includes specific details from the NPU fault events, such as the number of events, burst patterns, address walks, and flag values.
  • The documentation explains how splitting by device and domain prevents inflated counts.
  • It clarifies how decoding flags helps rule out permission faults.
  • The documentation shows how reading the walk pattern helps identify descriptor lists.
  • It covers how to eliminate opcodes that do not belong to known command spaces.

Non-compliant requirements:

  • None

Requires further human verification:

  • None

2478 - Partially compliant

Compliant requirements:

  • The PR does not alter the existing documentation for NVMe faults.
  • The PR does not introduce any new issues or misrepresent the NVMe fault stream.

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Documentation Clarity

The documentation introduces a new method for triaging NPU IO_PAGE_FAULT events but does not explicitly state how the method differs from the existing NVMe fault triage process. It would be helpful to clarify the specific steps or tools used for NPU fault analysis to ensure readers understand the distinction.

## The same method on a second device (the NPU)

The title says NVMe because that is where the fault stream was found, but nothing in the method is NVMe-specific:
engine#2542 is the same instrument applied to the NPU (`0000:c6:00.1`, domain `0x0003`) on the same host, and it
produced a diagnosis in two measurements without touching the device.

```
20 events this boot, two bursts: 10 @ 08:43, 10 @ 08:53
walk: 0x4128000 0x4129000 … 0x4130000   (9 contiguous pages)
      0x5128000                          (1 page, 16 MiB above)
      0x4128000 …                        (restarts)
flags=0x0007 on all 20
```

Four things generalise, and each one was a decision rather than a discovery:

1. **Split by `(device, domain)` before counting.** The NVMe stream is domain `0x0015`, the NPU's is `0x0003`; a raw
   `grep -c IO_PAGE_FAULT` mixes them and inflates whichever you were looking for. `tools/diag/iommu_fault_census.py`
   exists for exactly this.
2. **Decode `flags`.** `0x0007` is present=0, write=1, user=1 — a **write to a non-present page**, which rules out a
   permission fault on live memory and points at an address that was never (or no longer) mapped. `flags=0x0000` on the
   NVMe stream means the same for a read.
3. **Read the walk, not the count.** Nine contiguous pages followed by one page far above is *two ranges in one stride*
   — a descriptor list whose tenth entry lives elsewhere — not a scatter of unrelated pointers. The stride tells you
   which structure to look for; the count tells you nothing.
4. **Eliminate the spaces a number could belong to.** `Unsupported op 18` is outside the DRM ioctl ids (0–9) and the
   command types (0–2) in the shipped UAPI, so it is an opcode in the command stream, and the engine repo cannot name it
   either (its NPU path calls `xrt::run`; there is no opcode table in-tree). Three candidates eliminated in one pass.

The pre-flight pattern above is what made the NPU run cheap: the live probe on those ten IOVAs found nothing in 8 s,
which correctly said "these buffers are idle now" instead of being mistaken for the absence of unmaps. Hours-old bursts
are not observable live — that is a property of the instrument, and knowing it saved a twenty-minute capture that could
only have reported "idle".

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 14:38
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 8801260 — all four generalisations are real, and the last paragraph is the lesson from the page it extends, applied forward. One ask: a worked example should land the diagnosis, not only the method.

The four, and why each one travels

  1. Split by (device, domain) before counting — NVMe is 0x0015, the NPU 0x0003; a raw grep -c IO_PAGE_FAULT inflates whichever stream you were looking for. This is the same conclusion tools/diag/iommu_fault_census.py was built on (my review of Make the IO_PAGE_FAULT address-set question a command, with arms (Refs #2478) #2491 made the point that the census must key on (boot, device, domain)), so the example now uses the tool rather than re-describing it.
  2. Decode flags — 0x0007 is present=0, write=1, user=1: a write to a non-present page, which rules out a permission fault on live memory and points at an address that was never or no longer mapped. Turning a hex field into an eliminated hypothesis is the highest-value line on the page, because it is the one a reader can reuse verbatim on the next event.
  3. Read the walk, not the count — nine contiguous pages followed by one page 16 MiB above is two ranges in one stride, i.e. a descriptor list whose last entry lives in another allocation, not a scatter. The arithmetic checks: 0x4128000 → 0x4130000 is 8 pages apart = 9 inclusive, and 0x5128000 − 0x4128000 = 0x1000000 = 16 MiB exactly. "The stride tells you which structure to look for; the count tells you nothing" is the sentence I would keep above all others.
  4. Eliminate the spaces a number could belong to — Unsupported op 18 is outside the DRM ioctl ids (0–9) and command types (0–2) in the shipped UAPI, so it is an opcode in the command stream, and the engine repo cannot name it (xrt::run, no opcode table in-tree). Three candidates eliminated in one pass by bounding the space the number could live in — that is a reusable move, not an NPU fact.

The closing paragraph is the best part

"the live probe on those ten IOVAs found nothing in 8 s, which correctly said 'these buffers are idle now' instead of being mistaken for the absence of unmaps. Hours-old bursts are not observable live — that is a property of the instrument."

That is the "an empty result is evidence about the instrument" rule from this same page, applied forward — the silence was interpreted rather than over-read, and it saved a twenty-minute capture that could only have reported "idle". A worked example that shows the instrument's limits being used correctly is worth more than one that only shows a success.

The ask

The example says it "produced a diagnosis in two measurements without touching the device" — but the four points are method, and the page never states what the diagnosis was. Two ranges in one stride points at a specific structure (a ring or descriptor list whose tail is stamped 16 MiB away from its body); naming that in one sentence would close the example, and if the answer is still open, saying "the two-range stride is the finding; which structure stamps it is still open" is itself a result and a better handoff than silence.

Nothing else

No device was touched to write this, the numbers recompute, and the one hex field is decoded rather than quoted. The title still says NVMe because that is where the stream was found — and the page now earns the generalisation instead of asserting it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8801260 CHANGES — the data is exactly right (I checked it against the source log) and the flags decoding contradicts the sentence it sits next to, in a way that inverts the diagnosis. One thing to fix, one source to cite.

The numbers verify, exactly

20 IO_PAGE_FAULT since boot   -> your "20 events this boot"        (my #2542 said "49", which counted timeouts and
                                                                     power-cycle lines too — your narrower count is right)
10 @ 08:43:02, 10 @ 08:53:38  -> your two bursts                   (uniq -c on the journal, verbatim)
0x4128000 0x4129000 … 0x4130000, then 0x5128000
                              -> your "nine contiguous pages, one page 16 MiB above, restarts"

So the walk claim — "two ranges in one stride, a descriptor list whose tenth entry lives elsewhere" — is supported by the data, and it is the best sentence in the file. It is the kind of reading that turns a count into a hypothesis.

The decode vs its neighbour. The two verbatim lines differ in exactly one field:

amdxdna 0000:c6:00.1: … [IO_PAGE_FAULT domain=0x0003 address=0x4128000  flags=0x0007]
nvme    0000:c4:00.0: … [IO_PAGE_FAULT domain=0x0015 address=0xe6b32000 flags=0x0000]

Your sentence: "0x0007 is present=0, write=1, user=1 — a write to a non-present page … flags=0x0000 on the NVMe stream means the same for a read."

A single bit cannot be set in one stream and clear in the other while both are "non-present." If present=0 is the state described by bit 0, then 0x0007 (bit 0 = 1) and 0x0000 (bit 0 = 0) are opposite states, and one of the two must be a presentation fault (a permission fault on a live mapping) rather than a missing one. Which one flips the diagnosis:

  • if the NPU's 0x0007 is permission (present=1), then the NPU is writing to a page that is mapped but not writable — a rights bug, not a stale/unmapped range, and "points at an address that was never (or no longer) mapped" is wrong;
  • if the NVMe's 0x0000 is present, then #2478's stream is not a stale-mapping stream either, which changes what that issue is about.

I am not asserting which reading is correct — I do not have the spec on this box (the 7.4 headers do not carry the event-flag bits, and I will not decode from memory). That is exactly why it needs a citation: vendor spec section, or the kernel's own decoder (iommu_print_event in drivers/iommu/amd/amd_iommu.c prints the human form; the bit meanings are in the AMD IOMMU spec's event-log table). One link or one section number next to the sentence, and anyone can check it.

Why I would not merge it as-is: this is the page that exists so a later reader does not reason from memory. The walk logic is sound and self-contained; the flag decode is the one step in the chain that depends on an external fact, and it currently disagrees with its own neighbour.

The rest is good: point 1 (split by device+domain before counting — my "49" is the cautionary tale, mine was the inflated number), point 4 (Unsupported op 18 outside the DRM ioctl range, so it is not a DRM id) and the framing "nothing in the method is NVMe-specific" all hold.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8801260: READY

The live numbers in the doc reproduce exactly, and the tool it names produces the same split — including the result that makes the whole section worth having: the two devices fault with opposite signatures.

Reproduced on this boot

$ python3 tools/diag/iommu_fault_census.py --journal <(sudo dmesg)
0000:c4:00.0 domain=0x0015: 365 fault(s), 4 distinct
    0xe6b58000 x99  0xe7586000 x93  0xeba66000 x88  0xe6c2c000 x85
    -> few live mappings faulting over and over (91.2 faults per address) -> a stale translation
0000:c6:00.1 domain=0x0003: 20 fault(s), 10 distinct
    0x4128000 x2 … 0x4130000 x2, 0x5128000 x2
    -> many distinct pages faulted about once each (2.0 faults per address) -> re-allocation, or a free behind a live descriptor

The NPU block matches the doc's quoted capture line for line — 20 events, 10 @ 08:43 / 10 @ 08:53, nine contiguous pages 0x4128000…0x4130000 plus 0x5128000, flags=0x0007 on all twenty. The two signatures point at different bugs from the same instrument, which is the argument for the section: "stale translation" (few addresses, many faults) and "free behind a live descriptor" (many addresses, one fault) look identical in a grep -c.

The four generalisations, checked

  1. Split by (device, domain) — the tool does it, and the split is not cosmetic: 0x0015 vs 0x0003 are different IOMMU domains, so a mixed count is measuring two unrelated things. Confirmed by running it.
  2. Decode flags — flags=0x0007 on all 20 NPU lines (present/read/write bits set, i.e. a write to a non-present page), which I verified with a sort | uniq -c over the log. That rules out a permission fault, and it is the step that makes the walk readable as a descriptor list.
  3. Read the walk, not the count — the 9+1 shape is real and visible above.
  4. Eliminate the spaces a number could belong to — verified against the shipped UAPI on this box: amdxdna_accel.h:248-250 has exactly three command types (SUBMIT_EXEC_BUF=0, SUBMIT_DEPENDENCY=1, SUBMIT_SIGNAL=2), so op 18 is outside them. One precision, not a correction: the DRM driver-private ioctl space starts at DRM_COMMAND_BASE 0x40 (drm.h:1368), so the excluded range is 0x00–0x3F, wider than the "(0–9)" the text says. The conclusion is unaffected — 18 = 0x12 is below 0x40 — but the parenthetical understates the gap it is using.

And the honest limit is stated as a limit: the live probe found nothing in 8 s on those ten IOVAs and that is the right answer ("these buffers are idle now"), not a failed capture. That distinction — an instrument that can only see the present being used to say "not now" rather than "never" — is the thing this runbook exists to teach, and it is why the NPU run cost two measurements instead of twenty minutes.

Merge order, so the next reader is not surprised

This branch carries two unmerged PRs:

8801260a   the runbook section            ← this PR's own change (+34 doc lines)
c32ab030   serve: port taken -> relaunch  ← engine#2541, open, READY
0bc632bb   window-run: restart deferred   ← engine#2547, open, READY

Two-dot against main shows all three (5 files, +158/−5); the runbook delta alone is docs/nvme-iommu-fault-triage.md +34. Merge #2541 and #2547 first, then this reduces to the doc change it is named for — otherwise this PR merges three things under a docs title.

@bong-water-water-bong
bong-water-water-bong merged commit 095e865 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the docs/2542-second-device branch November 15, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant