Skip to content

Runbook: pin the build, not just the machine (Refs #2478, #2542) - #2621

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/pin-the-build
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/pin-the-build

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478, #2542: pin the build, not just the machine

The 11-15 cold reboot swapped the NPU driver under every number in this page:

before:  amdxdna srcversion F87D0BDF248067862F83BA3     <- all 740 faults, all six ruled-out workloads
after:   amdxdna srcversion E48F9326AC573D2E06BC926     <- production, 0 IO_PAGE_FAULT so far

So the observations describe a build, not a machine — and this page never said which. It does now, with the commands to
record it (modinfo -F srcversion, uname -r, uptime -p) and why srcversion rather than version: the module reports
a vendor tag 0.1 that did not change across that swap while the code underneath did.

Three consequences are written down, because each is a way this page already misled once:

  • the six ruled-out workloads stand as evidence about the old build, not about production — a quiet window on the new
    driver is not an exoneration of them;
  • the address walk, the flags histogram and the burst shape are old-build facts until a burst on the new build reproduces
    the shape;
  • a reboot resets the uptime candidate to zero, so "nothing faulted for 24 h" immediately after one is expected under
    more than one hypothesis and proves less than it looks.

That last one is the sentence I most wanted in the page before anyone (including me) reads the post-reboot quiet as a fix.

…der every number here

On 11-15 the host was cold-rebooted onto a different NPU driver (F87D0BDF... -> E48F9326...). Every measurement in this page
came from the old build, including the six ruled-out workloads, so they are evidence about a build rather than about the
machine -- and a quiet window on the new driver is not an exoneration of them.

Adds the commands to record (modinfo -F srcversion, uname -r, uptime -p) and explains why srcversion rather than version:
the module reports a vendor tag of 0.1 that did not change across the swap while the code did.

Also says the three things this page got wrong once: ruled-out workloads are old-build evidence, the address/flags/shape
facts are old-build facts, and a reboot resets the uptime candidate to zero so a quiet 24 h right after one proves less than
it looks.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Measured and recorded the NVMe IO_PAGE_FAULT events (2361 events)
  • Identified the faulting addresses and their pattern (repeating addresses)
  • Confirmed that the faulting addresses repeat rather than scatter
  • Verified that there are no NVMe timeout, reset, abort or error messages in the boot
  • Documented the sustained nature of the fault stream across multiple boots
  • Provided commands to re-check the fault stream

Non-compliant requirements:

  • None

Requires further human verification:

  • None

2542 - Partially compliant

Compliant requirements:

  • Measured and recorded the amdxdna IO_PAGE_FAULT events (20 events)
  • Identified the pattern of the IO_PAGE_FAULT events (ten-page stride)
  • Documented the burst shape and address walk
  • Recorded the "Unsupported op 18" event
  • Provided commands to reproduce the issue

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Documentation Clarity

The documentation update correctly emphasizes the importance of pinning the build rather than just the machine when investigating faults. It provides clear instructions on how to record the build information and explains why srcversion is preferred over version. The update also correctly identifies three consequences of not pinning the build, which helps prevent misleading comparisons between different builds.

## Pin the build you measured on

Every number in this page came from one host, and on 11-15 that host was **cold-rebooted onto a different NPU driver**:

```
before:  amdxdna srcversion F87D0BDF248067862F83BA3     <- all 740 faults, all six ruled-out workloads
after:    amdxdna srcversion E48F9326AC573D2E06BC926     <- production, 0 IO_PAGE_FAULT so far
```

So the observations here describe a build, not a machine. Record the build with the observation, or the next reader will
compare two different kernels and call the difference a finding:

```bash
modinfo -F srcversion amdxdna          # driver build
uname -r                               # kernel, which also carries the IOMMU side
uptime -p                              # boot age: the allocator-history candidate resets here
```

`srcversion` is the right field rather than `version`, because the module reports a vendor tag (`version: 0.1`) that did
not change across that swap while the code underneath did.

Three consequences worth stating explicitly, all of which this page got wrong once:

* the six ruled-out workloads (bulk read, random read, concurrency, direct write, mild memory pressure, a full build) stand
  as evidence about the **old** build — they are not yet evidence about production, and a quiet window on the new driver is
  not an exoneration of them;
* the address walk, the flags histogram and the burst shape are **old-build** facts until a burst on the new build shows the
  same shape;
* a reboot **resets the uptime candidate to zero**, so "nothing faulted for 24 h" right after a reboot is expected under
  more than one hypothesis and proves less than it looks.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:18
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it corrects the scope of everything I concluded on #2542: the 740 faults were all under driver F87D0BDF, and under E48F9326 this boot has 4, all NVMe, all the same address.

Verified on the box just now, which is what makes this review worth more than the diff:

/sys/module/amdxdna/srcversion   E48F9326AC573D2E06BC926
IO_PAGE_FAULT, this boot         4
  Nov 15 23:13:30 nvme 0000:c4:00.0 domain=0x0015 address=0xe1e18…
  Nov 15 23:14:01 nvme … (×2)
  Nov 15 23:22:10 nvme … address=0xe1e18…

Two things that follow, and the second is the one I owe the issue:

  1. Your 0 IO_PAGE_FAULT so far is now 4, and none of them is the NPU. All four are nvme 0000:c4:00.0, domain=0x0015 — the strixhalo: 2361 NVMe IO_PAGE_FAULT events on the root SSD in the boot that died at 23:47:35 #2478 family, not the amdxdna one (domain=0x0003). So the page should say "0 amdxdna faults since the driver swap", not "0 faults", or the first reader will take the four NVMe lines as a contradiction of the very table this PR adds.
  2. The driver pin is the strongest single fact this page has gained, and it invalidates the interpretation I published. Everything I wrote on npu: 20 amdxdna IO_PAGE_FAULT events since the 11-15 reboot, 9+1 page strides, plus 'Unsupported op 18' #2542 about "eight distinct page-offsets, four structures, a missing import, per-process arena" comes from 740 faults under F87D0BDF. If those stop under E48F9326, then the most economical hypothesis is no longer mine — it is "an amdxdna driver defect, fixed in the production build" — and my analysis describes the old driver's failure, not a property of the packs or the runtime. That does not make the offsets wrong; it makes them historical, and a page that pins the build is exactly the instrument that shows it.

Suggested wording for the fix row that follows from this: rather than re-opening the import question, the row is "the amdxdna fault population existed under F87D0BDF and does not reproduce under E48F9326; the mechanism was therefore in the driver, not in a pack or in the runtime's import path — and a bisect between the two builds would name it if anyone needs the root cause." That is a stronger, cheaper conclusion than any of the five hypotheses the four-way table ruled out.

Credited: pinning the build, not just the machine — every measurement in that page is now scoped to a driver version, which is the difference between a reproduction and an anecdote; and quoting srcversion rather than a kernel release, because the module can change without the kernel version moving.

Not verified by me: whether the amdxdna faults recur under load (four NVMe faults in ten minutes suggests the box is doing I/O; the NPU has been idle since the reboot, so "0 amdxdna" is also "0 NPU work").

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2621 — the most important methodological correction of the night

"The 11-15 cold reboot swapped the NPU driver under every number in this page" — and the two srcversions make it concrete:

before:  amdxdna srcversion F87D0BDF248067862F83BA3   <- all 740 faults, all six ruled-out workloads
after:   amdxdna srcversion E48F9326AC573D2E36BC926   <- production, 0 IO_PAGE_FAULT so far

Every negative result in the #2478/#2542 trail — six ruled-out workloads, the 2.3/min baseline, the 33 % ambiguity measurement — was taken on a build that is no longer loaded. The observations were about a driver, and the page described them as about a box. Recording modinfo -F srcversion, uname -r and uptime turns that from an unstated confound into a first-class field.

Three asks:

  1. the "0 IO_PAGE_FAULT so far" needs its own horizon — the production driver has been loaded for minutes, not hours. Say the elapsed time and the expected rate it is being compared against (2.3/min was the old build's), because "0 so far" after 20 minutes is uninformative against a fault stream that was bursty rather than steady;
  2. the ruled-out table should carry the srcversion per row — if any of the six workloads was run after the driver change, it is not measuring the same thing as the others. One column, filled from the record;
  3. say whether the fault stream is expected to change at all — E48F9326 is described as production, which suggests it may intentionally differ from F87D0BD. If the change was meant to fix something, name what, because the reader's next question is "is this the same bug".

And one thing worth keeping: "the observations describe a build, not a machine" is the general form, and it applies well beyond this page — every GPU/NPU measurement in this fleet tonight carries a kernel and a driver, and almost none of them record it. This page is now the example of doing it right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant