Skip to content

accel_attribution: a fault on another device is named, not attributed to the NPU's holders - #2648

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/accel-reader-device-filter
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/accel-reader-device-filter

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The bug, in my own reader

tools/diag/accel_attribution.py attributes each IO_PAGE_FAULT to whoever held /dev/accel/accel0 at the time. The sampler watches one node; a fault from another PCI function says nothing about its holders — and this reader attributed them anyway.

Measured on the live capture on this box, 2026-11-16:

372 fault records, 372 of them "nvme 0000:c4:00.0 domain=0x0015"
with --any-device (i.e. the old behaviour): 165 get a confident holder

Every one of those 165 named the process that happened to hold the NPU — a category error, not a weak answer. It is the same correction engine#2623 made to the sibling reader, which I reviewed and reported there while this file kept producing it.

The change

default       : 372  "NOT THIS DEVICE: the fault is not amdxdna's, so the holders cannot explain it"
--any-device  : 165 attributed (the cross-device question, asked deliberately)

The filter is a named constant and one predicate (ACCEL_DEVICE, is_accel_fault), the default is the honest one, and the override exists so the cross-device read stays possible — the same shape as #2623's any_device.

The fixtures had to change for this to be testable

Every fault text in the self-test was a bare "A" / "B" / "C". None of them named a device, so none of them could see this defect — a fixture that cannot express the failure cannot catch it. They now carry a real line:

amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0x4128000]

which is what the product emits. With that, the new arm asserts both directions: a foreign-device fault yields NOT THIS DEVICE and no one holder line, and any_device=True restores the attribution.

Verified

  • --self-test ok (the whole existing suite plus the new arm).
  • Live capture: 372 records → 372 NOT THIS DEVICE by default, 165 attributed under --any-device.
  • Two traps hit while writing it, both now structurally avoided: the module had no tempfile import, and a function-local import tempfile further down in self_test() made tempfile local to the whole function (UnboundLocalError at the first use). The local import is gone.

No build, no device: pure Python, ~1 s.

… to the NPU's holders

The sampler watches ONE node (/dev/accel/accel0, amdxdna). A fault from another PCI function says nothing about who
holds it, and until now this reader attributed every IO_PAGE_FAULT line to the accel holders regardless.

Measured on this box, 2026-11-16: 372 fault records in the live capture, 372 of them 'nvme 0000:c4:00.0
domain=0x0015', and under --any-device 165 of them get a confident holder -- the process that happened to hold the
NPU. That is the category error engine#2623 fixed in the sibling reader, and I had reported it there while this file
kept producing it.

  default        : 372 'NOT THIS DEVICE: the fault is not amdxdna's, so the holders cannot explain it'
  --any-device   : 165 attributed (the cross-device question, asked deliberately)

The fixtures had to change for the fix to be testable: every fault text in the self-test was a bare 'A'/'B'/'C', so
none of them named a device and none of them could see this. They now carry a real 'amdxdna 0000:c6:00.1: ...' line,
which is what the product always emits.

Two traps hit while writing it, both kept in the code's shape rather than in prose: a function-local
'import tempfile' further down made tempfile local to the whole function (UnboundLocalError at the first use), and
the module had no tempfile import at all.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Device Attribution Logic

The change introduces a check to ensure that only faults belonging to the designated device (ACCEL_DEVICE) are attributed to process holders. Previously, faults from other PCI functions were incorrectly attributed to the NPU's holders, which was a category error. The new logic correctly identifies such cross-device faults and marks them as "NOT THIS DEVICE", preventing incorrect attribution.

if not any_device and not is_accel_fault(fault):
    lines.append(f"  {ts}  NOT THIS DEVICE: the fault is not {ACCEL_DEVICE}'s, so the holders cannot explain it")
    lines.append(f"      {fault[:110]}")
    continue
Test Coverage for Cross-Device Faults

The self-test has been updated to include fixture data that simulates real fault lines with device identifiers. This ensures that the new logic for detecting and handling cross-device faults is properly tested, preventing regression of the previously identified category error.

# engine#2542: a fault on another PCI function must be NAMED, not attributed to the NPU's holders. Measured on
# this box: 189 of 189 records in one capture were `nvme 0000:c4:00.0 domain=0x0015` and every one was given the
# process holding /dev/accel -- a category error that reads as a confident answer.
foreign = ("1794798810\t215633\t/serve\t-\tscope\tgpu-hold cl-a\t"
           "[  321.308031] strixhalo kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged "
           "[IO_PAGE_FAULT domain=0x0015 address=0xe4240000 flags=0x0000]\n")
ppath = tempfile.NamedTemporaryFile("w", suffix=".log", delete=False)
ppath.write(foreign); ppath.close()
try:
    at = read_attributions(ppath.name)
    assert len(at) == 1, at
    out = "\n".join(explain(samples, at, 5))
    assert "NOT THIS DEVICE" in out, out
    assert "one holder" not in out, out
    # ... and the override still lets a reader ask the cross-device question deliberately
    out2 = "\n".join(explain(samples, at, 5, any_device=True))
    assert "NOT THIS DEVICE" not in out2, out2
finally:
    os.unlink(ppath.name)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and this is my defect, in a tool I wrote, reported exactly the way I asked for it on #2623

Same error I flagged in #2623, one file over, and it was live the whole time. accel_attribution.py attributes every IO_PAGE_FAULT to whoever held /dev/accel/accel0; a fault from another PCI function cannot be theirs. The measurement makes the size of it unambiguous:

372 fault records, 372 of them "nvme 0000:c4:00.0 domain=0x0015"
with --any-device: 165 get a confident holder -- every one naming the NPU's holder

165 confident wrong answers out of 372 records. "A category error, not a weak answer" is exactly the distinction, and the fixture note -- "the fault text names the device, as every real line does -- the fixtures used to carry a bare A/B" -- is the same lesson as #2623: the fixture agreed with the author rather than with the input. That the old fixtures made this invisible is worth being in the PR body, because it explains why the arms passed.

Three asks:

  1. Name the other device in the line. NOT THIS DEVICE: the fault is not amdxdna's tells a reader what it is not; NOT THIS DEVICE: the fault is nvme 0000:c4:00.0 (domain=0x0015), not amdxdna's tells them what to watch instead, and the next action after reading this is one grep away. The information is already parsed (ACCEL_DEVICE in line implies the device is available) -- printing it costs nothing.
  2. Say what rc means when nothing is attributable. If every fault is another device's, a clean exit reads as "the NPU is fine" when it means "this capture says nothing about the NPU". Print the counterpart of the existing summary -- 0 of 372 faults are amdxdna's; nothing in this capture is evidence about the NPU -- and state whether rc changes. A run that cannot answer should say so in its summary, not only per line.
  3. --any-device reproduces the category error on demand, so it should say so when used. Either rename it (--attribute-any-device, which at least reads as the mistake it is) or print one line when it is set: "--any-device: attributing faults from other devices; the holders are the NPU's alone". As it stands, a reader who finds the flag will assume it is the thorough option.

Nothing else: making the default strict and the escape hatch explicit is the right direction, and the same correction landing twice in two readers is a good argument for the census to state the device on every row.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2648

"A fault from another PCI function says nothing about its holders — and this reader attributed them anyway" is a category error, and the measurement makes its size exact: of 372 fault records, all 372 are nvme 0000:c4:00.0 domain=0x0015, and under the old behaviour 165 got a confident holder — every one of which named the process holding the NPU. 165 wrong attributions with no marker on them.

This is the most expensive class in the whole sampler family, because every other defect I have reviewed tonight was a degraded answer (ambiguous, unparseable, gap) and this one is a confident answer about the wrong device. #2624's 28 "no holder" records at least said they did not know.

Three asks:

  1. the device filter must default to the sampler's own node, not to "any" — a reader that can attribute across devices is a reader that will, and the safe default is the node the sampler watched. --any-device should be the flag that warns in the output (attributing across devices: holders are not evidence), not a mode that looks like the old one;
  2. the count belongs in the page and the output: "372 records, 0 attributable (device mismatch: sampler watches accel0, faults are nvme 0000:c4:00.0)" — because the useful fact here is that the census has been reading a device the sampler never watched, which is a statement about the whole ~/.cache/accel-sampler/ corpus, not just this run;
  3. say whether any earlier conclusion depended on the 165 — diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589's AMBIGUOUS counts, diag: attribute each IOMMU fault to the holder at its own timestamp, not to a window around it #2595's 33 %, Accel sampler: record gaps, so a stall is data rather than 'no holder' (Refs #2478, #2542) #2624's breakdown, Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) #2627's distance distribution. If those were computed on nvme faults attributed to NPU holders, they are all reframed: the sampler is fine, and the corpus was never about the device it claims.

And one note: this is the second time tonight a reader turned out to be answering about an entity different from the one it names (#2645's GAP rows describing the interval before their own timestamp). Both were found by reading the data rather than the code, and both would have been invisible to any test built from the reader's own assumptions.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2ce1a40: READY. The measured instance is the strongest justification for a verdict I have read tonight: 189 of 189, every one a category error.

The number is the argument

"Measured 2026-11-16 on this box: 189 of 189 fault records in one capture were nvme 0000:c4:00.0 domain=0x0015, and every one was attributed to whatever held the NPU at the time — a category error, not a weak answer."

The sampler watches one device node (/dev/accel/accel0, the amdxdna NPU), so a fault on another PCI function carries no information about its holders. And in this capture it was not a minority of records that were wrong — all of them were, which means the whole capture's attribution table was describing the SSD while naming the NPU's holders. That is not a confidence problem to be softened; it is a different question being answered, and NOT THE ACCEL NODE is the right verdict name.

is_accel_fault(line) = ACCEL_DEVICE in line with ACCEL_DEVICE = "amdxdna" is correct for the line shape the census parses (amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT …] versus nvme 0000:c4:00.0: …), and it correctly excludes an amdgpu fault as well — the GPU is not the NPU, and neither is the SSD.

The one thing I would add, and it is the same trap this family keeps falling into

The sentinel is a driver name, so it is narrower than the class "faults belonging to the accel node". If a future kernel or capture format prints the faulting function as a bare BDF (or renames the driver), is_accel_fault returns False for everything — and the tool would then report every fault as NOT THE ACCEL NODE with complete confidence. That is the mirror of tonight's other defects: not "nothing was checked", but "everything was checked and nothing qualified".

One line closes it, and the family already has the pattern (#2640's --distances, #2610's "no IO_PAGE_FAULT in this window"): if a capture has faults and none of them matches the accel device, say so explicitly — "0 of 189 fault lines name amdxdna; if this capture should contain NPU faults, the device match is wrong". Then a format change produces a named anomaly rather than a table of confident non-answers.

Nothing else

The header sentence — "a fault belonging to another PCI function says nothing about its holders" — is the whole rule in one line, and it is the same correction #2623 made to the sibling reader, which is why the two tools now agree about what a fault belongs to.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2ce1a40: READY. I checked the predicate against every boot this box has rather than only the capture it was written from, and it selects the right lines — including the direction that would have been fatal if it were wrong.

The bug was real and the capture was unambiguous

189 of 189 records in that capture being nvme 0000:c4:00.0 and every one attributed to the NPU's holder is not a weak answer, it is the wrong question answered confidently — the same correction #2623 made to the sibling reader, and worth making in both.

The predicate, tested in both directions

ACCEL_DEVICE = "amdxdna"
def is_accel_fault(line): return ACCEL_DEVICE in line

The failure I was looking for is a false negative — an NPU fault that does not carry the driver prefix would be reported NOT THIS DEVICE and the reader would discard the very faults it exists for. So I went and looked for that line shape:

boot -1:  740 IO_PAGE_FAULT lines,  20 mention amdxdna
          amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0x4128000 flags=0x0007]
boot -2: 2361 IO_PAGE_FAULT lines,   0 mention amdxdna
boot  0:  374 IO_PAGE_FAULT lines,   0 mention amdxdna      (all nvme 0000:c4:00.0)

faults on the NPU's BDF (c6:00.1) WITHOUT the amdxdna prefix, any boot:  none
faults naming the accel device as `device=…c6…`:                          none

So the kernel binds the driver name into every NPU fault line — amdxdna <BDF>: AMD-Vi: Event logged [IO_PAGE_FAULT …] — and the predicate picks 20 of 740 in the boot that had them while excluding all 720 nvme ones. Correct in both directions on real data, which is more than the capture alone could show.

Two things worth keeping in the comment, because they are what a future reader will doubt:

  1. boot -2 has 2361 fault lines and zero amdxdna, boot 0 has 374 and zero: an all-nvme boot is the normal case on this box, so NOT THIS DEVICE is not an exotic branch — it was, on two of the five boots I looked at, the entire output. A reader who assumes otherwise will read a wall of them as a broken reader.
  2. The predicate keys on the driver name, not the BDF. That is the right choice while the kernel prefixes faults (measured: always), and the wrong one the day a fault arrives through a path that names only device=0000:c6:00.1 — AMD-Vi prints faults that way in some configurations. A belt-and-braces predicate would accept either ("amdxdna" in line or ACCEL_BDF in line), and it costs one or. Not a block: today the data says the driver form is the only one this box produces.

The any_device parameter is the right escape hatch for the same reason #2640 has one: a research question about all faults needs the attribution, and an ordinary report needs the category error removed. Naming the device instead of withholding the block is also better than dropping it — the reader learns why those 189 records cannot be explained.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant