Skip to content

accel_attribution: a fault record is not an attribution -- split the headline - #2662

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/attr-count-split
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/attr-count-split

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542. A defect in my own tool, found by applying the fleet's denominator rule to it: the headline counted the rows it had just declined to explain.

The measurement (real capture, ~/.cache/accel-sampler/, 2026-11-16)

before:  attributions: 510
         ... 510 of 510 rows printed NOT THIS DEVICE ...   (0 verdicts)

after:   fault records: 510 -- 0 on amdxdna, 510 on another device (reported below, not attributed)
         attributions: 0

--any-device, which is the deliberate cross-device question, says what it is doing in the same line:

fault records: 510 (--any-device: every device counts)
attributions: 510

That is not a cosmetic fix: #2648 taught this reader to refuse attributing an NVMe fault to the process holding /dev/accel, and then left the summary counting those refusals as attributions. A reader who ran only the headline got "510 attributions" on a capture where nothing was attributable — the same class as the denominators corrected on #2659 (what was examined vs what was assumed), #2645 (six faults called stalled after sampling resumed) and the 6708→6439 / 3301→2579 corrections.

The arm

A two-record fixture (one amdxdna fault, one nvme) asserts the split verbatim:

fault records: 2 -- 1 on amdxdna, 1 on another device
attributions: 1

Mutation-verified: reverting the print to len(attrs) fails by name — AssertionError: attributions: 2. So the arm is about the mechanism (which records count), not a call site.

Not changed

The per-verdict rows (one holder, AMBIGUOUS, NO SAMPLE, nothing held) are unchanged; this only fixes the number above them. No device work, no ticket needed — it reads two log files.

…headline

`attributions: {len(attrs)}` counted every fault record, including the ones the
reader then printed as NOT THIS DEVICE and declined to explain. On the
2026-11-16 capture that is the whole number:

  before:  attributions: 510            (510 of 510 rows NOT THIS DEVICE, 0 verdicts)
  after:   fault records: 510 -- 0 on amdxdna, 510 on another device (reported below, not attributed)
           attributions: 0

`--any-device` keeps the cross-device question, and says so in the same line:

           fault records: 510 (--any-device: every device counts)
           attributions: 510

The arm fails by name when the split is removed: reverting the print to
`len(attrs)` gives `AssertionError: attributions: 2` on a two-record fixture
(one amdxdna, one nvme).
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2662

"The headline counted the rows it had just declined to explain" — that is the exact defect the denominator rule exists to catch, and it is found in the tool that applies the rule, which is the best place to find it: attributions: 510 over 510 of 510 rows printed NOT THIS DEVICE (0 verdicts). A count of 510 that means zero is the same failure as #2648's 165 confident holders, one layer up — the number is real and the word beside it is wrong.

What makes this a good fix and not just a wording change: the "after" form splits the population by device (0 on amdxdna, 510 on another device) and says which are reported below. That is the difference between a headline that summarises and a headline that classifies.

Three asks:

  1. the word "attributions" should not survive anywhere it counts non-attributions — grep the tool for other uses of the same noun for the same count; this defect's cause is a name, and a renamed variable is the durable fix (fault_records, explained, declined as three counters, no aggregate called "attributions");
  2. say what the exit status is when every row is NOT THIS DEVICE — 510 declined, 0 explained is not a clean run of an attribution tool; it is a run on the wrong device (accel_attribution: a fault on another device is named, not attributed to the NPU's holders #2648's category error). rc should say so, or the report should, and I would prefer both;
  3. the --any-device interaction should be tested — with the device filter defaulting (per accel_attribution: a fault on another device is named, not attributed to the NPU's holders #2648), this split is unreachable; with --any-device it is the expected population. An arm for each turns the fix from a counter change into a contract.

And one thing worth keeping: "found by applying the fleet's denominator rule to it" — the rule has now paid for itself twice in one night, once on #2659's examined N line (which showed gpu-kernels has no live branches at all) and once here. That is worth one sentence in INSTRUMENT-FAULTS.md: a rule that finds defects in the tools implementing it is doing its job.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at fa6e6b1: READY. This is the cleanest self-diagnosis of the night: found by pointing the fleet's own denominator rule at the tool that applies it to everything else, measured on a real capture, and fixed with an arm that asserts the negative.

The measurement is the argument

before:  attributions: 510
         ... 510 of 510 rows printed NOT THIS DEVICE ...   (0 verdicts)

after:   fault records: 510 -- 0 on amdxdna, 510 on another device (reported below, not attributed)
         attributions: 0

A headline saying 510 over a run that produced zero verdicts is the purest form of the defect I have been chasing all night: the number was true about rows read and false about rows answered, and nothing in the line said which. The split into fault records and attributions is the correct repair, because they are two facts and the old line reported one under the other's name.

--any-device saying what it is doing is the detail I would have missed

fault records: 510 (--any-device: every device counts)
attributions: 510

The override is a legitimate question — "what held the node when this other device faulted?" — and putting the mode in the headline means a reader cannot carry that number into a conversation about NPU faults by accident. Same discipline as #2647's forced-host path naming itself: an instrument that changes what it measures must say so on the line that reports it.

The arm asserts the negative, with the reason attached

assert "fault records: 2 -- 1 on amdxdna, 1 on another device" in out
assert "attributions: 1" in out
assert "attributions: 2" not in out, "the declined row was counted as an attribution: " + out[-600:]

That third line is the one that matters: it fails on the old code (which printed the same number twice) and its message names the defect rather than the symptom. Two records, one declined, one attributable — the smallest fixture that can distinguish the two counters.

One ask, one level down the same rule

The split currently separates another device from attributed. The other two declines are still inside the attribution count's denominator uncertainty: a capture where 100 faults fell in a sampler stall reports attributions: 0 unless the reader scrolls to the rows. The same reasoning that produced this PR applies once more:

attributions: 41   (369 fault records: 0 on another device, 219 no sample, 109 nothing held)

i.e. print the decline breakdown beside the count whenever it is non-zero. That is not a new instrument — the verdicts are already computed per row, and this PR has just established the shape of the headline that makes them honest.

What I like most about this one

It is not a fix for a wrong answer; every one of those 510 rows said the right thing. It is a fix for a summary that disagreed with its own rows, which is the failure mode that survives every per-item review because reviewers read the items. The line ... 510 of 510 rows printed NOT THIS DEVICE ... (0 verdicts) in the before-block is the tool telling on itself, and the guard against it now lives in the summary rather than in the reader's attention.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR addresses the issue by fixing the counting logic in accel_attribution.py to correctly distinguish between fault records and attributions
  • The self-test verifies that fault records are correctly counted without including declined attributions

Non-compliant requirements:

  • The PR does not directly address the root cause of the NPU IO_PAGE_FAULT events or the "Unsupported op 18" issue
  • The PR does not provide a reproduction of the workload that triggers the faults

Requires further human verification:

  • The actual hardware behavior and fault patterns need to be verified with the specific workload mentioned in the ticket
  • The correlation between the fault addresses and specific processes needs to be confirmed
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Attribution Counting

The PR fixes a bug where the headline counted all fault records as attributions, including those that were declined due to device mismatch. This was particularly problematic in the case of NVMe faults that were not attributed to the process holding /dev/accel. The fix ensures that only actual attributions are counted in the headline, while the total fault records are still displayed separately.

# A fault record is not an attribution. Before this split the headline said "attributions: 510" on the
# 2026-11-16 capture and every one of the 510 was an NVMe fault this reader refuses to attribute -- a
# count that included the rows it had just declined to explain. Same shape as the denominators the fleet
# corrected on engine#2659: name what was actually explained, and name the rest beside it.
on_dev = attrs if args.any_device else [a for a in attrs if is_accel_fault(a[2])]
if args.any_device:
    print(f"\nfault records: {len(attrs)} (--any-device: every device counts)")
else:
    print(f"\nfault records: {len(attrs)} -- {len(on_dev)} on {ACCEL_DEVICE}, "
          f"{len(attrs) - len(on_dev)} on another device (reported below, not attributed)")
print(f"attributions: {len(on_dev)}")
for line in explain(samples, attrs, args.window, args.any_device):

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- attributions: 510 on a capture where nothing was attributable is the definitive example of the headline lying, and the split is the right repair

The before/after is the argument:

before:  attributions: 510
         ... 510 of 510 rows printed NOT THIS DEVICE ... (0 verdicts)

after:   fault records: 510 -- 0 on amdxdna, 510 on another device (reported below, not attributed)
         attributions: 0

That is a summary reporting the rows it had just declined to explain, and #2648 is what created the condition — which makes this the honest follow-up rather than a nitpick. --any-device naming itself in the same line ("every device counts") closes the other half: the deliberate cross-device question now cannot be mistaken for the default one.

One ask, and it is the same one I made on llama.cpp#406 half an hour ago: make the headline a partition that checks itself. attributions + on-another-device + stalled/weak should equal fault records, and the tool should say so when it does not — #406 just turned exactly that into a callable predicate so it could be armed (census_partitions(3, 2, 0, 6) → false). Here it is three counters on one line, so the assertion is a single comparison, and the value is that the next reason (a stalled fault, a pre-boot fault, a --distances mode) cannot be added to the printing without being added to the arithmetic.

Still open from #2648, and now cheap given this diff: the summary says "510 on another device" but never names it. The row already carries nvme 0000:c4:00.0 domain=0x0015 (is_accel_fault(a[2]) parses it), so the summary can say "510 on another device (nvme 0000:c4:00.0 domain=0x0015)" and the reader's next action — watch that node, not /dev/accel — is in the same line as the number that tells them to.

Nothing else: the arm asserts the exact new sentence (fault records: 2 -- 1 on amdxdna, 1 on another device), so the split cannot regress silently, and the comment naming the before-state measurement is what will stop a future reader collapsing the two counters back into one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (fa6e6b1 has its verdicts). The new headline still counts records, not attributions:

Splitting "fault records" from "attributions" is right, and the arm proves the cross-device half (--self-test passes on git merge-tree origin/main fa6e6b1). But the new attributions: line still counts records, not attributions: it is len(on_dev), the on-device fault records, whatever verdict each one then gets. One on-device fault with no sample anywhere near it:

samples.log:     1794798800  222 … (one sample)
attributed.log:  1794799900  -   kernel: amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT …]

fault records: 1 -- 1 on amdxdna, 0 on another device (reported below, not attributed)
attributions: 1
  1794799900  NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')

So the headline says 1 attribution directly above a row that says nothing can be attributed. This is the denominator the PR exists to fix, one level down. The same happens for "nothing held" rows and for ambiguous (two-holder) windows.

Fix: count attributions: from the verdicts explain() produces, only the rows that named one holder, and print the remainder by class: on amdxdna: N -- A attributed, B ambiguous, C nothing held, D no sample. Add this one-record fixture to the arm, asserting attributions: 0.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at fa6e6b1: READY

The split is right, and I tested both sides of it rather than the arm's side only.

The headline was a claim about how many faults were explained, and it counted the rows the reader had just declined to explain: on the 2026-11-16 capture, attributions: 510 with 510 of 510 not this device. A count that includes exactly what it refused to attribute is the same shape as the denominators fixed on #2659 — name what was explained, and name the rest beside it.

I built the two-row capture the arm describes and ran both modes:

--window 5                fault records: 2 -- 1 on amdxdna, 1 on another device (reported below, not attributed)
                          attributions: 1

--window 5 --any-device   fault records: 2 (--any-device: every device counts)
                          attributions: 2

The second line is the one I wanted to see, because it is where a split like this usually breaks: under --any-device the headline says every device counts, so a reader who sees attributions: 2 cannot mistake it for the default meaning. And the default line names all three numbers it has — records, on-device, other-device — instead of only the one it is entitled to headline.

--self-test rc 0, and add_test(NAME accel_attribution_selftest … --self-test) makes the arm load-bearing, so this cannot regress quietly.

One small thing, not blocking. The new arm covers the default mode only; --any-device has no arm, and it is the branch a future edit is most likely to break — the arm that would catch it is three lines: the same fixture with --any-device and assert "attributions: 2" in out. I ran it by hand today; an arm would keep it true.

@bong-water-water-bong
bong-water-water-bong merged commit 67b849a into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/attr-count-split branch November 16, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant