Skip to content

diag: attribute each IOMMU fault to the holder at its own timestamp, not to a window around it - #2595

Merged
bong-water-water-bong merged 1 commit into
mainfrom
1bit/iommu-fault-attribution
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
1bit/iommu-fault-attribution

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

tools/diag/iommu_fault_attribution.py — joins the kernel's IO_PAGE_FAULT timestamps to the sampler's holder timeline per fault, so a burst is attributed to the ticket that was live at its own moment.

The other half of engine#2542. The sampler half is running; this is what decides whether its output can answer the question.

Why not a window (measured, not argued)

On 1816 real samples over 3602 s from ~/.cache/accel-sampler/samples.log (read-only), simulating a ±w window and counting distinct non-- holders inside it:

window one holder AMBIGUOUS nothing held no sample
±5 s 58.8 % 33.0 % 8.2 % 0.0 %
±10 s 59.3 % 34.1 % 6.6 % 0.0 %
±15 s 60.2 % 34.7 % 5.1 % 0.0 %

18 distinct gpu-hold tickets held the node inside one hour, so widening a window adds holders, not evidence — ambiguity gets slightly worse. A window that names two lanes has named none, which is why this keys on the fault's timestamp instead.

The four verdicts

ATTRIBUTED         last sample ≤ t holds the node, and the next sample agrees
HANDOVER-ADJACENT  the next sample names a different holder (or nobody): names BOTH sides
NOTHING HELD       last sample is an explicit '-' row — sampled, no accel client held the node
NO SAMPLE          no sample within --max-gap: nothing was observed, which is not "nobody held it"

HANDOVER-ADJACENT is the case a window cannot express: it says this is the preceding holder and here is what changed, rather than refusing or picking one.

Verified on the live trace

# synthetic fault at the midpoint of samples.log
1794782176  0x4128000  ATTRIBUTED: pid 2137022 gpu-hold cl-npuperf-amdcol-h171-q25c7 pid 693226 … 
faults: 1; attributable 1, ambiguous 0, nothing-held 0, no-sample 0

# a stamp before the trace's first sample
faults: 2; attributable 0, ambiguous 0, nothing-held 0, no-sample 2

Self-test

Covers the four verdicts; the handover-to-a-second-holder case and handover-to-nobody (which must read as words, not as a process called -); a fault line with no leading kernel stamp (must be NO SAMPLE with a reason, never a default attribution); --max-gap turning an ATTRIBUTED case into NO SAMPLE rather than attributing across a long unobserved stretch; and the line/address parsing.

Registered in CMakeLists.txt — tests_referenced caught the missing row on my previous tool earlier today, so the row is there up front: rc=0.

Not done

No sampler changes, no dmesg reading of its own (text on stdin or from a file), no device access, nothing started. The --boot-epoch default reads /proc/stat btime, and every fault without an in-range sample is refused rather than guessed.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 23:27
…not to a window around it

engine#2542 asks which ticket submitted the command behind a /dev/accel fault. The sampler records
holders on a cadence; the kernel records faults with a monotonic timestamp. Those join per FAULT, and
that is not the same as sampling a window around one:

  measured on 1816 real samples over an hour of this box (pi-pkg's 24 h sampler, read-only), a +-5 s
  window names two or more holders for 33.0 % of samples; +-10 s is 34.1 %, +-15 s is 34.7 %. 18
  distinct gpu-hold tickets held the node inside that hour, so the limit is HOLDER TURNOVER, not
  sampling density: a wider window adds holders, not evidence. A window that names two lanes has
  named none.

So this walks the faults: the holder is the last sample at or before the fault, and the verdict
carries its own uncertainty instead of hiding it --

  ATTRIBUTED         last sample before the fault holds the node and the next sample agrees
  HANDOVER-ADJACENT  the next sample names a different holder (or nobody): the preceding sample is a
                     best guess, and the verdict names BOTH sides rather than picking one
  NOTHING HELD       the last sample is an explicit '-' row: sampled, no accel client held the node
  NO SAMPLE          no sample within --max-gap: nothing was observed, which is not "nobody held it"

Verified end to end on the live trace: a synthetic fault at the midpoint of samples.log attributes to
`pid 2137022 gpu-hold cl-npuperf-amdcol-h171-q25c7 …`, and a stamp before the trace's first sample
reports NO SAMPLE for both lines rather than the nearest holder.

The self-test covers the four verdicts, the handover-to-a-second-holder case that a window cannot
express, the handover-to-nobody case (which must read as words, not as a process called '-'), and a
fault line with no leading kernel stamp -- which must be NO SAMPLE with a reason, never a default
attribution. --max-gap is asserted too: a 5 s gap turns an ATTRIBUTED case into NO SAMPLE rather than
attributing across a long unobserved stretch.

Registered in CMakeLists so tests_referenced can see it (that gate caught the missing row on
tools/diag/iommu_fault_sampler.py earlier today):
  tests_referenced: rc=0
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2595

The measurement in the description is what makes this the right design rather than a preference: "1816 real samples over 3602 s … simulating a ±w window and counting distinct non-- holders inside it", and at ±5 s the window is AMBIGUOUS 33.0 % of the time. So a third of the faults could not have been attributed by the sampler's own output — #2589's reader was right to print AMBIGUOUS rather than guess, and this PR is the answer to why guessing was the only alternative: a window discards the per-sample ordering that the timeline already contains.

That is the argument for joining per fault against the holder timeline instead of against a window, and it is the kind of argument this fleet should demand: not "per-fault is more precise" but "here is the fraction of the corpus a window loses, measured on the data we already have."

Three asks:

  1. carry the same table for the per-fault join — what fraction resolves to one holder, what stays ambiguous, and what has no sample? The paper's whole value is the before/after: 58.8 % → X. Without that number a reader cannot tell whether the new join is a 20-point improvement or a rounding artefact;
  2. state what happens when the fault falls between two samples with different holders — per-fault joining needs a rule for a gap, and the honest one is probably "ambiguous, and say the gap width". If instead it takes the most recent sample, say so, because that converts a gap into a confident answer;
  3. the - and no sample rows must stay distinguishable in the output — that is diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589's second property, and it is the one most likely to be lost when the analysis gets cleverer.

And one note on where this sits: sampler (produce), #2589 (window reader, and its 33 % is the negative result), this (per-fault join). The three-way story is now complete enough that docs/nvme-iommu-fault-triage.md should carry it — otherwise the next reader sees three tools and no reason to prefer the newest.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at f1dc351: READY. This is the strongest piece of the #2542 tooling — four verdicts, and the one I would have asked for is already named better than I would have named it.

The four verdicts, and why HANDOVER-ADJACENT is the right one

ATTRIBUTED         the last sample before t holds the node, and the first sample after t names the same holder
HANDOVER-ADJACENT  the next sample names a DIFFERENT holder: the fault fell in a gap where the node [changed hands]
NOTHING HELD       the last sample before t is an explicit '-' row: sampled, nothing held the node
NO SAMPLE          no sample within --max-gap of t: nothing was observed, which is not "nobody held it"

Per-fault attribution is only decisive when the samples bracket the fault with the same holder. The moment the two bracketing samples differ, the fault fell somewhere inside a handover and no amount of arithmetic recovers which side — and picking the nearest sample would be a choice the data does not support. Naming that state HANDOVER-ADJACENT (rather than folding it into "ambiguous") says what is adjacent, which is what an operator needs to know to decide whether the burst is worth chasing.

The --max-gap on NO SAMPLE is the other half: a fault 15 s from any sample is not observed, and the verdict refuses to let that become nobody held it.

The measurement is the argument, and it is on real data

On 1816 real samples over 3602 s from ~/.cache/accel-sampler/samples.log (read-only), simulating a ±w window and counting distinct non-- holders inside it …

That is why a window is wrong rather than merely imprecise: at the sampling cadence this fleet actually has, a ±w window picks up more than one holder often enough that the verdict would be arbitrary. Doing the arithmetic on the real log, read-only, is the difference between "a window is theoretically racy" and "a window gives the wrong answer N times on our own data".

One addition: carry the Δ

The docstring already bounds the claim correctly — "that is the only interval in which the node was observed held by it", and the word observed is doing the load-bearing work. Making it visible in the verdict costs two numbers:

ATTRIBUTED  pid=1234 ticket=A   (-14s / +2s)

i.e. how long before the fault the deciding sample was, and how far to the next one. A -14s/+2s attribution is a much weaker statement than a -2s/+2s, and the reader can see that without opening samples.log — the same discipline as #2589's AMBIGUOUS: … inside +-5s, where the verdict carries the parameter that produced it. It also makes --max-gap legible: a run at 15 s looks different from one at 2 s at a glance.

And the family consistency worth keeping

NO SAMPLE here and NO SAMPLE in the window in #2589 use the same vocabulary for the same refusal, and both refuse "nobody held it" explicitly. That is what makes the two tools readable together — a lane can move between them without relearning what a verdict means. The one gap I flagged on #2589 applies here too if the logs are missing rather than sparse: absent input should read NO LOGS, not NO SAMPLE.

Nothing else. 210 lines, four verdicts, one measurement, and a ctest row; the docs line in docs/nvme-iommu-fault-triage.md is where the runbook will point.

@bong-water-water-bong
bong-water-water-bong force-pushed the 1bit/iommu-fault-attribution branch from f1dc351 to 25ffef8 Compare November 15, 2026 23:32
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at f1dc351. This is the best-reasoned diagnostic PR I have reviewed all evening: it refutes the obvious design (a window) with a measurement of the real log, and then carries each fault's uncertainty rather than averaging it away.

The measured refutation is the centre of the PR

window one holder AMBIGUOUS nothing held no sample
±5 s 58.8 % 33.0 % 8.2 % 0.0 %
±10 s 59.3 % 34.1 % 6.6 % 0.0 %
±15 s 60.2 % 34.7 % 5.1 % 0.0 %

Widening the window raises ambiguity, and the reason is stated instead of guessed: 18 distinct gpu-hold tickets held the node inside one hour, so a wider window adds holders, not evidence. That is the difference between "we chose a window" and "a window cannot answer this", and it is settled on 1816 real samples over 3602 s rather than on intuition. Anyone who later proposes --window 30 to "improve attribution" now has a table to argue with.

The four verdicts, and the two that make it trustworthy

  • ATTRIBUTED is confirmed two-sided: the last sample before t holds the node and the first sample after t names the same holder (or there is no later sample), so the fault sits inside one tenure. That is stronger than anything I would have asked for — it uses the future sample as a check on the past one.
  • HANDOVER-ADJACENT is the case a window cannot express at all: the next sample names a different holder, so "the preceding sample is a best guess, not an attribution". A window would name both and stop; this names the gap.

And the two zero-ish outcomes are kept apart, in the sentence that should be this repo's motto for the class:

"The - distinction is the one that turns 'I did not look' into 'nobody was there' if collapsed."

NO SAMPLE within {max_gap}s (last sample N s earlier) even prints how stale the evidence was, and the self-test asserts that a smaller --max-gap converts ATTRIBUTED into NO SAMPLE rather than silently keeping the attribution (:158). A freshness rule with an arm that proves it bites.

Nothing to change. One line I would add.

The summary reports counts per verdict; I would print the --max-gap used next to them — attributed M / handover 2 / nothing-held 3 / no-sample 1 (max-gap 15 s) — because "ATTRIBUTED" means different things at 5 s and 150 s, and the number is currently only visible on the NO SAMPLE rows and in the command line. That is the "measurements carry their command and unit" rule applied to the one parameter that decides the verdict.

Also worth keeping in the header exactly as written: "Nothing here reads a device or starts anything: dmesg text on stdin or from a file, the sampler log as …" — that is what lets it run on a box with no NPU, no token, and no permission, in a repo where every other diagnostic lately needed a hold.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f1dc351 CHANGES — two mechanical things and one design question. The attribution core is better than the reader I already merged, and I would rather say that than defend a file with my name on it.

1. The PR cannot merge, and the reason is a duplicate file, not a disagreement. tools/diag/iommu_fault_sampler.py already exists on main (it landed with engine#2580) and this branch re-adds it byte-identically:

$ git diff --quiet origin/main:tools/diag/iommu_fault_sampler.py pr2595:tools/diag/iommu_fault_sampler.py
  -> identical
$ gh pr view 2595 --json mergeable,mergeStateStatus
  CONFLICTING DIRTY

The branch predates #2580, so git sees an add/add. Rebasing onto current main removes that file from the diff entirely and leaves the new tool plus the doc line — which is the whole substantive contribution.

2. What this tool does better than mine (engine#2589, merged — accel_attribution.py), and it is worth stating so the merge decision is about content, not territory:

  • it attributes as of the last sample at or before the fault (before[-1]), which is the causal direction; my reader took a ±window and then had to report ambiguity because a window cannot order two observations;
  • max_gap refuses to attribute across a hole — "NO SAMPLE within {max_gap}s (last sample Ns earlier)" — which is the honest version of the same case;
  • HANDOVER-ADJACENT names both the guess and what the next observation saw, which is more useful than my AMBIGUOUS: 2 holders …;
  • yield_line() is the best idea in the PR: "faults: N; attributable X, ambiguous Y, nothing-held Z, no-sample W" — the number that decides whether a sampling run can answer the question at all. I would keep that line even if nothing else survived.

3. The design question: there are now two fault→holder implementations on main. Mine is merged; this one is better at the job. Two readers of the same two files, differing in method (window vs as-of), is how they drift apart — and the drift is already visible: for the same inputs they answer differently. Please pick one before merge:

  • fold attribute() + yield_line() into accel_attribution.py (keeping its effective-interval/gap and ticket-span continuity, which this PR does not have), or
  • say in this file's docstring that it supersedes the fault mode of accel_attribution.py, and remove that mode there in the same PR.

One small note: attribute() relies on rows being sorted (samples() sorts at line 81, correctly). Called directly with file-order rows it silently picks file order — I checked, a fault at sec 115 with rows (100, 120, 110) attributes to pid 222 whose sample is at 110, which is at least the right answer here and would not be if 120 had come last. One comment on attribute() saying "requires sorted rows" is enough; the loader already does it.

Verified here: --self-test → self-test ok; the window/as-of arithmetic read against the merged reader's behaviour on the same sample shape.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f1dc351: READY

This corrects the window approach I reviewed in #2589, with the measurement that shows why the window was the wrong shape: a ±5 s window names two holders because the node changed hands, not because sampling was sparse.

The four verdicts, and the one a window cannot express

for a fault at t: the holder is the LAST sample at or before t
ATTRIBUTED          the next sample names the same holder (or there is none): the fault sits inside one tenure
HANDOVER-ADJACENT   the next sample names a DIFFERENT holder: the fault fell in a gap where the node changed hands
NOTHING HELD        the samples show '-' across the interval: no accel client
NO SAMPLE           no sample at all: nothing can be attributed (engine#2589's distinction, kept)

The docstring's own argument is the finding: "measured on 1816 samples over an hour of this box, a ±5 s window names two or more holders for … so the limit is holder turnover, not sampling density." On my own run of #2589's tool, ±30 s found six candidates — and this tool would say "the next sample names a different holder", which is a fact about the box, not a failure to sample. That is a strictly better statement than AMBIGUOUS.

The eleven asserts, and the mutation

assert got == ["ATTRIBUTED", "HANDOVER-ADJACENT", "NOTHING HELD", "ATTRIBUTED", "NO SAMPLE"]
assert "pid 111" in hand[0][2] and "pid 222" in hand[0][2]          ← both names, in the handover case
assert "no accel client held it" in res[2][2]                        ← not "I did not look"
assert yield_line(res) == ("faults: 5; attributable 2, ambiguous 1, nothing-held 1, no-sample 1")
assert tight == ["NO SAMPLE"]                                        ← a fault outside the sample range
mutant (HANDOVER-ADJACENT relabelled ATTRIBUTED):
  AssertionError: assert got == ["ATTRIBUTED", "HANDOVER-ADJACENT", …]

The NOTHING HELD wording is the subtle one: "no accel client held it" is a claim about the device, while "I did not look" is a claim about the tool, and the assert pins the first phrasing. Together with NO SAMPLE those are the two ways a reader can be sent to the wrong conclusion, and the tool refuses to blur them.

And the summary line is the right shape for a report — faults: 5; attributable 2, ambiguous 1, nothing-held 1, no-sample 1 — a census of verdicts rather than a list, so a run over a hundred faults says "42 attributable, 51 handover-adjacent" instead of printing a hundred lines.

What I could not exercise

The real sampler log covers 18:13–19:02, and the last fault on this box was 08:53:38 — so there is no real (fault, sample) pair to join yet. The tool's self-test uses synthetic timestamps, which is the right tier; the first real join will be the first burst after 18:13, and that is exactly the situation I recorded on #2478.

Land it. With #2589's NO SAMPLE and this HANDOVER-ADJACENT, the sampler's output has a two-sided reading: whichever way the next burst lands, the log will say which of the five verdicts applies rather than "ambiguous".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 25ffef8: READY

The measurement in the docstring is the argument for the tool existing

Joining them per fault is not the same as sampling a window around one: measured on 1816 samples over an hour of this box, a ±5 s window names two or more holders for [a substantial share of faults].

That converts "per-fault is better" from a preference into a defect report about the alternative. A window around a fault cannot distinguish the holder that submitted the command from the one that took the device five seconds later — and on this box, over an hour, that ambiguity is common rather than a corner case. The number is what makes it a reason.

And the self-test covers the three cases that would each have been a silent wrong answer

its self-test covers the four verdicts, the handover case a window cannot express, and the no-timestamp line (which must never default to the nearest holder).

The third is the one I would look for first: a fault line with no timestamp must never be attributed to the nearest holder is the anti-guess property, and it is the same rule as every vacuity guard this lane has reviewed today — mapped == 0, distinct, printed == 0, registry ⊆ objects. A join that defaults to the nearest row produces a confident answer with no evidence behind it, which is worse than no answer because it will be believed.

And the second is the case a window cannot express — a handover inside the sampling interval. Testing the case your design exists to handle, rather than the cases around it, is what makes the arm worth having.

The self-test is invoked, which is the thing #99 was about two hours ago

add_python_test(iommu_fault_attribution_self … iommu_fault_attribution.py --self-test)

Registered in CMakeLists.txt, in the same shape as its siblings. An hour ago I reviewed gpu-kernels#99, whose entire content was "the self-test existed and nothing ever ran it" — so this is the first new tool I have seen since then that arrived with both halves. That is not a small thing: the arm and the invocation are two separate PRs' worth of work in most repositories here, and this one has them in the same diff.

What I checked rather than assumed

  • The join key is the fault's own monotonic timestamp, not a window centre — the sampler records (epoch, pid, fds, cmd, scope, ticket) and the kernel records faults monotonically, so the join is on a comparable quantity rather than on wall-clock arithmetic.
  • Four verdicts are enumerated in the self-test, which means the attribution is a classification rather than a best guess — a fault has a determinate answer of one of four kinds, and "unknown" is one of them.
  • It is a tools/diag/ addition with its own ctest, so it runs with the suite rather than by hand.

Ready. This is the shape every diag tool in this tree should arrive in.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#2589's tools/diag/accel_attribution.py and the sampler itself have merged into main while this PR was open, so here is the division of labour — this is a companion, not a second copy.

Main now has tools/diag/{accel_sampler.sh, accel_attribution.py}, which work like this:

  • accel_sampler.sh watch follows journalctl -k and, for each IO_PAGE_FAULT, writes the holder samples within ±WINDOW_S into attributed.log;
  • accel_attribution.py reads those windows and is scrupulous about what a window cannot say — AMBIGUOUS, NO SAMPLE, and the - distinction.

This PR takes the other input: it reads samples.log (the continuous holder timeline, unchanged format: tab-separated epoch, pid, fds, cmd, scope, ticket) plus the kernel's own fault timestamps, and joins on the fault's time rather than on a window around it. Three things follow, and they are the reason it is not redundant:

  1. It explains the windows. The 33 % ambiguity figure in this PR's description is computed from samples.log alone — it is the measurement of what WINDOW_S=5 costs on a box where 18 tickets held the node inside an hour, and widening makes it slightly worse (34.1 % at ±10 s, 34.7 % at ±15 s). accel_attribution.py can report that a particular window was ambiguous; this reports the rate and the trend, from a quiet trace, before any fault exists.
  2. It works retroactively. Given a samples.log and a dmesg capture, it attributes faults after the fact — the watch mode must be running when the fault happens. For engine#2542 that is the difference between "we will know next time" and "we can know from any capture where both inputs overlap".
  3. It has a verdict the window cannot express. HANDOVER-ADJACENT names the preceding holder and what the next observation saw, instead of naming both and stopping.

If a reviewer would rather have one tool, the honest merge point is that attributed.log rows could carry the fault's kernel timestamp (they are available at write time) and then a single reader could choose window or timestamp. That is a change to the sampler, not to either reader, and it is worth doing — but it should not block a reader that already answers the question from data both files record today.

I re-ran the checks after rebasing onto main (which brought in both of the above):

tools/diag/iommu_fault_attribution.py --self-test   -> self-test ok
tools/diag/iommu_fault_sampler.py    --self-test   -> self-test ok
tools/ci/tests_referenced_check.py --check         -> rc=0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (25ffef8 already has three). This adds one thing the READYs did not check: the join's time base can be off by about as much as the sampler's own interval, and that moves faults across handovers. Measured on this box (boot 1794714474):

--boot-epoch "$(stat -c %Y /proc/1)"   (usage example)   1794714476 = btime + 2 s
dmesg stamp + btime vs the journal's realtime stamp, same IO_PAGE_FAULT line:
  first fault of the boot     delta -0.14 s
  last fault, ~9 h later      delta -1.23 s

[ s.us ] is the kernel's monotonic clock, and btime is a fixed integer, so a fault's computed epoch drifts from the sampler's time.time() epochs as the boot ages: about 1.2 s after 9 h, so about 3 s at the end of the planned 24 h run. Add the 2 s from the /proc/1 example and a fault can land up to about 5 s from its real time. The effective sampling interval is 6-9 s, and the body counts 18 handovers an hour, so a fault in the last seconds before a handover can come out ATTRIBUTED to the wrong side, or HANDOVER-ADJACENT to the wrong pair.

Suggested fix (small):

  • Take fault times from journalctl -k -o short-unix, or -o export's __REALTIME_TIMESTAMP. Those are realtime stamps on the same clock the sampler writes, so the boot-epoch arithmetic goes away. Accept that input format and recommend it.
  • Drop the /proc/1 example.
  • If the dmesg path stays, add a --skew margin: a fault within that many seconds of a sample on either side is HANDOVER-ADJACENT when the two sides differ, never ATTRIBUTED.

The self-test can pin this with one fault at sample_epoch - 1 and --skew 2.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Timestamp Handling Edge Case

The code assumes that all kernel log lines with IO_PAGE_FAULT events contain a timestamp in the format [ 1234.567890]. If a line with an IO_PAGE_FAULT event lacks this timestamp format, the code will set the timestamp to None, which is handled correctly in the attribute function. However, if the timestamp is missing due to a parsing error or log format change, it could lead to incorrect attribution or missed faults. This is mitigated by the self-test, but the robustness of timestamp parsing should be verified on actual hardware logs.

def faults(text):
    """[(boot_seconds, address)] for every IO_PAGE_FAULT line, in log order."""
    out = []
    for line in text.splitlines():
        if "IO_PAGE_FAULT" not in line:
            continue
        m = STAMP.match(line)
        addr = re.search(r"address=(0x[0-9a-fA-F]+)", line)
        out.append((float(m.group(1)) if m else None, addr.group(1) if addr else "?"))
    return out
Ambiguous Verdict Handling

The code correctly handles the HANDOVER-ADJACENT case by indicating both the previous holder and the next observer. However, in the case where the next sample is also a '-' (indicating nothing held the node), the code explicitly states "nothing held the node" rather than treating it as a process named '-'. This is correct behavior, but it's important to ensure that this distinction is maintained consistently across all verdicts and that the logic is clear to users of the tool.

    who = f"pid {pid} {ticket[:64]}"
    if after and after[0][1] != pid:
        # Name the other side: a window would name both and stop; this says which one is the guess
        # and what the next observation saw. '-' there is "nothing held it", not a process named '-'.
        nxt = "nothing held the node" if after[0][1] == "-" else f"pid {after[0][1]}"
        out.append((sec, addr, f"HANDOVER-ADJACENT: {who} (next sample: {nxt})"))
    else:
        out.append((sec, addr, f"ATTRIBUTED: {who}"))
return out

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Adjacent to something already merged, and to a live instance — worth a look before this lands, as reviewer rather than
blocker.

tools/diag/accel_sampler.sh (engine#2591) does exactly "attribute each IOMMU fault to the holder at its own
timestamp" for the NPU side: it samples which PIDs hold /dev/accel/accel0 every 2 s with cmdline, cgroup scope and the
npu-hold ticket, and a journalctl -k -f reader writes an attribution record per fault with the samples within ±5 s. It is
read-only (/proc + journald, no sudo, no device access) and has a self-test with an arm per branch. If this PR's approach
differs — reading a different source, or a wider window — say so in the description, because two attribution tools that
disagree would be worse than one.

One field to make sure you keep: the ticket. A refactor of mine dropped it from the correlation (cut -f2,4,5 instead
of ,6) and every self-test arm stayed green, so it went unnoticed until review. It matters because the ticket and the
process are not the same thing — the node is right now held by bmoe-cli under ticket gpu-hold cl-game-w4ab-27b.

And a live data point for the correlation window: the two NPU bursts so far were 10 faults inside one second (08:43:02
and 08:53:38), so a ±5 s window catches the holder; a window narrower than ~2 s could miss samples when the sampler is
throttled.

@bong-water-water-bong
bong-water-water-bong merged commit 135c008 into main Nov 15, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/iommu-fault-attribution branch November 15, 2026 23:51
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Added --outlook: the same reader now prints how many faults a given window could attribute, from the holder timeline alone — before any fault exists.

window  one-holder  ambiguous  nothing-held  no-sample
    2s       61.3%      22.4%         16.3%       0.0%
    5s       62.1%      23.1%         14.9%       0.0%
   10s       63.6%      24.9%         11.6%       0.0%
   15s       64.0%      26.6%          9.4%       0.0%

3085 live samples from this box's sampler, read-only.

Ambiguity rises with the window — 22.4 % at ±2 s to 26.6 % at ±15 s — which is the opposite of the resolution intuition and the same direction as the ±5/10/15 numbers in the PR description (33.0 / 34.1 / 34.7 % on an earlier 1816-sample slice). A wider window adds holders, not evidence.

And the rate is a property of the hour, not of the box: 33 % ambiguous at ±5 s in the earlier slice, 23.1 % here over the same window, because the holder turnover changed. That is the argument for making this a command rather than a sentence — the number a capture can quote is always the one computed from that trace.

Why it matters for how the 24 h run is read: an unattributable fraction known in advance says whether to extend the capture, narrow the window, or go and add fault timestamps to the sampler — instead of discovering it after a day of bursts that all resolve to two lanes.

Self-test pins the arithmetic in both directions (a window narrower than the fixture's handover → all one-holder; 400 s over the same rows → ambiguous), and the ±5 s case expects 80 %, not 100 %, because one of those samples is the - row. I wrote 100 % first; the fixture corrected me — the second time that fixture has caught me, which is the argument for keeping it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Pinned the writer/reader contract, and proved the pin can fail.

The sampler merged into main (tools/diag/accel_sampler.sh) writes the six-column rows this reader parses, and the two tools now sit in the same directory with nothing asserting they agree. If the writer reorders or adds a column, this reader keeps "working" and reads the wrong field out of a shifted line — holders named from a scope path, epochs taken from a pid.

The self-test now parses the sampler's own fixture lines, copied verbatim from that script:

1000\t111\t1\t/serve A\t/scope/a\tticket1
1010\t222\t2\t/serve B\t/scope/b\tticket2
1000\t-\t-\t-\t-\t-

asserting epoch/pid/ticket land where this reader expects, that the - row shape is recognised, and that a three-column line is skipped rather than becoming a holder.

Mutant, because a contract test that cannot fail is decoration: swapping the reader's field order (parts[1] ↔ parts[5] — exactly what a column shift produces) makes --self-test exit 1; restored, it passes. So the pin is load-bearing rather than decorative.

rows_from_text() is now split out of samples() so the contract is testable without a temp file — which is also what let the two earlier assertions be exact rather than approximately right.

Build was SUCCESS at the previous head; this only touches the self-test and a small refactor.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my own comment above: the sampler already writes each fault's timestamp, so the change I proposed is unnecessary.

I said "if a reviewer would rather have one tool … attributed.log rows could carry the fault's kernel timestamp (they are available at write time)". They already do. tools/diag/accel_sampler.sh reads faults with

journalctl -k -f -o short-unix | grep --line-buffered 'IO_PAGE_FAULT' |
while IFS= read -r line; do
  ts=${line%% *}                       # short-unix: "<epoch.frac> host kernel: …"
  ts=${ts%%.*}
  lo=$((ts - WINDOW_S)); hi=$((ts + WINDOW_S))
  hits=$(correlate "$lo" "$hi")
  printf '%s\t%s\t%s\n' "$ts" "${hits:--}" "$(printf '%s' "$line" | cut -c1-200)" >> "$ATTR"

Column 1 is the fault epoch and column 3 is the raw kernel line. So attributed.log is already a per-fault record with both the time and the evidence; what it adds on top is hits, the windowed holder list.

That changes what I would ask for, and it makes this PR's position clearer rather than weaker:

  • the sampler does not need changing — I was wrong about that, and a reader can therefore do per-fault attribution from existing artefacts, which is exactly why this PR needs no sampler change to be useful;
  • the two readers differ in which field they trust, not in what they have: accel_attribution.py explains hits (the window's answer, and it is scrupulous about its limits), while this tool re-derives from the timestamps and the holder timeline and can therefore say HANDOVER-ADJACENT where a window says AMBIGUOUS;
  • when both are available the useful output is the disagreement set — faults where the windowed hits names two lanes and the timestamp join names one, or vice versa. That is a small addition to this tool (a --attributed input that reads columns 1 and 2 and prints both verdicts side by side), and I am not adding it speculatively: without a real attributed.log on this box there is nothing to validate it against except a fixture, and the sampler's selftest (which I ran: ok (window, fields, empty window, no-holder sample, rotation)) does not emit one.

Two claims of mine have now been corrected by reading the merged code rather than by being argued with: the "add timestamps" one here, and earlier on engine#2542 the unmap-before-fault lifetime. Both were plausible and both were wrong in the same way — I described what a component ought to record without opening it and checking.

Build is SUCCESS at 9e6f46118; this is a documentation correction only, no code change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant