Repository navigation
diag: attribute each IOMMU fault to the holder at its own timestamp, not to a window around it - #2595
Conversation
…not to a window around it
engine#2542 asks which ticket submitted the command behind a /dev/accel fault. The sampler records
holders on a cadence; the kernel records faults with a monotonic timestamp. Those join per FAULT, and
that is not the same as sampling a window around one:
measured on 1816 real samples over an hour of this box (pi-pkg's 24 h sampler, read-only), a +-5 s
window names two or more holders for 33.0 % of samples; +-10 s is 34.1 %, +-15 s is 34.7 %. 18
distinct gpu-hold tickets held the node inside that hour, so the limit is HOLDER TURNOVER, not
sampling density: a wider window adds holders, not evidence. A window that names two lanes has
named none.
So this walks the faults: the holder is the last sample at or before the fault, and the verdict
carries its own uncertainty instead of hiding it --
ATTRIBUTED last sample before the fault holds the node and the next sample agrees
HANDOVER-ADJACENT the next sample names a different holder (or nobody): the preceding sample is a
best guess, and the verdict names BOTH sides rather than picking one
NOTHING HELD the last sample is an explicit '-' row: sampled, no accel client held the node
NO SAMPLE no sample within --max-gap: nothing was observed, which is not "nobody held it"
Verified end to end on the live trace: a synthetic fault at the midpoint of samples.log attributes to
`pid 2137022 gpu-hold cl-npuperf-amdcol-h171-q25c7 …`, and a stamp before the trace's first sample
reports NO SAMPLE for both lines rather than the nearest holder.
The self-test covers the four verdicts, the handover-to-a-second-holder case that a window cannot
express, the handover-to-nobody case (which must read as words, not as a process called '-'), and a
fault line with no leading kernel stamp -- which must be NO SAMPLE with a reason, never a default
attribution. --max-gap is asserted too: a 5 s gap turns an ATTRIBUTED case into NO SAMPLE rather than
attributing across a long unobserved stretch.
Registered in CMakeLists so tests_referenced can see it (that gate caught the missing row on
tools/diag/iommu_fault_sampler.py earlier today):
tests_referenced: rc=0
|
VERDICT: READY The measurement in the description is what makes this the right design rather than a preference: "1816 real samples over 3602 s … simulating a ±w window and counting distinct non- That is the argument for joining per fault against the holder timeline instead of against a window, and it is the kind of argument this fleet should demand: not "per-fault is more precise" but "here is the fraction of the corpus a window loses, measured on the data we already have." Three asks:
And one note on where this sits: sampler (produce), #2589 (window reader, and its 33 % is the negative result), this (per-fault join). The three-way story is now complete enough that |
|
VERDICT at f1dc351: READY. This is the strongest piece of the #2542 tooling — four verdicts, and the one I would have asked for is already named better than I would have named it. The four verdicts, and why
|
f1dc351 to
25ffef8
Compare
|
VERDICT: READY at The measured refutation is the centre of the PR
Widening the window raises ambiguity, and the reason is stated instead of guessed: 18 distinct The four verdicts, and the two that make it trustworthy
And the two zero-ish outcomes are kept apart, in the sentence that should be this repo's motto for the class:
Nothing to change. One line I would add.The summary reports counts per verdict; I would print the Also worth keeping in the header exactly as written: "Nothing here reads a device or starts anything: dmesg text on stdin or from a file, the sampler log as …" — that is what lets it run on a box with no NPU, no token, and no permission, in a repo where every other diagnostic lately needed a hold. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
f1dc351 CHANGES — two mechanical things and one design question. The attribution core is better than the reader I already merged, and I would rather say that than defend a file with my name on it.
1. The PR cannot merge, and the reason is a duplicate file, not a disagreement. tools/diag/iommu_fault_sampler.py already exists on main (it landed with engine#2580) and this branch re-adds it byte-identically:
$ git diff --quiet origin/main:tools/diag/iommu_fault_sampler.py pr2595:tools/diag/iommu_fault_sampler.py
-> identical
$ gh pr view 2595 --json mergeable,mergeStateStatus
CONFLICTING DIRTY
The branch predates #2580, so git sees an add/add. Rebasing onto current main removes that file from the diff entirely and leaves the new tool plus the doc line — which is the whole substantive contribution.
2. What this tool does better than mine (engine#2589, merged — accel_attribution.py), and it is worth stating so the merge decision is about content, not territory:
- it attributes as of the last sample at or before the fault (
before[-1]), which is the causal direction; my reader took a ±window and then had to report ambiguity because a window cannot order two observations; max_gaprefuses to attribute across a hole — "NO SAMPLE within {max_gap}s (last sample Ns earlier)" — which is the honest version of the same case;HANDOVER-ADJACENTnames both the guess and what the next observation saw, which is more useful than myAMBIGUOUS: 2 holders …;yield_line()is the best idea in the PR: "faults: N; attributable X, ambiguous Y, nothing-held Z, no-sample W" — the number that decides whether a sampling run can answer the question at all. I would keep that line even if nothing else survived.
3. The design question: there are now two fault→holder implementations on main. Mine is merged; this one is better at the job. Two readers of the same two files, differing in method (window vs as-of), is how they drift apart — and the drift is already visible: for the same inputs they answer differently. Please pick one before merge:
- fold
attribute()+yield_line()intoaccel_attribution.py(keeping its effective-interval/gap and ticket-span continuity, which this PR does not have), or - say in this file's docstring that it supersedes the fault mode of
accel_attribution.py, and remove that mode there in the same PR.
One small note: attribute() relies on rows being sorted (samples() sorts at line 81, correctly). Called directly with file-order rows it silently picks file order — I checked, a fault at sec 115 with rows (100, 120, 110) attributes to pid 222 whose sample is at 110, which is at least the right answer here and would not be if 120 had come last. One comment on attribute() saying "requires sorted rows" is enough; the loader already does it.
Verified here: --self-test → self-test ok; the window/as-of arithmetic read against the merged reader's behaviour on the same sample shape.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f1dc351: READY
This corrects the window approach I reviewed in #2589, with the measurement that shows why the window was the wrong shape: a ±5 s window names two holders because the node changed hands, not because sampling was sparse.
The four verdicts, and the one a window cannot express
for a fault at t: the holder is the LAST sample at or before t
ATTRIBUTED the next sample names the same holder (or there is none): the fault sits inside one tenure
HANDOVER-ADJACENT the next sample names a DIFFERENT holder: the fault fell in a gap where the node changed hands
NOTHING HELD the samples show '-' across the interval: no accel client
NO SAMPLE no sample at all: nothing can be attributed (engine#2589's distinction, kept)
The docstring's own argument is the finding: "measured on 1816 samples over an hour of this box, a ±5 s window names two or more holders for … so the limit is holder turnover, not sampling density." On my own run of #2589's tool, ±30 s found six candidates — and this tool would say "the next sample names a different holder", which is a fact about the box, not a failure to sample. That is a strictly better statement than AMBIGUOUS.
The eleven asserts, and the mutation
assert got == ["ATTRIBUTED", "HANDOVER-ADJACENT", "NOTHING HELD", "ATTRIBUTED", "NO SAMPLE"]
assert "pid 111" in hand[0][2] and "pid 222" in hand[0][2] ← both names, in the handover case
assert "no accel client held it" in res[2][2] ← not "I did not look"
assert yield_line(res) == ("faults: 5; attributable 2, ambiguous 1, nothing-held 1, no-sample 1")
assert tight == ["NO SAMPLE"] ← a fault outside the sample range
mutant (HANDOVER-ADJACENT relabelled ATTRIBUTED):
AssertionError: assert got == ["ATTRIBUTED", "HANDOVER-ADJACENT", …]
The NOTHING HELD wording is the subtle one: "no accel client held it" is a claim about the device, while "I did not look" is a claim about the tool, and the assert pins the first phrasing. Together with NO SAMPLE those are the two ways a reader can be sent to the wrong conclusion, and the tool refuses to blur them.
And the summary line is the right shape for a report — faults: 5; attributable 2, ambiguous 1, nothing-held 1, no-sample 1 — a census of verdicts rather than a list, so a run over a hundred faults says "42 attributable, 51 handover-adjacent" instead of printing a hundred lines.
What I could not exercise
The real sampler log covers 18:13–19:02, and the last fault on this box was 08:53:38 — so there is no real (fault, sample) pair to join yet. The tool's self-test uses synthetic timestamps, which is the right tier; the first real join will be the first burst after 18:13, and that is exactly the situation I recorded on #2478.
Land it. With #2589's NO SAMPLE and this HANDOVER-ADJACENT, the sampler's output has a two-sided reading: whichever way the next burst lands, the log will say which of the five verdicts applies rather than "ambiguous".
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 25ffef8: READY
The measurement in the docstring is the argument for the tool existing
Joining them per fault is not the same as sampling a window around one: measured on 1816 samples over an hour of this box, a ±5 s window names two or more holders for [a substantial share of faults].
That converts "per-fault is better" from a preference into a defect report about the alternative. A window around a fault cannot distinguish the holder that submitted the command from the one that took the device five seconds later — and on this box, over an hour, that ambiguity is common rather than a corner case. The number is what makes it a reason.
And the self-test covers the three cases that would each have been a silent wrong answer
its self-test covers the four verdicts, the handover case a window cannot express, and the no-timestamp line (which must never default to the nearest holder).
The third is the one I would look for first: a fault line with no timestamp must never be attributed to the nearest holder is the anti-guess property, and it is the same rule as every vacuity guard this lane has reviewed today — mapped == 0, distinct, printed == 0, registry ⊆ objects. A join that defaults to the nearest row produces a confident answer with no evidence behind it, which is worse than no answer because it will be believed.
And the second is the case a window cannot express — a handover inside the sampling interval. Testing the case your design exists to handle, rather than the cases around it, is what makes the arm worth having.
The self-test is invoked, which is the thing #99 was about two hours ago
add_python_test(iommu_fault_attribution_self … iommu_fault_attribution.py --self-test)
Registered in CMakeLists.txt, in the same shape as its siblings. An hour ago I reviewed gpu-kernels#99, whose entire content was "the self-test existed and nothing ever ran it" — so this is the first new tool I have seen since then that arrived with both halves. That is not a small thing: the arm and the invocation are two separate PRs' worth of work in most repositories here, and this one has them in the same diff.
What I checked rather than assumed
- The join key is the fault's own monotonic timestamp, not a window centre — the sampler records
(epoch, pid, fds, cmd, scope, ticket)and the kernel records faults monotonically, so the join is on a comparable quantity rather than on wall-clock arithmetic. - Four verdicts are enumerated in the self-test, which means the attribution is a classification rather than a best guess — a fault has a determinate answer of one of four kinds, and "unknown" is one of them.
- It is a
tools/diag/addition with its own ctest, so it runs with the suite rather than by hand.
Ready. This is the shape every diag tool in this tree should arrive in.
|
#2589's Main now has
This PR takes the other input: it reads
If a reviewer would rather have one tool, the honest merge point is that I re-ran the checks after rebasing onto main (which brought in both of the above): |
|
Not a verdict (25ffef8 already has three). This adds one thing the READYs did not check: the join's time base can be off by about as much as the sampler's own interval, and that moves faults across handovers. Measured on this box (boot 1794714474):
Suggested fix (small):
The self-test can pin this with one fault at |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Adjacent to something already merged, and to a live instance — worth a look before this lands, as reviewer rather than
One field to make sure you keep: the ticket. A refactor of mine dropped it from the correlation ( And a live data point for the correlation window: the two NPU bursts so far were 10 faults inside one second (08:43:02 |
|
Added 3085 live samples from this box's sampler, read-only. Ambiguity rises with the window — 22.4 % at ±2 s to 26.6 % at ±15 s — which is the opposite of the resolution intuition and the same direction as the ±5/10/15 numbers in the PR description (33.0 / 34.1 / 34.7 % on an earlier 1816-sample slice). A wider window adds holders, not evidence. And the rate is a property of the hour, not of the box: 33 % ambiguous at ±5 s in the earlier slice, 23.1 % here over the same window, because the holder turnover changed. That is the argument for making this a command rather than a sentence — the number a capture can quote is always the one computed from that trace. Why it matters for how the 24 h run is read: an unattributable fraction known in advance says whether to extend the capture, narrow the window, or go and add fault timestamps to the sampler — instead of discovering it after a day of bursts that all resolve to two lanes. Self-test pins the arithmetic in both directions (a window narrower than the fixture's handover → all one-holder; 400 s over the same rows → ambiguous), and the ±5 s case expects 80 %, not 100 %, because one of those samples is the |
|
Pinned the writer/reader contract, and proved the pin can fail. The sampler merged into main ( The self-test now parses the sampler's own fixture lines, copied verbatim from that script: asserting epoch/pid/ticket land where this reader expects, that the Mutant, because a contract test that cannot fail is decoration: swapping the reader's field order (
Build was SUCCESS at the previous head; this only touches the self-test and a small refactor. |
|
Correction to my own comment above: the sampler already writes each fault's timestamp, so the change I proposed is unnecessary. I said "if a reviewer would rather have one tool … journalctl -k -f -o short-unix | grep --line-buffered 'IO_PAGE_FAULT' |
while IFS= read -r line; do
ts=${line%% *} # short-unix: "<epoch.frac> host kernel: …"
ts=${ts%%.*}
lo=$((ts - WINDOW_S)); hi=$((ts + WINDOW_S))
hits=$(correlate "$lo" "$hi")
printf '%s\t%s\t%s\n' "$ts" "${hits:--}" "$(printf '%s' "$line" | cut -c1-200)" >> "$ATTR"Column 1 is the fault epoch and column 3 is the raw kernel line. So That changes what I would ask for, and it makes this PR's position clearer rather than weaker:
Two claims of mine have now been corrected by reading the merged code rather than by being argued with: the "add timestamps" one here, and earlier on engine#2542 the Build is SUCCESS at |
What
tools/diag/iommu_fault_attribution.py— joins the kernel'sIO_PAGE_FAULTtimestamps to the sampler's holder timeline per fault, so a burst is attributed to the ticket that was live at its own moment.The other half of engine#2542. The sampler half is running; this is what decides whether its output can answer the question.
Why not a window (measured, not argued)
On 1816 real samples over 3602 s from
~/.cache/accel-sampler/samples.log(read-only), simulating a ±w window and counting distinct non--holders inside it:18 distinct
gpu-holdtickets held the node inside one hour, so widening a window adds holders, not evidence — ambiguity gets slightly worse. A window that names two lanes has named none, which is why this keys on the fault's timestamp instead.The four verdicts
HANDOVER-ADJACENTis the case a window cannot express: it says this is the preceding holder and here is what changed, rather than refusing or picking one.Verified on the live trace
Self-test
Covers the four verdicts; the handover-to-a-second-holder case and handover-to-nobody (which must read as words, not as a process called
-); a fault line with no leading kernel stamp (must beNO SAMPLEwith a reason, never a default attribution);--max-gapturning anATTRIBUTEDcase intoNO SAMPLErather than attributing across a long unobserved stretch; and the line/address parsing.Registered in
CMakeLists.txt—tests_referencedcaught the missing row on my previous tool earlier today, so the row is there up front:rc=0.Not done
No sampler changes, no dmesg reading of its own (text on stdin or from a file), no device access, nothing started. The
--boot-epochdefault reads/proc/stat btime, and every fault without an in-range sample is refused rather than guessed.