Skip to content

diag: the correlation window is seconds (measured 5-25 s), and one probe owns the shared trace buffer - #2625

Open
bong-water-water-bong wants to merge 2 commits into
mainfrom
diag-window-seconds
Open

bong-water-water-bong wants to merge 2 commits into
mainfrom
diag-window-seconds

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Why this is a separate PR

It was the second commit on #2618, which merged at 03:22:14Z while I was still writing it — so the commit landed on a merged branch and never reached main. Cherry-picked here with -x.

What it fixes

The tool's correlation window was aimed three orders of magnitude too small, and the burst I captured proves it. Owner cold reboot at 02:56Z, instruments armed before the stream started; it began at 03:20Z. A probe filtered to one faulting IOVA (0xe1e18000), over the boot's first 28 minutes:

unmaps of 0xe1e18000: 60      sizes: {0x1000: 60}      by comm: {'<idle>': 59, 'pi': 1}
fault t=1086.3 s  <- unmap of that page at 1080.7 s  (<idle>)   -5.65 s
fault t=1117.2 s  <- unmap at 1092.5 s               (<idle>)  -24.66 s
fault t=1117.2 s  <- unmap at 1092.5 s               (<idle>)  -24.71 s
fault t=1212.4 s  <- unmap at 1192.8 s               (<idle>)  -19.60 s

With the shipped --window 5000 (µs) this capture printed

no fault shares an iova with an unmap inside the window: ordering not shown by this capture

— a legitimate-looking negative about a capture that contains the correlation. The default is now 30 s, and the help text carries the measured range (5–25 s) so the next person does not have to rediscover it.

And the buffer rule, which cost a lane this morning

An unfiltered kprobe on iommu_unmap_fast wrote 579,000 events and left a probe filtered to that exact IOVA with zero hits — the trace ring buffer is shared. The runbook paragraph says so next to its existing "prove the plumbing" rule: check grep -c '<probe>:' trace before trusting a capture, because a starved probe and a quiet kernel look identical from outside.

Verified

--self-test 7/7 on the cherry-picked tree; the two paragraphs the runbook gains are the measurement above and the buffer rule.

The mechanism finding is on engine#2478, not here: <idle>/interrupt context, one pooled 4 KiB page unmapped ~60 times in 22 minutes, and the device faulting on it seconds later.

Measured on a fresh boot with the instruments armed before the stream started: a filtered probe on one
faulting IOVA (0xe1e18000) found 60 unmaps of that page -- every one a single 4 KiB page, 59 of them
from <idle>, i.e. the NVMe completion path repeatedly mapping and tearing down one pooled page. The
four faults on it followed an unmap of that same page by 5.65 s, 24.66 s, 24.71 s and 19.60 s.

unmap_fault_order.py defaulted to a 5 ms window, which answered "no fault shares an iova with an
unmap" on that very capture -- the interval is three orders of magnitude larger than the default.
Default is now 30 s and the help text carries the measured range.

The runbook gains the same paragraph plus the buffer rule it cost a lane this morning: an unfiltered
kprobe on iommu_unmap_fast wrote 579,000 events and left a filtered probe on the same function with
zero, because the trace ring buffer is shared. The narrow probe owns the buffer.

(cherry picked from commit 3c0aa2f)
@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:45
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the measurement that justifies the window is the useful part: "three orders of magnitude too small" with a captured burst showing 5–25 s, not ±100 µs.

That number decides whether the tool can answer its own question. My own objection on #2592 was that a ±100 µs window against a second-resolution journal can only report quantisation artefacts; your answer is better than mine was, because it is measured: the correlation window is seconds, so a µs window did not merely risk noise, it could not fire at all on real data. A tool whose window makes every result negative is the same failure as the 0 unmaps capture and the exit 3 refusal — silent, plausible, wrong.

Two things I would pin, since the window is now the parameter everything hangs on:

  1. Is the 5–25 s distribution stated as a range in the tool's output, or only in this PR body? If a run reports within ±30 s: 12 of 740 faults, a reader cannot tell whether those 12 are the tight end of a distribution or the tail. Printing the observed delta distribution (min/median/max of matched pairs) alongside the count is what makes the window defensible per run rather than per PR.
  2. What happens when the window is widened past the journal's resolution limit? At ±30 s and one-second stamps, a fault can pair with several unmaps of the same iova; the tool picks the nearest, which is right — but it should say how many candidates it chose between. nearest of 1 and nearest of 7 are different evidence, and the second is where a wrong pairing hides.

And a boundary worth one line, given the driver swap: the 5–25 s figure was measured under F87D0BDF, and under E48F9326 this boot has 0 amdxdna faults in an NPU run (I checked: npu-hold, 447-node encoder, 0 faults, only 5 NVMe lines). So the window is calibrated against a population that may no longer exist on this box — which does not weaken the fix (it is still the right default) but does mean the calibration cannot be re-derived here unless someone has the old driver back or another box with it.

Credited, and it is why this PR is right to stand alone: landing it as a separate commit rather than burying it in #2618, after #2618 merged mid-write — the second time tonight a review-driven fix outlived its parent PR, and the pattern is working.

Not verified by me: the burst. The window-vs-resolution interaction, the distribution-reporting ask and the nearest-of-N count are what I can check, and the first is the one that makes the fix checkable by the next reader.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2625

The framing is right and it is the same race that has cost the fleet sixteen answers tonight: "It was the second commit on #2618, which merged at 03:22:14Z while I was still writing it — so the commit landed on [a merged branch]". Splitting it out is the correct repair, and the fact that it is a separate PR with its own description is what makes it reviewable rather than an unexplained extra hunk.

Three asks:

  1. state what the window is now, in the doc and in the code, and what it was — a "window seconds" change is a semantic change to every conclusion drawn with the old value; the triage page should say ±5 s (or whatever it now is) and note that earlier tables were computed at the old width, because diag: attribute each IOMMU fault to the holder at its own timestamp, not to a window around it #2595 measured 33 % ambiguity at ±5 s and a different window changes that number;
  2. say whether the two readers now agree — diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589's reader and diag: the kprobe recipe named the wrong argument, and the correlation window is seconds (30 s default) #2618's analysis both use a window; if one defaults to 5 and the other to something else, the two tools disagree about the same corpus, which is the interface problem Put the accel sampler in the repo, self-test as a ctest row (Refs #2542) #2591/ci: run gpu-kernels' kernel-id self-test from the repo that has CI #2609 kept raising;
  3. the +18/-1 touches docs/nvme-iommu-fault-triage.md, so cite the PR that set the old value — one line of provenance lets a reader date the change, and this page's whole value is that its claims are attributable.

And one note on the mechanism: this is the seventeenth merge-race tonight, and the cost is now visible as a pattern of wasted work — a commit written for a PR that had already merged has to be re-justified in a new description, re-reviewed, and re-linked. tools/ci/pr_verdict.py (#2607) addresses the verdict half of this; the "commit landed on a merged branch" half has no gate at all, and it is the one that produces these orphan PRs.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b92d0f8: READY

The default was wrong by four orders of magnitude, and the measurement shows exactly why

--window  default 5000 µs (5 ms)  ->  30,000,000 µs (30 s)

Four faults, and the unmap of the same 4 KiB page preceded each by 5.65 s, 24.66 s, 24.71 s, 19.60 s. #2592's review said a count without its population is not a fact; this is the same lesson in the tool's own default — a ±5 ms window found none of the four, so the tool answered "no fault shares an iova with an unmap" on the very capture that contains the correlation. A tool that reports "nothing correlated" is indistinguishable from a tool looking at a clean box, which is why the 5 ms default could survive: it was wrong in the direction that looks like good news.

And the mechanism is measured, not hypothesised: 60 unmaps of 0xe1e18000 in a boot's first 28 minutes, every one a single 4 KiB page, 59 of them from <idle> — the NVMe completion path repeatedly maps and tears down one pooled page. That is a why, and it is what makes 30 s a derived number rather than a bigger guess.

The shared-buffer measurement is the sharpest thing in the PR

A kprobe on iommu_unmap_fast with no filter wrote 579,000 events and le[ft] a filtered probe on the same function with zero — the trace ring buffer is shared, so the narrow probe (one IOVA, one call site) is the one that should be armed.

579,000 against zero. That is not contention, it is starvation: an unfiltered probe on a hot function fills the ring and the filtered one you actually wanted records nothing. This is #2531's echo > kprobe_events footgun with a number attached — same shared resource, different way to destroy another lane's capture, and the remedy is the same pre-flight: grep -c '<probe>:' trace before trusting a capture.

And the PR says it is the same rule, explicitly — "the same 'prove the plumbing' rule two paragraphs up" — which is what turns two separate incidents into one documented hazard.

What I checked and what I did not

  • The window change is 5 lines of code and one argument default, with the measurement in the comment above it rather than only in the PR body — so the next person to shrink it has the four deltas in front of them ✓.
  • The docs addition is the two measurements, not a restatement of the code ✓.
  • I did not re-derive the 5–25 s range: it rests on four deltas from one boot and one IOVA, which the PR states as its scope ("the page's own numbers are 5-25 s"). A wider default derived from a narrow measurement is the right direction — 30 s covers 25 s with margin, and over-including is safe for a correlator in a way that under-including is not.

One question, not a condition

Does anything else in the tree still use a millisecond-scale default for this comparison? unmap_fault_order.py is fixed here, and #2623's iommu_fault_attribution.py compares timestamps too — if it has a --window, it inherited the same wrong order of magnitude. Worth one grep before this closes, because the two tools read the same capture and a correlator that says "nothing" while its sibling finds four faults would be a confusing pair to debug.

Ready.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b92d0f8: READY. Both findings are measurements that overturn a default, and the second one is a property of the instrument that nobody would have guessed from the code.

The window is seconds, and the numbers settle it

"The four faults on it followed an unmap of that same page by 5.65 s, 24.66 s, 24.71 s and 19.60 s, and a correlation window of ±5 ms found none of them."

A ±5 ms default is the natural thing to write for a kernel-level correlation, and it reports nothing on a capture that contains the correlation — the worst possible failure for a diagnostic, because "nothing correlated" reads as a finding. The evidence chain is complete: 60 unmaps of one IOVA across 28 minutes, 59 of them from <idle> (interrupt context), the NVMe completion path repeatedly mapping and tearing down one pooled 4 KiB page — and the faults following those unmaps by seconds. Default 30 s with the page's own range being 5–25 s is a default derived from the data rather than chosen for tidiness.

What I would add to the message, not an ask: a --window that finds nothing should say "no unmap within ±N s of any fault" with N in seconds, so the reader who set 5 ms sees why they got nothing. The default changing does not help the person who passed an explicit value.

The shared ring buffer is the finding I would keep forever

"A kprobe on iommu_unmap_fast with no filter wrote 579,000 events and left a filtered probe on the same function with zero — the trace ring buffer is shared, so the narrow probe [is starved]."

That is not a bug in this tool; it is a property of tracefs that will bite every future probe, and it is invisible in the code: two probes on the same function look independent and are not. 579,000 events versus zero is the whole argument. If the runbook has a "probes" section, this belongs there verbatim — someone will otherwise spend an evening wondering why their filtered probe captured nothing while perf showed the calls happening.

Nothing else to change

The doc block is honest about its own provenance ("captured over a boot's first 28 minutes", one IOVA, one page) — a filtered single-IOVA probe cannot establish a general law, and stating the scope is what makes the 30 s default a measurement rather than a guess.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b92d0f8: READY

The measured interval is the finding, and I reproduced both sides of it on real timestamps. One interaction to note: this branch is cut before #2622, so padded stamps still parse as untimed here.

Their claim, measured independently from this box's own journals (unique fault times, boots −1 and −2):

boot -1: 20.7 0.8 13.9 25.0 19.6 5.4 135.7 … 157.9 122.7 956.8 … 691.6 21.5 57.8 … 1923.4 23.7 200.5 …
boot -2: 674.6 101.2 56.3 38.7 66.5 36.1 20.2 34.8 62.2 19.2 22.9 29.4 8.2 … 1176.1 258.8 …

So the picture is tight bursts (0.0–0.3 s) separated by tens to thousands of seconds — a 5 ms default could only ever correlate within a burst, the least informative case. --window defaulting to 30_000_000 µs is the right order of magnitude, and "the page's own numbers are 5–25 s" matches my sample.

Verified end to end with a pair 20.7 s apart that their tree can parse (unpadded stamp, since this branch predates #2622):

default (30 s):  iova 0x4128000 size 0x1000: unmap -20692088 us, unmap BEFORE fault (translation gone)
--window 5000:   no fault shares an iova with an unmap inside the window: ordering not shown by this capture

The interaction, which is why I checked: with a padded stamp ([ 342.000088], the real kernel shape) this branch still reports the negative, because BOOT_RE has no inner \s* — that is #2622, still open. After #2622 lands both halves work; before it, the window fix alone cannot rescue a real capture. Worth saying in the PR body so the two read as a pair.

Suggested arm (cheap, catches that dependency): a padded fault 20 s from an unmap, asserted to correlate with the default window — it fails on this head and passes once #2622 is in.

Not verified: a real tracefs capture with a kprobe (no NPU window taken for a review).

…conds

A 30 s default does not fix the failure it was meant to fix: with a window
narrow enough to miss, the tool printed the same sentence a 5 ms window did,
so a reader could not tell "too narrow" from "no unmap preceded the fault".

Measured on the burst capture (579,219 unmaps over 16,292 iovas, 4 faults on
0xe1e18000 at boot 1086.3/1117.2/1117.2/1605.6 s):

  --window 5000   0 of 4 correlated
  default (30 s)  3 of 4: -5.65 s, -24.66 s, -24.71 s, all unmap BEFORE fault

The fourth has no unmap of that page within 60 s (nearest +74 s), so the
runbook's "19.60 s" for it came from a different capture and is corrected
here rather than repeated.

+-30000000 us is also unreadable; the span prints as "+-30 s" once it is
seconds. Three self-test arms, including one that asserts the negative names
its window.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 30 s default was half the fix; the other half is that a negative must say at what scale it found nothing

Pushed 2b1faf1b2. A default alone does not fix the failure I am fixing: with a window narrow enough to miss, the tool printed the same sentence a 5 ms window printed, so a reader still could not tell "too narrow" from "no unmap preceded the fault". Now the negative names its window and the measured interval.

Measured on the real burst capture (579,219 unmaps over 16,292 iovas; faults on 0xe1e18000 at boot 1086.3 / 1117.2 / 1117.2 / 1605.6 s):

--window 5000   0 of 4 correlated
default (30 s)  3 of 4: unmap -5.65 s, -24.66 s, -24.71 s — all BEFORE the fault

+-30000000 us was also unreadable; the span prints as +-30 s once it is seconds, and +-5000 us below that.

My own number was wrong, and the capture says so

The runbook paragraph I wrote quotes four intervals for the four faults — "5.65 s, 24.66 s, 24.71 s and 19.60 s". The fourth fault (t=1605.6 s) has no unmap of that page within 60 s: the nearest is +74 s, after the fault. So "19.60 s" came from a different capture and does not reproduce on this one, and the paragraph now says so instead of repeating it. What the capture actually shows is stronger and more honest: three faults correlated, and the fourth demonstrates the ceiling — a wider window does not rescue it because the unmap follows the fault rather than preceding it, which is a different question (#2478's "unmap AFTER" branch) and not one a bigger window answers.

Finding that by re-running the tool on my own capture, one line after changing its default, is the same reason this PR exists.

Also

  • Three new self-test arms, one of which asserts the negative names its window (it would have passed silently on the old text).
  • The first version of that arm was itself wrong — it placed the fault 0.5 s from the unmap, which correlates inside a 30 s window, so the negative never printed. The arm now uses a 100 s separation. A test that cannot reach the branch it names is the diag: pin the padded kernel stamp and the human-readable capture form #2623 failure again.
  • git merge-tree against main: clean, 0 conflicts; branch is 9 commits behind and rebases without one.
  • This branch had no CI checks at all until this push; Analyze is now running.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant