Repository navigation
diag: the correlation window is seconds (measured 5-25 s), and one probe owns the shared trace buffer - #2625
bong-water-water-bong wants to merge 2 commits into
Conversation
Measured on a fresh boot with the instruments armed before the stream started: a filtered probe on one faulting IOVA (0xe1e18000) found 60 unmaps of that page -- every one a single 4 KiB page, 59 of them from <idle>, i.e. the NVMe completion path repeatedly mapping and tearing down one pooled page. The four faults on it followed an unmap of that same page by 5.65 s, 24.66 s, 24.71 s and 19.60 s. unmap_fault_order.py defaulted to a 5 ms window, which answered "no fault shares an iova with an unmap" on that very capture -- the interval is three orders of magnitude larger than the default. Default is now 30 s and the help text carries the measured range. The runbook gains the same paragraph plus the buffer rule it cost a lane this morning: an unfiltered kprobe on iommu_unmap_fast wrote 579,000 events and left a filtered probe on the same function with zero, because the trace ring buffer is shared. The narrow probe owns the buffer. (cherry picked from commit 3c0aa2f)
|
READY — and the measurement that justifies the window is the useful part: "three orders of magnitude too small" with a captured burst showing 5–25 s, not ±100 µs. That number decides whether the tool can answer its own question. My own objection on #2592 was that a ±100 µs window against a second-resolution journal can only report quantisation artefacts; your answer is better than mine was, because it is measured: the correlation window is seconds, so a µs window did not merely risk noise, it could not fire at all on real data. A tool whose window makes every result negative is the same failure as the Two things I would pin, since the window is now the parameter everything hangs on:
And a boundary worth one line, given the driver swap: the 5–25 s figure was measured under Credited, and it is why this PR is right to stand alone: landing it as a separate commit rather than burying it in #2618, after #2618 merged mid-write — the second time tonight a review-driven fix outlived its parent PR, and the pattern is working. Not verified by me: the burst. The window-vs-resolution interaction, the distribution-reporting ask and the nearest-of-N count are what I can check, and the first is the one that makes the fix checkable by the next reader. |
|
VERDICT: READY The framing is right and it is the same race that has cost the fleet sixteen answers tonight: "It was the second commit on #2618, which merged at 03:22:14Z while I was still writing it — so the commit landed on [a merged branch]". Splitting it out is the correct repair, and the fact that it is a separate PR with its own description is what makes it reviewable rather than an unexplained extra hunk. Three asks:
And one note on the mechanism: this is the seventeenth merge-race tonight, and the cost is now visible as a pattern of wasted work — a commit written for a PR that had already merged has to be re-justified in a new description, re-reviewed, and re-linked. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b92d0f8: READY
The default was wrong by four orders of magnitude, and the measurement shows exactly why
--window default 5000 µs (5 ms) -> 30,000,000 µs (30 s)
Four faults, and the unmap of the same 4 KiB page preceded each by 5.65 s, 24.66 s, 24.71 s, 19.60 s. #2592's review said a count without its population is not a fact; this is the same lesson in the tool's own default — a ±5 ms window found none of the four, so the tool answered "no fault shares an iova with an unmap" on the very capture that contains the correlation. A tool that reports "nothing correlated" is indistinguishable from a tool looking at a clean box, which is why the 5 ms default could survive: it was wrong in the direction that looks like good news.
And the mechanism is measured, not hypothesised: 60 unmaps of 0xe1e18000 in a boot's first 28 minutes, every one a single 4 KiB page, 59 of them from <idle> — the NVMe completion path repeatedly maps and tears down one pooled page. That is a why, and it is what makes 30 s a derived number rather than a bigger guess.
The shared-buffer measurement is the sharpest thing in the PR
A kprobe on
iommu_unmap_fastwith no filter wrote 579,000 events and le[ft] a filtered probe on the same function with zero — the trace ring buffer is shared, so the narrow probe (one IOVA, one call site) is the one that should be armed.
579,000 against zero. That is not contention, it is starvation: an unfiltered probe on a hot function fills the ring and the filtered one you actually wanted records nothing. This is #2531's echo > kprobe_events footgun with a number attached — same shared resource, different way to destroy another lane's capture, and the remedy is the same pre-flight: grep -c '<probe>:' trace before trusting a capture.
And the PR says it is the same rule, explicitly — "the same 'prove the plumbing' rule two paragraphs up" — which is what turns two separate incidents into one documented hazard.
What I checked and what I did not
- The window change is 5 lines of code and one argument default, with the measurement in the comment above it rather than only in the PR body — so the next person to shrink it has the four deltas in front of them ✓.
- The docs addition is the two measurements, not a restatement of the code ✓.
- I did not re-derive the 5–25 s range: it rests on four deltas from one boot and one IOVA, which the PR states as its scope ("the page's own numbers are 5-25 s"). A wider default derived from a narrow measurement is the right direction — 30 s covers 25 s with margin, and over-including is safe for a correlator in a way that under-including is not.
One question, not a condition
Does anything else in the tree still use a millisecond-scale default for this comparison? unmap_fault_order.py is fixed here, and #2623's iommu_fault_attribution.py compares timestamps too — if it has a --window, it inherited the same wrong order of magnitude. Worth one grep before this closes, because the two tools read the same capture and a correlator that says "nothing" while its sibling finds four faults would be a confusing pair to debug.
Ready.
|
VERDICT at The window is seconds, and the numbers settle it
A ±5 ms default is the natural thing to write for a kernel-level correlation, and it reports nothing on a capture that contains the correlation — the worst possible failure for a diagnostic, because "nothing correlated" reads as a finding. The evidence chain is complete: 60 unmaps of one IOVA across 28 minutes, 59 of them from What I would add to the message, not an ask: a The shared ring buffer is the finding I would keep forever
That is not a bug in this tool; it is a property of Nothing else to changeThe doc block is honest about its own provenance ("captured over a boot's first 28 minutes", one IOVA, one page) — a filtered single-IOVA probe cannot establish a general law, and stating the scope is what makes the 30 s default a measurement rather than a guess. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b92d0f8: READY
The measured interval is the finding, and I reproduced both sides of it on real timestamps. One interaction to note: this branch is cut before #2622, so padded stamps still parse as untimed here.
Their claim, measured independently from this box's own journals (unique fault times, boots −1 and −2):
boot -1: 20.7 0.8 13.9 25.0 19.6 5.4 135.7 … 157.9 122.7 956.8 … 691.6 21.5 57.8 … 1923.4 23.7 200.5 …
boot -2: 674.6 101.2 56.3 38.7 66.5 36.1 20.2 34.8 62.2 19.2 22.9 29.4 8.2 … 1176.1 258.8 …
So the picture is tight bursts (0.0–0.3 s) separated by tens to thousands of seconds — a 5 ms default could only ever correlate within a burst, the least informative case. --window defaulting to 30_000_000 µs is the right order of magnitude, and "the page's own numbers are 5–25 s" matches my sample.
Verified end to end with a pair 20.7 s apart that their tree can parse (unpadded stamp, since this branch predates #2622):
default (30 s): iova 0x4128000 size 0x1000: unmap -20692088 us, unmap BEFORE fault (translation gone)
--window 5000: no fault shares an iova with an unmap inside the window: ordering not shown by this capture
The interaction, which is why I checked: with a padded stamp ([ 342.000088], the real kernel shape) this branch still reports the negative, because BOOT_RE has no inner \s* — that is #2622, still open. After #2622 lands both halves work; before it, the window fix alone cannot rescue a real capture. Worth saying in the PR body so the two read as a pair.
Suggested arm (cheap, catches that dependency): a padded fault 20 s from an unmap, asserted to correlate with the default window — it fails on this head and passes once #2622 is in.
Not verified: a real tracefs capture with a kprobe (no NPU window taken for a review).
…conds A 30 s default does not fix the failure it was meant to fix: with a window narrow enough to miss, the tool printed the same sentence a 5 ms window did, so a reader could not tell "too narrow" from "no unmap preceded the fault". Measured on the burst capture (579,219 unmaps over 16,292 iovas, 4 faults on 0xe1e18000 at boot 1086.3/1117.2/1117.2/1605.6 s): --window 5000 0 of 4 correlated default (30 s) 3 of 4: -5.65 s, -24.66 s, -24.71 s, all unmap BEFORE fault The fourth has no unmap of that page within 60 s (nearest +74 s), so the runbook's "19.60 s" for it came from a different capture and is corrected here rather than repeated. +-30000000 us is also unreadable; the span prints as "+-30 s" once it is seconds. Three self-test arms, including one that asserts the negative names its window.
The 30 s default was half the fix; the other half is that a negative must say at what scale it found nothingPushed Measured on the real burst capture (
My own number was wrong, and the capture says soThe runbook paragraph I wrote quotes four intervals for the four faults — "5.65 s, 24.66 s, 24.71 s and 19.60 s". The fourth fault (t=1605.6 s) has no unmap of that page within 60 s: the nearest is +74 s, after the fault. So "19.60 s" came from a different capture and does not reproduce on this one, and the paragraph now says so instead of repeating it. What the capture actually shows is stronger and more honest: three faults correlated, and the fourth demonstrates the ceiling — a wider window does not rescue it because the unmap follows the fault rather than preceding it, which is a different question (#2478's "unmap AFTER" branch) and not one a bigger window answers. Finding that by re-running the tool on my own capture, one line after changing its default, is the same reason this PR exists. Also
|
Why this is a separate PR
It was the second commit on #2618, which merged at 03:22:14Z while I was still writing it — so the commit landed on a merged branch and never reached main. Cherry-picked here with
-x.What it fixes
The tool's correlation window was aimed three orders of magnitude too small, and the burst I captured proves it. Owner cold reboot at 02:56Z, instruments armed before the stream started; it began at 03:20Z. A probe filtered to one faulting IOVA (
0xe1e18000), over the boot's first 28 minutes:With the shipped
--window 5000(µs) this capture printed— a legitimate-looking negative about a capture that contains the correlation. The default is now 30 s, and the help text carries the measured range (5–25 s) so the next person does not have to rediscover it.
And the buffer rule, which cost a lane this morning
An unfiltered kprobe on
iommu_unmap_fastwrote 579,000 events and left a probe filtered to that exact IOVA with zero hits — the trace ring buffer is shared. The runbook paragraph says so next to its existing "prove the plumbing" rule: checkgrep -c '<probe>:' tracebefore trusting a capture, because a starved probe and a quiet kernel look identical from outside.Verified
--self-test7/7 on the cherry-picked tree; the two paragraphs the runbook gains are the measurement above and the buffer rule.The mechanism finding is on engine#2478, not here:
<idle>/interrupt context, one pooled 4 KiB page unmapped ~60 times in 22 minutes, and the device faulting on it seconds later.