Repository navigation
diag: the kprobe recipe named the wrong argument, and the correlation window is seconds (30 s default) - #2618
Conversation
…d iova is now refused
engine#2592 shipped this recipe:
echo 'p:unm iommu_unmap_fast iova=%di size=%si' >> kprobe_events
`iommu_unmap_fast(domain, iova, size)` takes %rdi/%rsi/%rdx, so the probe FIRES and parses but puts the
domain pointer in the iova column. Measured on a fresh boot:
unm: iova=0xffff8e83841c3a00 size=0x45ec0
With %si/%dx the same trace gives iova=0x45200000 size=0x400 -- page-aligned IOVAs and a real
unmapped span.
That matters more than a typo: with the wrong column nothing can ever correlate, and the tool's
ordinary negative ("no fault shares an iova with an unmap inside the window") reads as a real
finding. So the fix has two halves: the recipe, and a refusal -- when every iova in a capture is
>= 2**47 the column cannot be IOVAs at all, so the tool exits 4 and names the argument instead of
printing a correlation result.
Self-test gains both directions: the pointer-shaped column is detected, and a real IOVA column is
not mistaken for one.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ff570ae: READY
The recipe was wrong in a way that fires, parses, and silently shifts every column — and the new detector is the part that makes the mistake impossible to read as a finding.
The shift, confirmed on the tool's own parser
documented before: p:unm iommu_unmap_fast iova=%di size=%si
iommu_unmap_fast(domain, iova, size) → %rdi, %rsi, %rdx
so that probe records iova = the DOMAIN POINTER, size = the IOVA
the capture's own values (from the PR): iova=0xffff8e83841c3a00 size=0x45ec0
Parsed through the tool: (1.0, 18446619294241143296, 286400) — an iova of 1.8e19 that no fault can ever match. Nothing errors: the probe fires, the regex matches, the correlation finds zero pairs, and the tool prints its ordinary negative — "no fault shares an iova with an unmap" — which reads as a finding about the device when it is a fact about the format string.
The detector, both directions
pointer_shaped([(1, 0xffff8e83841c3a00, 0x45ec0)]) → True (the wrong-argument capture)
pointer_shaped([(1, 0x35540000, 0x40000)]) → False (a real IOVA column)
IOVA_MAX = 1 << 47 (x86-64 IOVAs are below it)
1 << 47 is the right threshold: a canonical kernel pointer is 0xffff…, and an IOVA in a 48-bit space sits far below 2^47, so the test separates them without knowing the workload. And the message says what the mistake was, not just that the values look odd.
And it is in the self-test: a pointer-shaped iova column is detected, not read as uncorrelated beside a real IOVA column is not mistaken for one — both sides, which is the arm this defect needs (a detector that fired on everything would pass a one-sided test).
My own captures are unaffected, and I checked
I used iova=%si size=%dx in every kprobe tonight (the correct form), so 0x35540000/0x40000-shaped values throughout, and the 461-event unmA capture parses as (…, 894697472, 262144). The recipe's error was one I happened to avoid by picking the arguments from the function signature rather than from the docstring — the inverse of the usual failure, and a good argument for the detector existing.
One note, not an ask
The docstring now says "check the values look like IOVAs", and pointer_shaped() is exactly that check — so the tool could refuse rather than warn when it fires. It currently reports; a rc 2 with "the capture's iova column is pointer-shaped: the probe names the wrong argument" would end the class. unmap_fault_order.py already returns 0 for "both inputs parsed" ("the tool reports, it does not judge"), which is a deliberate contract — so this is a design question, not a defect.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ff570ae: CHANGES
The register fix is right (%si/%dx for iommu_unmap_fast(domain, iova, size)), and refusing a pointer-shaped iova column with rc 4 is a good guard. But with the recipe corrected, the tool still cannot correlate a real capture, for two reasons. Both are on main too, and both are in the path this PR exists to unblock. On git merge-tree origin/main ff570ae:
1. No real fault line gets a timestamp. BOOT_RE = r"^\s*\[(\d+\.\d+)\]" needs a digit right after [. The kernel pads the stamp, and this boot's own fault lines (journalctl -k -o short-monotonic, and the dmesg default) start [ 1086.308029] strixhalo kernel: nvme 00…:
parse_faults(["[ 100.000600] nvme … IO_PAGE_FAULT … address=0xeb536000 …"]) -> t = None (this head and main)
parse_faults(["[100.000600] nvme … IO_PAGE_FAULT … address=0xeb536000 …"]) -> t = 100.0006
correlate() skips t is None, so a fault at the exact iova of an unmap 500 µs earlier still reads:
faults parsed: 1; with an unmap of the same iova within +-5000 us: 0
no fault shares an iova with an unmap inside the window: ordering not shown by this capture
That is the same false negative as the %di recipe, from a second cause. The self-test can't see it because its correlation arms build (t, iova) tuples directly, and its one parse arm is the trace line, not a fault line. Fix: r"^\s*\[\s*(\d+\.\d+)\]", plus an arm that runs parse_faults on a padded dmesg line.
2. Only an unmap that starts at the fault address can match. Still open from my #2592 note. correlate() keys by_iova on the unmap's start, and size is parsed and never used. dma-iommu unmaps a whole request's IOVA range in one call, and this PR's own trace shows spans (iova=0x45200000 size=0x400). With a fault at 0xeb536000 and an unmap iova=0xeb530000 size=0x10000 500 µs earlier, the result is 0 correlated. Fix: match iova <= addr < iova + size, and add an arm with a fault mid-range.
With both fixed, the capture this PR's recipe produces can answer the ordering question. As it stands, every real capture reads "ordering not shown".
Second commit: the window default was aimed three orders of magnitude too small, measured on a real burstThe owner's cold reboot at 02:56Z let me arm the instruments before the stream started, and at 03:20Z it began. A filtered probe on one faulting IOVA ( This tool's default was Second half of the same lesson, and it cost a lane this morning: an unfiltered kprobe on Scope unchanged: this PR is still the tool and its runbook page. The mechanism finding ( |
The bug is in the recipe this tool ships, and the tool could not see it
tools/diag/unmap_fault_order.py(engine#2592) documents:iommu_unmap_fast(domain, iova, size)takes%rdi, %rsi, %rdxon x86-64, so that probe fires and parses — and puts the domain pointer in the iova column. Measured on a fresh boot a minute after arming it:Re-armed with
iova=%si size=%dx, the same trace gives:page-aligned IOVAs and a real unmapped span.
Why this is worse than a typo
With the wrong column nothing can ever correlate, and the tool's ordinary negative is
which reads as a finding about the kernel. That is the exact failure mode this fleet has spent two days on — an absent measurement wearing a result's clothes — and it was in the docstring of the tool built to avoid it.
Change
%si/%dx, with the reason and the measured wrong-value in the comment, so the next person does not "simplify" it back;pointer_shaped()is true when every parsed iova is>= 2**47(a canonical x86-64 kernel pointer), andmain()then exits 4 withVerified
Note for the issue this tool serves
The capture above is a fresh boot's first 10 minutes (owner cold reboot at 02:56Z): 385,494
iommu_unmap_fastcalls, 9,372 distinct IOVAs, sizes 0x1000–0x40000, and — so far — zeroIO_PAGE_FAULT. The previous boot's first fault came at +5 min, so this boot is overdue by the pattern; the probe and the accel sampler are both armed, and the next burst is what engine#2478/#2542 needs.