Skip to content

diag: the kprobe recipe named the wrong argument, and the correlation window is seconds (30 s default) - #2618

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag-iova-not-pointer
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag-iova-not-pointer

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The bug is in the recipe this tool ships, and the tool could not see it

tools/diag/unmap_fault_order.py (engine#2592) documents:

echo 'p:unm iommu_unmap_fast iova=%di size=%si' >> kprobe_events

iommu_unmap_fast(domain, iova, size) takes %rdi, %rsi, %rdx on x86-64, so that probe fires and parses — and puts the domain pointer in the iova column. Measured on a fresh boot a minute after arming it:

unm: (iommu_unmap_fast+0x4/0x10) iova=0xffff8e83841c3a00 size=0x45ec0

Re-armed with iova=%si size=%dx, the same trace gives:

unm: (iommu_unmap_fast+0x4/0x10) iova=0x45200000 size=0x400
unm: (iommu_unmap_fast+0x4/0x10) iova=0x45180000 size=0x400

page-aligned IOVAs and a real unmapped span.

Why this is worse than a typo

With the wrong column nothing can ever correlate, and the tool's ordinary negative is

no fault shares an iova with an unmap inside the window: ordering not shown by this capture

which reads as a finding about the kernel. That is the exact failure mode this fleet has spent two days on — an absent measurement wearing a result's clothes — and it was in the docstring of the tool built to avoid it.

Change

  1. the recipe is %si/%dx, with the reason and the measured wrong-value in the comment, so the next person does not "simplify" it back;
  2. a pointer-shaped iova column is refused: pointer_shaped() is true when every parsed iova is >= 2**47 (a canonical x86-64 kernel pointer), and main() then exits 4 with
this capture's iova column is not IOVA-shaped (every value >= 2**47, e.g. 0xffff8e83841c3a00): the kprobe is
naming the wrong argument. iommu_unmap_fast(domain, iova, size) is %si/%dx, not %di/%si -- re-arm and check that
the values look like IOVAs before reading any result from it.

Verified

--self-test   7/7 ok   (the two new cases: a pointer-shaped column IS detected; a real IOVA column is NOT)
real capture  unmap events parsed: 385,494 over 9,372 distinct iova(s); sizes 0x1000 .. 0x40000   rc 0
synthetic     iova=0xffff8e83841c3a00 -> the refusal above                                            rc 4

Note for the issue this tool serves

The capture above is a fresh boot's first 10 minutes (owner cold reboot at 02:56Z): 385,494 iommu_unmap_fast calls, 9,372 distinct IOVAs, sizes 0x1000–0x40000, and — so far — zero IO_PAGE_FAULT. The previous boot's first fault came at +5 min, so this boot is overdue by the pattern; the probe and the accel sampler are both armed, and the next burst is what engine#2478/#2542 needs.

…d iova is now refused

engine#2592 shipped this recipe:

    echo 'p:unm iommu_unmap_fast iova=%di size=%si' >> kprobe_events

`iommu_unmap_fast(domain, iova, size)` takes %rdi/%rsi/%rdx, so the probe FIRES and parses but puts the
domain pointer in the iova column. Measured on a fresh boot:

    unm: iova=0xffff8e83841c3a00 size=0x45ec0

With %si/%dx the same trace gives iova=0x45200000 size=0x400 -- page-aligned IOVAs and a real
unmapped span.

That matters more than a typo: with the wrong column nothing can ever correlate, and the tool's
ordinary negative ("no fault shares an iova with an unmap inside the window") reads as a real
finding. So the fix has two halves: the recipe, and a refusal -- when every iova in a capture is
>= 2**47 the column cannot be IOVAs at all, so the tool exits 4 and names the argument instead of
printing a correlation result.

Self-test gains both directions: the pointer-shaped column is detected, and a real IOVA column is
not mistaken for one.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The kprobe recipe is corrected to use %si/%dx.
  • Logic is added to detect pointer-shaped IOVAs and refuse processing.
  • A self-test case is included to verify the detection.
  • Documentation and comments are updated to explain the fix.

Non-compliant requirements:

  • The PR does not include updated documentation in docs/, README, or the wiki to reflect the corrected kprobe usage.

Requires further human verification:

  • The PR does not include any tests that simulate the actual NPU workload that triggers the IO_PAGE_FAULT events, which is needed to fully validate the fix in the context of the reported issue.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect kprobe argument format

The kprobe recipe in tools/diag/unmap_fault_order.py was using %di/%si instead of %si/%dx, which caused the domain pointer to be interpreted as an IOVA. This led to uncorrelatable data and could mask real issues. The fix corrects this to use the proper argument placeholders.

echo 'p:unm iommu_unmap_fast iova=%si size=%dx' >> kprobe_events
Pointer-shaped IOVA detection

The tool now includes logic to detect when the iova column contains pointer-shaped values (>= 2**47), which indicates that the kprobe was configured incorrectly. When such values are detected, the tool refuses to process the capture and reports an error, preventing false negatives in correlation.

IOVA_MAX = 1 << 47          # x86-64 IOMMU IOVAs are below this; a canonical kernel pointer (0xffff8...) is not


def pointer_shaped(unmaps):
    """True when the capture's iova column cannot be IOVAs at all.

    A kprobe whose format string names the wrong argument (engine#2592 review: `%di` instead of `%si` puts the
    domain pointer in the iova field) yields values like 0xffff8e83841c3a00. Nothing can correlate them, and this
    tool's ordinary negative -- "no fault shares an iova with an unmap" -- would read as a real finding. Every iova
    at or above 2**47 is that mistake, not a measurement.
    """
    return bool(unmaps) and all(u[1] >= IOVA_MAX for u in unmaps)

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:07

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ff570ae: READY

The recipe was wrong in a way that fires, parses, and silently shifts every column — and the new detector is the part that makes the mistake impossible to read as a finding.

The shift, confirmed on the tool's own parser

documented before:  p:unm iommu_unmap_fast iova=%di size=%si
iommu_unmap_fast(domain, iova, size) → %rdi, %rsi, %rdx
  so that probe records  iova = the DOMAIN POINTER,  size = the IOVA
the capture's own values (from the PR):  iova=0xffff8e83841c3a00 size=0x45ec0

Parsed through the tool: (1.0, 18446619294241143296, 286400) — an iova of 1.8e19 that no fault can ever match. Nothing errors: the probe fires, the regex matches, the correlation finds zero pairs, and the tool prints its ordinary negative — "no fault shares an iova with an unmap" — which reads as a finding about the device when it is a fact about the format string.

The detector, both directions

pointer_shaped([(1, 0xffff8e83841c3a00, 0x45ec0)])  →  True       (the wrong-argument capture)
pointer_shaped([(1, 0x35540000,       0x40000)])    →  False      (a real IOVA column)
IOVA_MAX = 1 << 47                                                  (x86-64 IOVAs are below it)

1 << 47 is the right threshold: a canonical kernel pointer is 0xffff…, and an IOVA in a 48-bit space sits far below 2^47, so the test separates them without knowing the workload. And the message says what the mistake was, not just that the values look odd.

And it is in the self-test: a pointer-shaped iova column is detected, not read as uncorrelated beside a real IOVA column is not mistaken for one — both sides, which is the arm this defect needs (a detector that fired on everything would pass a one-sided test).

My own captures are unaffected, and I checked

I used iova=%si size=%dx in every kprobe tonight (the correct form), so 0x35540000/0x40000-shaped values throughout, and the 461-event unmA capture parses as (…, 894697472, 262144). The recipe's error was one I happened to avoid by picking the arguments from the function signature rather than from the docstring — the inverse of the usual failure, and a good argument for the detector existing.

One note, not an ask

The docstring now says "check the values look like IOVAs", and pointer_shaped() is exactly that check — so the tool could refuse rather than warn when it fires. It currently reports; a rc 2 with "the capture's iova column is pointer-shaped: the probe names the wrong argument" would end the class. unmap_fault_order.py already returns 0 for "both inputs parsed" ("the tool reports, it does not judge"), which is a deliberate contract — so this is a design question, not a defect.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ff570ae: CHANGES

The register fix is right (%si/%dx for iommu_unmap_fast(domain, iova, size)), and refusing a pointer-shaped iova column with rc 4 is a good guard. But with the recipe corrected, the tool still cannot correlate a real capture, for two reasons. Both are on main too, and both are in the path this PR exists to unblock. On git merge-tree origin/main ff570ae:

1. No real fault line gets a timestamp. BOOT_RE = r"^\s*\[(\d+\.\d+)\]" needs a digit right after [. The kernel pads the stamp, and this boot's own fault lines (journalctl -k -o short-monotonic, and the dmesg default) start [ 1086.308029] strixhalo kernel: nvme 00…:

parse_faults(["[  100.000600] nvme … IO_PAGE_FAULT … address=0xeb536000 …"])  ->  t = None    (this head and main)
parse_faults(["[100.000600] nvme … IO_PAGE_FAULT … address=0xeb536000 …"])    ->  t = 100.0006

correlate() skips t is None, so a fault at the exact iova of an unmap 500 µs earlier still reads:

faults parsed: 1; with an unmap of the same iova within +-5000 us: 0
  no fault shares an iova with an unmap inside the window: ordering not shown by this capture

That is the same false negative as the %di recipe, from a second cause. The self-test can't see it because its correlation arms build (t, iova) tuples directly, and its one parse arm is the trace line, not a fault line. Fix: r"^\s*\[\s*(\d+\.\d+)\]", plus an arm that runs parse_faults on a padded dmesg line.

2. Only an unmap that starts at the fault address can match. Still open from my #2592 note. correlate() keys by_iova on the unmap's start, and size is parsed and never used. dma-iommu unmaps a whole request's IOVA range in one call, and this PR's own trace shows spans (iova=0x45200000 size=0x400). With a fault at 0xeb536000 and an unmap iova=0xeb530000 size=0x10000 500 µs earlier, the result is 0 correlated. Fix: match iova <= addr < iova + size, and add an arm with a fault mid-range.

With both fixed, the capture this PR's recipe produces can answer the ordering question. As it stands, every real capture reads "ordering not shown".

@bong-water-water-bong
bong-water-water-bong merged commit a0e37d7 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag-iova-not-pointer branch November 16, 2026 03:22
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Second commit: the window default was aimed three orders of magnitude too small, measured on a real burst

The owner's cold reboot at 02:56Z let me arm the instruments before the stream started, and at 03:20Z it began. A filtered probe on one faulting IOVA (0xe1e18000) over the boot's first 28 minutes:

unmaps of 0xe1e18000: 60      sizes: {0x1000: 60}      by comm: {'<idle>': 59, 'pi': 1}
fault t=1086.3 s  <- unmap of that page at 1080.7 s  (<idle>)   -5.65 s
fault t=1117.2 s  <- unmap at 1092.5 s               (<idle>)  -24.66 s
fault t=1117.2 s  <- unmap at 1092.5 s               (<idle>)  -24.71 s
fault t=1212.4 s  <- unmap at 1192.8 s               (<idle>)  -19.60 s

This tool's default was --window 5000 (µs) and on that very capture it printed "no fault shares an iova with an unmap inside the window: ordering not shown by this capture" -- a legitimate-looking negative about a capture that contains the correlation, at three orders of magnitude below the interval. The default is now 30 s and the help text carries the measured range (5–25 s).

Second half of the same lesson, and it cost a lane this morning: an unfiltered kprobe on iommu_unmap_fast wrote 579,000 events and left a probe filtered to that exact IOVA with zero hits -- the trace ring buffer is shared. The runbook paragraph now says so, next to its existing "prove the plumbing" rule: check grep -c '<probe>:' trace before trusting any capture, because a starved probe and a quiet kernel look identical from outside.

Scope unchanged: this PR is still the tool and its runbook page. The mechanism finding (<idle>/IRQ context, one pooled 4 KiB page, seconds-scale fallout) is on engine#2478, where the probe that now owns the buffer belongs.

@bong-water-water-bong bong-water-water-bong changed the title diag: the kprobe recipe named the wrong argument, and a pointer-shaped iova is now refused diag: the kprobe recipe named the wrong argument, and the correlation window is seconds (30 s default) Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant