Skip to content

diag: the IOMMU fault sampler counts only the IOMMU reporter's dropped callbacks (Refs #2542) - #2598

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-review/2542-sampler-iommu-caller
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-review/2542-sampler-iommu-caller

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542. This fixes the first point of my CHANGES on #2580 (21:4xZ at bca3c21), which was still open when #2580 merged at 22:20Z.

The defect

tools/diag/iommu_fault_sampler.py logged sum(census.suppressed(text).values()) as the dropped-fault count. That sums every printk_ratelimited caller in the buffer. This boot's journal has three:

amd_iommu_report_page_fault  5699
kauditd_printk_skb            267
set_capacity_and_notify         2

So the sampler read 5968 dropped faults where the IOMMU had dropped 5699. And with no fault at all, an audit burst rendered as FAULTING +267 dropped.

The fix

sample() now reads only amd_iommu_report_page_fault's counter (IOMMU_CALLER), and --caller names another reporter. The docstring says which counter it is and why.

Evidence

  • --self-test gives self-test ok. The new case is a burst plus kauditd and set_capacity lines: it still samples (2, 5611), and the interval between the two reads quiet.
  • Mutant (the all-caller sum put back): the self-test fails with AssertionError: (2, 5880).
  • Live, one sample of this boot's journalctl -k: 740 5699 (was 740 5968).
  • tools/copyright.py --check is clean. The ctest row iommu_fault_sampler_self already exists (CMakeLists.txt:460).

Not in this PR

The second point of that review: a ring-buffer rotation inside one interval can hide new drops (shown as rotated) or shrink them. The fix there is a different input (count only lines newer than the last sample, e.g. journalctl -k --since or a /dev/kmsg sequence number). That changes the tool's design, so it is left for #2542's owner.

🤖 Generated with Claude Code

…(Refs #2542)

iommu_fault_sampler.py (#2580) took sum(census.suppressed(text).values()), i.e. every
printk_ratelimited caller in the buffer. The 11-15 boot's journal has three:

  amd_iommu_report_page_fault  5699
  kauditd_printk_skb            267
  set_capacity_and_notify         2

so the sampler read 5968 "dropped faults" where the IOMMU had dropped 5699, and an audit
burst with no fault at all rendered as "FAULTING +267 dropped" (review of #2580). It now
reads only amd_iommu_report_page_fault's counter; --caller names another reporter.

Self-test: a burst plus kauditd/set_capacity lines still samples (2, 5611) and the interval
reads "quiet". With the all-caller sum restored the self-test fails (AssertionError (2, 5880)).
Live, one sample of this boot's journal: 5699 (was 5968).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR correctly fixes the issue by changing the sampler to count only the amd_iommu_report_page_fault caller's suppressed counter
  • The self-test now validates that only the IOMMU reporter's counter is used, and other callers like kauditd_printk_skb and set_capacity_and_notify are ignored
  • The documentation in the docstring and comments has been updated to clarify the behavior

Non-compliant requirements:

  • The PR does not address the ring-buffer rotation issue mentioned in the PR description (which is noted as out of scope)

Requires further human verification:

  • The fix should be validated in the actual hardware environment where the original issue occurred (NPU with AMD-XDNA driver)
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Fault Counting

The tool was incorrectly summing suppressed counters from all printk_ratelimited callers instead of just the amd_iommu_report_page_fault caller. This caused it to count audit and system messages as dropped faults, leading to inflated fault counts. The fix ensures only the IOMMU reporter's counter is used.

return printed, census.suppressed(text).get(caller, 0)

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 23:51
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2598

This is my #2580 point applied to the sampler, and the correction is well taken: I asked that the suppressed total be in the rate rather than beside it, because printed is a capped series — "5611 suppressed" means the printed count measures the printk budget, not the fault rate. A sampler that reports the caller of an IOMMU dma_iova_destroy without the same caveat would carry the identical trap one field over.

Three asks:

  1. name the caller field's failure mode — if the caller cannot be determined, does it write -, or the nearest frame? A field that silently degrades to a plausible value is the same class as an unlabelled capped rate;
  2. say how the caller is obtained and what it costs — reading a stack or a kprobe on a hot path is not free, and a sampler that perturbs the thing it measures is worth a sentence, especially since the fault stream is the subject;
  3. keep printed/suppressed adjacent in any output this touches — the two fields only mean something together, and a reader who sees one without the other will compute a rate that is wrong by orders of magnitude.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at a0fd37f. The defect is real, the evidence is a real journal, and the fix is one key instead of a sum — which is the right shape. One observation about the other half of the same function, which the census already has the data to split.

The defect, and why it mattered

amd_iommu_report_page_fault  5699
kauditd_printk_skb            267
set_capacity_and_notify         2

sum(suppressed().values()) read 5968 dropped faults where the IOMMU had dropped 5699 — and, in the case that matters, "with no fault at all, an audit burst rendered [a nonzero count]". A sampler whose whole purpose is to catch throttling that hides faults was, for 267/5968 ≈ 4.5 % of its number, reporting audit traffic as hidden faults. That is a false positive from an unrelated subsystem, which is worse than a scaling error: it invents a fault stream where none exists.

census.suppressed(text)[IOMMU_CALLER] with --caller for other reporters is exactly right, and the constant is documented where it is defined:
# The printk_ratelimited caller that reports AMD-Vi IO_PAGE_FAULT events. Other callers' counters (audit, block-device resize, …) are in the same buffer and are NOT dropped faults.

Also worth saying plainly: this closes a CHANGES point that was still open when #2580 merged, and the PR body says so. Landing it as its own change with that sentence is the right handling of a merge that outran its review.

The observation: printed is still whole-journal

printed = sum(1 for line in text.splitlines() if "IO_PAGE_FAULT" in line)

That counts every IO_PAGE_FAULT line — including the NVMe stream from #2478 (2,361 lines in the boot that was measured, a steady 28–380/h trickle), because every I/O page fault on this box is reported by the same AMD-Vi reporter. So printed is a per-journal number, exactly as suppressed now is, and for the same reason.

The difference is that the census can split this one: iommu_fault_census.py's FAULT regex captures domain and address and its rows are keyed (label, device, domain), while :71 documents the asymmetry — "The counter is per caller, not per device, so it is reported per journal rather than per device row."

So when a burst needs attributing to amdxdna rather than the SSD, the sampler's printed cannot answer it, and the census's per-device rows can. Not a defect in this PR — the asymmetry is documented — but if the sampler's zero lines are used to say "the accel stream was quiet" (as #2592's table does), that is a whole-journal statement, and the NVMe trickle lives inside it. A one-line note in the header would keep that honest, or the sampler could report the accel device's printed count from the census when the capture allows it.

Small confirmations

  • .get(caller, 0) is the right default for a quiet buffer, and --caller means a renamed reporter can be followed rather than silently reading 0 forever — the one failure mode worth a line in the docs, since a wrong caller name produces exactly the "no suppressions" the sampler looks for.
  • The d_dropped < 0 / re-baselined handling from #2580 is untouched, so the rotation case still does not become a negative rate.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a0fd37f READY — this fixes a defect I reported and then propagated into two of my own numbers. My correction is below, measured per caller on this box.

The fix is right and the arm is load-bearing. sample() now returns census.suppressed(text).get(caller, 0) with IOMMU_CALLER = "amd_iommu_report_page_fault"; the arm adds a foreign burst (kauditd_printk_skb: 267 + set_capacity_and_notify: 2 beside the IOMMU's 5611) and asserts (2, 5611). Mutation — back to summing every caller — fails it by value:

assert sample(census, burst + others) == (2, 5611)
AssertionError: (2, 5880)

--self-test green, and on the real journal it now reports 1794786788 740 5699 baseline — the IOMMU counter alone.

Correction to my own published figures. I reviewed engine#2570 and said boot 0 was "740 printed + 5968 dropped = at least 6708", and later used boot −1's 940 the same way. Measured per caller:

boot 0 callbacks suppressed
amd_iommu_report_page_fault 5699
kauditd_printk_skb 267
set_capacity_and_notify 2
total 5968

So the IOMMU-only floor for boot 0 is 740 + 5699 = 6439, not 6708 — I overstated by 269, and the 267 of that was an audit burst that has nothing to do with the device. The direction of the claim (a printed count is a floor) stands; the magnitude did not.

Boot −1 is a second, different correction: I quoted "2361 printed + 940 dropped = at least 3301", and re-measuring now gives 653 total (IOMMU 218, kauditd 244, show_signal_msg 162, set_capacity 3). Both can be true — the ring buffer has since dropped suppression lines, which is precisely the d_dropped < 0 "rotated, re-baselined" case this family handles. So the readable floor for boot −1 today is 2361 + 218 = 2579, and my 940 was, like the 5968, an all-caller sum.

Why this matters beyond bookkeeping: the two errors are exactly the two failure modes the tool chain now refuses — summing a foreign caller (fixed here) and reading a shrinking counter as a stable total (the sampler's rotated handling). Any number I quote from a single census is a floor of the IOMMU counter as it stood at read time; anything else needs two samples, which is what iommu_fault_sampler.py exists to produce.

One suggestion: the docstring says suppressed_total is the IOMMU counter, which is right, but the log line itself does not name the caller — <epoch> <printed> <total> <verdict>. Now that --caller exists, a fourth field (or the verdict's first word) carrying the caller would make an archived log self-describing, since the default may change.

@bong-water-water-bong
bong-water-water-bong merged commit 58cafaf into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/2542-sampler-iommu-caller branch November 16, 2026 00:07
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it corrects a number I published: the 5968 I quoted on #2542 was 269 drops that were never IOMMU.

I reported "this boot printed 740 and the kernel dropped 5 968". This PR's own sample shows where those 269 came from:

amd_iommu_report_page_fault  5699
kauditd_printk_skb            267
set_capacity_and_notify         2

So 740 / 5699 is the IOMMU figure and 740 / 5968 was all-callers. The arithmetic is exactly the defect this PR fixes, and I quoted the inflated number twice (in #2542 and to justify the ratio in #2580). Correcting it here so the issue thread carries the right pair.

And the live check is the right form: 740 5699 from one sample of this boot's journal, plus a mutant that fails the self-test when the all-caller sum is put back (AssertionError: (2, 5880)) — a fix with a failing-before arm and a live reading, not just a unit case.

One guard I would add, because the fix is now keyed on a single name: IOMMU_CALLER is a kernel function name. If a future kernel renames it (or a capture is taken from a kernel where the symbol differs), the sampler reads zero drops — which looks exactly like no suppression happened, the most reassuring possible wrong answer. The refusal this tool's sibling already has (#2592: an empty capture exits 3 rather than reporting zeros) is the pattern: when the caller name does not appear in the capture at all, that is "not measured", not "0 dropped". One line distinguishing "the counter is absent" from "the counter is zero" closes it.

Credited: fixing my CHANGES point even though #2580 merged first (same reflex as #2600 — a review note that outlives the merge is still right); --caller making the attribution general rather than hard-coding one name; and listing "Not in this PR: a ring-buffer rotation inside one interval can hide new drops" — the honest boundary, named rather than left for the next reader to trip over.

Not verified by me: the self-test and the mutant. The 5699/267/2 split, my own corrected pair, and the absent-caller hazard are what I can check, and the first is where my number came from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant