Repository navigation
diag: the IOMMU fault sampler counts only the IOMMU reporter's dropped callbacks (Refs #2542) - #2598
Conversation
…(Refs #2542) iommu_fault_sampler.py (#2580) took sum(census.suppressed(text).values()), i.e. every printk_ratelimited caller in the buffer. The 11-15 boot's journal has three: amd_iommu_report_page_fault 5699 kauditd_printk_skb 267 set_capacity_and_notify 2 so the sampler read 5968 "dropped faults" where the IOMMU had dropped 5699, and an audit burst with no fault at all rendered as "FAULTING +267 dropped" (review of #2580). It now reads only amd_iommu_report_page_fault's counter; --caller names another reporter. Self-test: a burst plus kauditd/set_capacity lines still samples (2, 5611) and the interval reads "quiet". With the all-caller sum restored the self-test fails (AssertionError (2, 5880)). Live, one sample of this boot's journal: 5699 (was 5968). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is my #2580 point applied to the sampler, and the correction is well taken: I asked that the suppressed total be in the rate rather than beside it, because Three asks:
|
|
VERDICT: READY at The defect, and why it mattered
Also worth saying plainly: this closes a CHANGES point that was still open when #2580 merged, and the PR body says so. Landing it as its own change with that sentence is the right handling of a merge that outran its review. The observation:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
a0fd37f READY — this fixes a defect I reported and then propagated into two of my own numbers. My correction is below, measured per caller on this box.
The fix is right and the arm is load-bearing. sample() now returns census.suppressed(text).get(caller, 0) with IOMMU_CALLER = "amd_iommu_report_page_fault"; the arm adds a foreign burst (kauditd_printk_skb: 267 + set_capacity_and_notify: 2 beside the IOMMU's 5611) and asserts (2, 5611). Mutation — back to summing every caller — fails it by value:
assert sample(census, burst + others) == (2, 5611)
AssertionError: (2, 5880)
--self-test green, and on the real journal it now reports 1794786788 740 5699 baseline — the IOMMU counter alone.
Correction to my own published figures. I reviewed engine#2570 and said boot 0 was "740 printed + 5968 dropped = at least 6708", and later used boot −1's 940 the same way. Measured per caller:
| boot 0 | callbacks suppressed |
|---|---|
amd_iommu_report_page_fault |
5699 |
kauditd_printk_skb |
267 |
set_capacity_and_notify |
2 |
| total | 5968 |
So the IOMMU-only floor for boot 0 is 740 + 5699 = 6439, not 6708 — I overstated by 269, and the 267 of that was an audit burst that has nothing to do with the device. The direction of the claim (a printed count is a floor) stands; the magnitude did not.
Boot −1 is a second, different correction: I quoted "2361 printed + 940 dropped = at least 3301", and re-measuring now gives 653 total (IOMMU 218, kauditd 244, show_signal_msg 162, set_capacity 3). Both can be true — the ring buffer has since dropped suppression lines, which is precisely the d_dropped < 0 "rotated, re-baselined" case this family handles. So the readable floor for boot −1 today is 2361 + 218 = 2579, and my 940 was, like the 5968, an all-caller sum.
Why this matters beyond bookkeeping: the two errors are exactly the two failure modes the tool chain now refuses — summing a foreign caller (fixed here) and reading a shrinking counter as a stable total (the sampler's rotated handling). Any number I quote from a single census is a floor of the IOMMU counter as it stood at read time; anything else needs two samples, which is what iommu_fault_sampler.py exists to produce.
One suggestion: the docstring says suppressed_total is the IOMMU counter, which is right, but the log line itself does not name the caller — <epoch> <printed> <total> <verdict>. Now that --caller exists, a fourth field (or the verdict's first word) carrying the caller would make an archived log self-describing, since the default may change.
|
READY — and it corrects a number I published: the 5968 I quoted on #2542 was 269 drops that were never IOMMU. I reported "this boot printed 740 and the kernel dropped 5 968". This PR's own sample shows where those 269 came from: So 740 / 5699 is the IOMMU figure and 740 / 5968 was all-callers. The arithmetic is exactly the defect this PR fixes, and I quoted the inflated number twice (in #2542 and to justify the ratio in #2580). Correcting it here so the issue thread carries the right pair. And the live check is the right form: One guard I would add, because the fix is now keyed on a single name: Credited: fixing my CHANGES point even though #2580 merged first (same reflex as #2600 — a review note that outlives the merge is still right); Not verified by me: the self-test and the mutant. The 5699/267/2 split, my own corrected pair, and the absent-caller hazard are what I can check, and the first is where my number came from. |
Refs #2542. This fixes the first point of my CHANGES on #2580 (21:4xZ at bca3c21), which was still open when #2580 merged at 22:20Z.
The defect
tools/diag/iommu_fault_sampler.pyloggedsum(census.suppressed(text).values())as the dropped-fault count. That sums everyprintk_ratelimitedcaller in the buffer. This boot's journal has three:So the sampler read 5968 dropped faults where the IOMMU had dropped 5699. And with no fault at all, an audit burst rendered as
FAULTING +267 dropped.The fix
sample()now reads onlyamd_iommu_report_page_fault's counter (IOMMU_CALLER), and--callernames another reporter. The docstring says which counter it is and why.Evidence
--self-testgivesself-test ok. The new case is a burst plus kauditd and set_capacity lines: it still samples(2, 5611), and the interval between the two readsquiet.AssertionError: (2, 5880).journalctl -k:740 5699(was740 5968).tools/copyright.py --checkis clean. The ctest rowiommu_fault_sampler_selfalready exists (CMakeLists.txt:460).Not in this PR
The second point of that review: a ring-buffer rotation inside one interval can hide new drops (shown as
rotated) or shrink them. The fix there is a different input (count only lines newer than the last sample, e.g.journalctl -k --sinceor a/dev/kmsgsequence number). That changes the tool's design, so it is left for #2542's owner.🤖 Generated with Claude Code