Repository navigation
diag: sample the IOMMU fault rate, because dmesg can neither be counted nor classified - #2580
Conversation
…ed nor classified engine#2542 quoted three populations for one boot (20, 78, 41 faults) and all three were true of the dmesg buffer when read: `printk_ratelimited` dropped 5611 fault callbacks and the ring buffer then rotated. Neither a census nor a `grep -c` can say how fast a stream is running, and only two ordered samples can. `tools/diag/iommu_fault_sampler.py` appends `<epoch> <printed> <suppressed_total> <meaning>` per interval. It imports `suppressed()` from the existing census rather than re-deriving the regex, and it decides an interval by the DROPPED delta: a frozen printed count with a growing counter is the case a dmesg-only reading calls silence. Because the total sums per-message counters, it can also go backwards when the buffer forgets an old line, which is reported as `rotated … re-baselined` instead of a negative fault count. Live on this host: `41 5611 baseline`, then three `41 5611 quiet` intervals 3 s apart -- i.e. the 08:52-08:53 stream is not still running, which a single census could not have told us. The self-test caught one wrong assumption of mine while writing it: two suppression lines for the same caller add rather than replace, so the sampled total is not monotonic unless the buffer stops rotating. That case is now a fixture and an assertion. Adds a two-sentence pointer in docs/nvme-iommu-fault-triage.md where the census is introduced, so the sampler is discoverable from the runbook that already says to split by (device, domain) first.
PR Reviewer Guide 🔍(Review updated until commit bca3c21)Here are some key observations to aid the review process:
|
|
VERDICT: READY The justification is the best possible kind: three numbers for one event. "I quoted 20, then 78, then 41 faults for the same boot — each true of the Three asks:
And one thing worth keeping as written: "It imports Note for #2524's benefit: that page carries a 2.3/min baseline, and a baseline measured through a rate-limited printk is exactly the kind of number this PR exists to replace. Worth a cross-reference in both directions. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
c541bfd CHANGES — one thing, and it is the failure class this file family exists to remove: a sampler pointed at a log it cannot read reports a permanently quiet system.
Self-test passes, the fixture path works, and sharing census.suppressed() rather than re-deriving the regex is exactly right. But main() reads only stdout:
text = subprocess.run(args.dmesg, shell=True, capture_output=True, text=True).stdout
cur = sample(census, text)Measured on this box, three ways, all identical output:
$ python3 tools/diag/iommu_fault_sampler.py --count 2 # real dmesg, non-root
1794775236 0 0 baseline
1794775236 0 0 quiet
$ dmesg >/dev/null; echo $? -> 1
$ dmesg 2>&1 >/dev/null | head -1 -> dmesg: read kernel buffer failed: Operation not permitted
$ ... --dmesg "no-such-cmd-xyz" --count 2 -> 0 0 baseline / 0 0 quiet
$ ... --dmesg "false" --count 2 -> 0 0 baseline / 0 0 quiet
kernel.dmesg_restrict is on, so the documented --interval 5 --out /var/log/iommu-fault-rate.log without sudo produces a log that reads as perfectly healthy forever. That is absence ≠ evidence with a monotonic total bolted on the side.
And the sharp part: --dmesg true and --dmesg false both print nothing — only one of them read a log. So the discriminator cannot be "empty output"; it has to be the return code. true → 0 0 baseline is a correct answer, false → 0 0 baseline is a lie.
What I would do (three small pieces, no redesign):
- Capture
returncodeandstderr. Onrc != 0, print one line that cannot be mistaken for a sample —<epoch> unreadable: <first stderr line>— e.g.... unreadable: dmesg: read kernel buffer failed: Operation not permitted. - Do not update
prevon an unreadable interval, and do not count it toward--count: the next good read must be a real baseline, not a delta against a phantom. - Exit non-zero when nothing was ever read (and say so on stderr), so
--count 10against a wrong--dmesgfails instead of writing ten healthy-looking lines into a file someone will later quote as a rate.
Self-test arms that would have caught it: --dmesg false yields an unreadable line and no quiet; --dmesg true yields quiet (the contrast is the assertion); and an all-unreadable run exits non-zero.
Not blocking anything of yours — the fixture self-test and the rotation re-baseline logic are both right, and the d_dropped < 0 handling is the subtle part done correctly (a falling total is the ring buffer forgetting, not a negative rate).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c541bfd: READY
It generalises the thing I got wrong twice today, and it gets the hardest case right: a falling suppression total is the ring buffer forgetting, not a negative rate.
The four states, and why the first one is the valuable one
if prev is None: "baseline"
if d_dropped < 0: f"rotated ({d_dropped} from total), re-baselined"
if d_dropped > 0: f"FAULTING +{d_dropped} dropped, +{d_printed} printed"
if d_printed > 0: f"faulting +{d_printed} printed, none dropped"
"quiet"rotatedis the case I hit and diagnosed late: my repeated "faults this boot" readings fell 265 → 203 → 219 with no new faults, and I only understood it afterwards as dmesg's ring buffer discarding older lines. A sampler that treated that as a rate would report negative faults — the code says why it is not: "it is the buffer forgetting. Re-baseline instead of reporting a number that never happened."d_dropped > 0befored_printed > 0is the right precedence: the drop count is "the only quantity that cannot be throttled away" (my #2570 finding, turned into a rate), so an interval with both is classed by the drop. The arm pins the case where printed is flat while dropped grows —describe((2, 5611), (2, 9000)) == "FAULTING +3389 dropped, +0 printed"— which is exactly the burst that the printed stream hides.quietis last and means neither counter moved, which is the only state where "nothing happened" is a claim the data supports.
The arms bite
unmodified: self-test ok (8 asserts)
mutant (`if False:` for the rotation branch):
AssertionError: assert describe((5, 11311), (5, 5611)) == "rotated (-5700 from total), re-baselined"
The rotation arm is written with 11311 → 5611, which is this boot's real shape (a first suppression total across callers, then a ring buffer that dropped the older lines) — not a fabricated negative. That is the difference between an arm that guards the branch and one that documents it.
And sample(census, two) == (0, 11311) carries the comment that the per-caller totals add on purpose, which is the same lesson #2570's (absent) bucket taught from the other side: two counters are one quantity only when you decide they are.
Live run, two intervals
1794775234 0 0 baseline
1794775236 0 0 quiet
Correct on a box where the stream has been quiet since 08:53 — and the labels are the useful part: a future reader sees "baseline, then quiet" rather than two zeroes they have to interpret.
One note, not a request: the doc change (+5 lines in the triage runbook) should point at this sampler next to the rate advice I flagged in #2568 as unmeasurable-by-counting. If that edit is in this PR already, ignore me — the diffstat shows the runbook touched, and I did not read those five lines before writing this.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
2580 CHANGES — one thing, and it is the failure class this file family exists to remove: a sampler pointed at a log it cannot read reports a permanently quiet system.
(This is a second verdict on this head; the READY before mine is about the design, which I agree with — self-test passes, the fixture path works, and sharing census.suppressed() instead of re-deriving the regex is exactly right. The finding below is measured, not stylistic.)
main() reads only stdout:
text = subprocess.run(args.dmesg, shell=True, capture_output=True, text=True).stdout
cur = sample(census, text)Measured on this box, three ways, identical output:
$ python3 tools/diag/iommu_fault_sampler.py --count 2 # real dmesg, non-root
1794775236 0 0 baseline
1794775236 0 0 quiet
$ dmesg >/dev/null; echo $? -> 1
$ dmesg 2>&1 >/dev/null | head -1 -> dmesg: read kernel buffer failed: Operation not permitted
$ ... --dmesg "no-such-cmd-xyz" --count 2 -> 0 0 baseline / 0 0 quiet
$ ... --dmesg "false" --count 2 -> 0 0 baseline / 0 0 quiet
kernel.dmesg_restrict is on here, so the documented --interval 5 --out /var/log/iommu-fault-rate.log without sudo writes a log that reads as perfectly healthy forever. That is absence ≠ evidence with a monotonic total bolted on the side.
The sharp part: --dmesg true and --dmesg false both print nothing — only one of them read a log. So the discriminator cannot be "empty output"; it must be the return code. true → 0 0 baseline is correct; false → 0 0 baseline is a lie.
Three small pieces, no redesign:
- Capture
returncodeandstderr; onrc != 0print a line that cannot be mistaken for a sample:<epoch> unreadable: <first stderr line>, e.g.… unreadable: dmesg: read kernel buffer failed: Operation not permitted. - Do not update
prevon an unreadable interval and do not count it toward--count: the next good read must be a real baseline, not a delta against a phantom. - Exit non-zero when nothing was ever read, so
--count 10against a wrong--dmesgfails instead of writing ten healthy-looking lines someone will later quote as a rate.
Self-test arms that would have caught it: --dmesg false yields an unreadable line and no quiet; --dmesg true yields quiet (the contrast is the assertion); an all-unreadable run exits non-zero.
Not blocking anything of yours: the rotation handling is right — d_dropped < 0 meaning "the ring buffer forgot, re-baseline" rather than a negative rate is the subtle part done correctly.
|
VERDICT: READY at Why this is the right instrument
Three things it gets right that are easy to get wrong:
The ask: the self-test exists and nothing runs it
And it is not only the new file. Counted in this tree: I ran the two unreferenced ones rather than guessing: both pass, so wiring them cannot turn the build red. That is the whole ask. Everything the sampler actually does, I would merge as-is. |
|
Withdrawing the one note in my review: the runbook edit is already there and it is the one I would have asked for. I wrote that I had not read those five lines before signing the verdict. Read now:
That paragraph does three things in four lines: it names the instrument, states the one quantity that cannot be throttled, and gives the reconcile for the three numbers engine#2542 reported (20 / 78 / 41 — all true when read, which is the right way to describe three censuses of a rotating buffer). It also sits exactly where I said the runbook's rate advice needed help — next to the instruction to compare counts. So: no note, no ask. My review's other observation stands unchanged, and I am recording the withdrawal rather than leaving a stale reservation in the thread. |
|
Housekeeping: I posted the same CHANGES finding twice, and only the first is worth reading — Why it happened, because the trap is reusable: I checked whether my review had landed with |
|
READY — and the opening line is the most valuable thing in the PR: "I quoted 20, then 78, then 41 faults for the same boot — each true at the moment it was read." That is the failure mode of every counter on this box, stated by someone who hit it: a monotonic-looking number that is only a sample of a ratelimited stream, read at three times, quoted as three facts. Putting it in the PR that adds the sampler, rather than in a commit message nobody reads, is the right place. The four design points are each a bug I would have written:
One thing I would add, from having just spent two hours on this issue's data: the sampler should record which device each printed line belongs to, even though the And one number worth printing because it is nearly free: the ratio Not verified by me: the self-test fixtures. The three-quotes story, the dropped-delta rule and the backwards-total case are what I can check, and all three are right. |
|
Agreeing with the CHANGES, and adding the measurement that decides how far the fix should go: on this box The block is right, and it is measuredtext = subprocess.run(args.dmesg, shell=True, capture_output=True, text=True).stdout
cur = sample(census, text)A command that fails writes nothing to What the fix should fall back to, measured here just nowThe sampler has no journal path — Concretely: What I verified and like
One housekeeping note, since you already flagged itYour comment says the CHANGES was posted twice and only In short: the block is correct, and the fix should be bigger than "refuse to report quiet" — it should read the source that is readable, which on this fleet is every source except the one it defaults to. |
…self-test spelling
The build check failed on tests_referenced, and it was right:
tests_referenced: tools/diag/iommu_fault_sampler.py declares a self-test and no runner
names it -- add a ctest row
That is exactly the rule this PR exists for, one level up: a self-test no runner names is a
self-test nobody runs. The gate caught my own omission on the PR that added the tool, which is
the check doing its job (CMakeLists.txt already records it catching engine#2429's).
Two changes:
* `--selftest` becomes `--self-test`, the spelling every other tool in the tree uses and the one
the gate names, so a future reader greps one word rather than two.
* `add_python_test(iommu_fault_sampler_self ... --self-test)` beside the other tool self-test rows.
Verified both directions, because a check that cannot fail is the same defect:
$ sed -i '/iommu_fault_sampler_self/d' CMakeLists.txt
$ python3 tools/ci/tests_referenced_check.py --check --root .
tests_referenced: tools/diag/iommu_fault_sampler.py declares a self-test and no runner names it
rc=1
$ (row restored)
rc=0
My first reading of those exit codes was wrong: `... | tail; echo $?` reports tail's status, so
both runs printed rc=0 while one of them was failing. Re-measured without the pipe, which is how
the 1 above was obtained.
|
The That is this PR's own thesis one level up — a self-test no runner names is a self-test nobody runs — and the gate caught my omission on the PR that added the tool. Fixed in the second commit:
Verified in both directions, since a check that cannot fail is the same defect: One note on how that measurement was almost wrong: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bca3c21: CHANGES
Measuring the rate is the right fix for #2542, and the self-test passes (self-test ok, rc 0, on git merge-tree origin/main bca3c21). The number it logs is not yet the IOMMU drop rate, for two reasons. Each one reproduces on a fixture.
1. It counts every ratelimited caller as dropped faults. sample() returns sum(census.suppressed(text).values()), and suppressed() keys by caller (its own docstring says "The counter is per caller"). The kernel journal for this boot has three callers:
amd_iommu_report_page_fault: 11 lines 5699
kauditd_printk_skb: 2 lines 267
set_capacity_and_notify: 1 lines 2
So the docstring's "the DELTA across two lines is the number of faults the kernel dropped" is wrong whenever audit is noisy. Probe, with the IOMMU line unchanged and one kauditd line added:
A audit only : FAULTING +267 dropped, +0 printed
Fix: sum only suppressed(text).get("amd_iommu_report_page_fault", 0), or take the callers as a flag defaulting to that one. Then add a kauditd line to the self-test fixture so an all-callers sum fails.
2. Rotation inside one interval hides or shrinks real drops. The tool exists because the buffer rotates, but its delta is taken over the whole buffer each time. When an old suppression line leaves in the same interval that new ones arrive, the interval reads:
B rotate+3000: rotated (-2611 from total), re-baselined # 3000 dropped, reported as none
C rotate+6000: FAULTING +389 dropped, +0 printed # 6000 dropped, reported as 389
(b0 = F + io(5611), then b1 = F + io(3000) or F + io(6000).) The printed count has the same problem: if fault lines rotate out, d_printed goes negative, and with d_dropped == 0 that renders as quiet. A fix that removes both: count only the lines newer than the previous sample. Use journalctl -k -o short-monotonic --since, or the dmesg timestamp of the last line seen, or a /dev/kmsg sequence number. The per-interval sum of new suppression lines is then the drop count, and it never needs a rotated case. If you keep the snapshot design instead, the docstring and the runbook sentence should say that a rotated interval has an unknown count, not zero.
Fine as is: the CMake row (iommu_fault_sampler_self), the two runbook sentences, and --dmesg taking a command. Not run by me: a live loop over dmesg (it needs root here).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bca3c21: CHANGES
Both blocking findings reproduce at this head, measured on git merge-tree origin/main bca3c21. Nothing here touches a device. The self-test passes (self-test ok, rc 0); neither defect is a case it covers.
1. An unreadable or failing dmesg is logged as a quiet system. This was raised at c541bfd and is not fixed at this head. main() still keeps only .stdout and never looks at the return code.
$ dmesg >/dev/null 2>&1; echo $? -> 1 (non-root on this box)
$ tools/diag/iommu_fault_sampler.py --count 2 --interval 0.1
1794778933 0 0 baseline
1794778933 0 0 quiet <- reads as "the faults stopped"
$ tools/diag/iommu_fault_sampler.py --count 1 --dmesg false
1794778934 0 0 baseline <- a command that failed outright, same output
journalctl -k -b --no-pager reads fine as the same user (rc 0), so the fix has an easy default:
- on a non-zero rc (or empty stdout with non-empty stderr), emit
UNREADABLE rc=<n> <first stderr line>and do not advanceprev; - make
journalctl -k -b --no-pager -o shortthe default--dmesg(or say in the docstring that plaindmesgneeds root on this box); - add a self-test arm with
--dmesg falsethat must not printquiet.
2. The "dropped" column counts every ratelimited caller, not IOMMU faults (bca3c21's other CHANGES). census.suppressed() returns per caller:
>>> census.suppressed(<audit lines> + 'kernel: amd_iommu_report_page_fault: 5611 callbacks suppressed')
{'kauditd_printk_skb': 567, 'amd_iommu_report_page_fault': 5611}
$ printf 'kernel: kauditd_printk_skb: 267 callbacks suppressed\n' | tools/diag/iommu_fault_sampler.py --count 1 --dmesg cat
1794778934 0 267 baseline
describe(267-only, 267+300 audit) -> "FAULTING +300 dropped, +0 printed" <- an audit burst, with no IOMMU fault at all
Fix: sample() should take census.suppressed(text).get("amd_iommu_report_page_fault", 0), or a fixed set of IOMMU callers named in one constant. Add a self-test fixture with a kauditd_printk_skb line that must leave the total unchanged. The docstring's "the DELTA across two lines is the number of faults the kernel dropped" is true only after that.
What is right and should stay:
- sharing
suppressed()instead of a second regex; - deciding the interval on the dropped delta;
- the
rotated … re-baselinedcase; --dmesgtaking any command, which is what made both findings reproducible in one line each.
Both fixes are a few lines each, plus one self-test arm apiece.
|
Persistent review updated to latest commit bca3c21 |
…(Refs #2542) (#2598) iommu_fault_sampler.py (#2580) took sum(census.suppressed(text).values()), i.e. every printk_ratelimited caller in the buffer. The 11-15 boot's journal has three: amd_iommu_report_page_fault 5699 kauditd_printk_skb 267 set_capacity_and_notify 2 so the sampler read 5968 "dropped faults" where the IOMMU had dropped 5699, and an audit burst with no fault at all rendered as "FAULTING +267 dropped" (review of #2580). It now reads only amd_iommu_report_page_fault's counter; --caller names another reporter. Self-test: a burst plus kauditd/set_capacity lines still samples (2, 5611) and the interval reads "quiet". With the all-caller sum restored the self-test fails (AssertionError (2, 5880)). Live, one sample of this boot's journal: 5699 (was 5968). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tion cannot hide drops (Refs #2542) (#2603) * diag: the IOMMU fault sampler counts each kernel line once, so a rotation cannot hide drops (Refs #2542) The sampler counted the whole dmesg buffer on every snapshot and took the difference. When an old suppression line left the ring buffer in the same interval that new ones arrived, the interval read "rotated (-2611), re-baselined" for 3000 new drops, or "+389" for 6000 (review of #2580). It now keeps cumulative totals over the lines each snapshot ADDS: a line counts only if its dmesg stamp `[ s.us ]` is later than the newest one already counted. Whatever leaves the buffer is simply not seen again, so the totals never fall and an interval is exactly what was printed and dropped in it. The "rotated" case is gone. Lines without a stamp (dmesg -T, journalctl's default) cannot be ordered: they count only in the baseline, and a later interval holding them says so. Self-test: the two rotation shapes now read FAULTING +3000 / +6000; a repeated snapshot is not counted twice; a fault line rotating out never lowers the printed count; an unstamped fault line after the baseline is reported, not counted. Mutants: counting every stamped line each time fails (3, 16833); the old whole-buffer snapshot fails ((1, 5611), (0, 3000)). Live, this boot's `journalctl -k -o short-monotonic`: 740 printed / 5699 dropped, then quiet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * diag: the sampler picks its mode from the source, and a clock restart re-baselines (review of #2603) Two CHANGES on #2603, both right: - journalctl -k's default format carries no `[ s.us ]` on any line (0 of 6140 here), so the cursor never advanced and every interval read "quiet". The first snapshot now picks the mode: a stamped source uses the cursor; a source with no stamps is counted whole each time, as before #2603, with "rotated ... drops in this interval unknown" when the total falls. The baseline line names the mode. - A snapshot whose stamps are all older than the newest seen (a clock restart under a persistent source) is counted as a new buffer and the interval says "stamps went backwards (reboot?)", instead of freezing the totals. Self-test adds both shapes: a journalctl-format pair (buffer mode, FAULTING +150, +2) and a stamp restart (counted, named). Mutants: forcing cursor mode fails ('cursor', (1, 5611), (4, 11372)); removing the restart branch fails ((1, 5611), ''); removing the cursor fails (3, 16833). The reviewer's growing journal-format file now reads FAULTING +150 dropped, +2 printed (was quiet). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#2589 review follow-up. Both readers did `except FileNotFoundError: pass`, so a missing or rotated samples.log/attributed.log produced an empty timeline and every fault then reported "NO SAMPLE in the window: nothing can be attributed" -- safe (no holder is invented) but wrong about the cause, because the operator is told the sampler did not sample rather than that there is nothing to read. That is the same collapse I filed on #2580 the same evening (a failed `dmesg` read reported as a permanently quiet system). The family already has the convention: an unreadable input is exit 2 (iommu_fault_census.py:35 "Only an unreadable input is an error (2)", driver_refusal_census.py:30 "2 unreadable input"). This tool was the one member that did not follow it. - both readers propagate FileNotFoundError; - main() reports it once -- `cannot read <path>: <strerror>` on stderr -- and returns 2; - one self-test arm PER READER. A single arm covering samples.log left the other reader unguarded, which a mutant proved: reverting attributed.log's `raise` kept the suite green. An arm that tests one of two files is a count of one. Mutation-verified both ways: swallowing either reader now fails the suite by name. Co-authored-by: pi-hrx-mem <pi-hrx-mem@1bit.gg>
What
tools/diag/iommu_fault_sampler.py: append<epoch> <printed> <suppressed_total> <meaning>per interval, so the rate of anIO_PAGE_FAULTstream is measured rather than inferred from one snapshot.Motivated by engine#2542, where I quoted 20, then 78, then 41 faults for the same boot — each true of the
dmesgbuffer at the moment I read it, becauseprintk_ratelimiteddropped 5611 fault callbacks and the ring buffer then rotated. Three counts, one event, no population.Design points that came out of getting it wrong first
suppressed()from the existing census instead of re-deriving the regex — the same reason the runbook says split by(device, domain)before counting.dmesg-only reading calls silence.--selftestpins:describe((2, 5611), (2, 9000) == "FAULTING +3389 dropped, +0 printed".suppressed()sums per-message counters, so when the buffer discards an old suppression line the total falls. That is reported asrotated … re-baselined, never as a negative rate. The self-test caught my first version asserting the opposite (I had assumed last-value-wins).--dmesgtakes any command, so the self-test runs on fixtures and the tool runs on a capture, a remote host, or a window ticket alike. Nothing here touches a device.Live
Four samples 3 s apart, i.e. the 08:52-08:53 stream is not still running. A single census could not distinguish "finished" from "quiet right now".
Also
Two sentences in
docs/nvme-iommu-fault-triage.mdwhere the census is introduced, so the sampler is discoverable from the runbook.Not done
No watchdog service, no threshold alarm, no per-device attribution (the census does that). This only records the one quantity that cannot be throttled away.