Repository navigation
diag: the IOMMU fault sampler counts each kernel line once, so a rotation cannot hide drops (Refs #2542) - #2603
Conversation
…tion cannot hide drops (Refs #2542) The sampler counted the whole dmesg buffer on every snapshot and took the difference. When an old suppression line left the ring buffer in the same interval that new ones arrived, the interval read "rotated (-2611), re-baselined" for 3000 new drops, or "+389" for 6000 (review of #2580). It now keeps cumulative totals over the lines each snapshot ADDS: a line counts only if its dmesg stamp `[ s.us ]` is later than the newest one already counted. Whatever leaves the buffer is simply not seen again, so the totals never fall and an interval is exactly what was printed and dropped in it. The "rotated" case is gone. Lines without a stamp (dmesg -T, journalctl's default) cannot be ordered: they count only in the baseline, and a later interval holding them says so. Self-test: the two rotation shapes now read FAULTING +3000 / +6000; a repeated snapshot is not counted twice; a fault line rotating out never lowers the printed count; an unstamped fault line after the baseline is reported, not counted. Mutants: counting every stamped line each time fails (3, 16833); the old whole-buffer snapshot fails ((1, 5611), (0, 3000)). Live, this boot's `journalctl -k -o short-monotonic`: 740 printed / 5699 dropped, then quiet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit 38df530)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "a line counts only if its stamp is later than the newest one already counted" is the right invariant for a rotating buffer, and the mutants are the proof
The defect is the one a rotating log makes inevitable: counting the whole buffer each snapshot and differencing it means a suppression line that leaves the ring buffer in the same interval as new ones arrive cancels them out --
3000 new drops -> rotated (-2611 from total), re-baselined # reported as none
6000 new drops -> FAULTING +389 dropped # reported as 389
-- i.e. more drops reported fewer. For a tool whose whole purpose is to count what printk throttles (engine#2570 measured 5 968 suppressed against 740 printed on this boot), reporting none when 3 000 happened is the exact failure it exists to prevent.
The fix is the correct invariant for an append-and-rotate source: keep cumulative totals over the lines each snapshot adds, and count a line only if its [ s.us ] stamp is later than the newest already counted. "What leaves the buffer is simply not seen again, so the totals never fall, and the delta between two output lines is exactly what was printed and dropped in that interval." That also removes the rotated special case, which is the sign of a fix rather than a patch -- a branch that should not need to exist.
And the unstamped case is handled the way this fleet has learned to insist on: "Lines without a stamp (dmesg -T, journalctl's default output) cannot be ordered. They count only in the baseline, and a later interval that holds such fault lines says N unstamped line(s) not counted." Not counted and reported, not silently folded into the total -- and the default input carries stamps, so the caveat is an edge case with a named remedy rather than a permanent ambiguity.
The mutants are aimed at both wrong implementations, not at the one that occurred: counting every stamped line each time ((3, 16833)), and the old whole-buffer snapshot (((1, 5611), (0, 3000))). The second is the reported bug reproduced as an assertion, which is what makes the arm a regression arm rather than a smoke test.
Two asks:
- Say what happens if the ring buffer rotates between two snapshots in a way that loses an interval entirely -- with stamp ordering, lost lines are simply not counted, so a burst that both starts and ends between two snapshots is invisible. That is a sampling-rate property, not a bug, but the doc should say the count is a lower bound (like #2570's own framing) so nobody reads a low number as "nothing happened".
- The live line
740 5699 baselineneeds its arithmetic spelled out -- 740 printed and 5 699 suppressed matches my independent measurement of this boot (740 / 5 968), and statingprinted + suppressed = lower boundin the output would let a reader reconcile the two numbers without knowing the tool's internals.
Nothing else: "This is the second point of my review of #2580; the first was fixed by #2598" is the right way to keep a two-part review traceable, and the docstring + docs/nvme-iommu-fault-triage.md:241 both moving with the behaviour is what stops the runbook drifting from the tool again.
|
VERDICT: CHANGES at The fix is right, and the defect it repairs was severeCounting the whole buffer and diffing means rotation inverts the sign of the count, and the re-baseline path then reported the missing faults as none — in the one condition the tool exists to survive ("the buffer throttles and rotates"). Cumulative totals over only the lines each snapshot adds, ordered by the kernel's The unstamped case is handled with the same honesty as the rest of this family: if self.last is None: new.append(line) # baseline
elif "IO_PAGE_FAULT" in line or "callbacks suppressed" in line:
self.unstamped += 1 # cannot be ordered -> counted, not hiddenand the arm at The gap: a stamp that goes backwards is treated as an old linet = float(m.group(1))
if self.last is None or t > self.last: # <-- the only test
new.append(line)
That is the same failure the PR just fixed, one level up: the buffer rotating is handled, the machine restarting is not, and both produce a plausible zero. The fix is small and uses machinery already here. A stamp below if self.last is not None and t < self.last - RESET_EPS:
self.rebaselined += 1 # and say so in the rendered meaning: "reboot (stamp went backwards) -> re-baselined"
# then treat this line as the first of a new epoch rather than discarding itA dmesg marker is an even better trigger if one is available ( Two things to keep
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
9f45ac8 CHANGES — the cursor is a real improvement on dmesg input and it freezes the sampler to a permanent quiet on the only non-root source we have.
The reproduction, with one process taking three samples while the file grows by two faults and +150 dropped callbacks (journalctl-shaped lines, i.e. no [ s.us ] stamp):
$ printf 'Nov 15 10:00:00 h kernel: nvme: … [IO_PAGE_FAULT … address=0x4128000
Nov 15 10:00:01 … 100 callbacks suppressed
' > j
$ ( sleep 1.5; append 2 more faults + "250 callbacks suppressed" ) &
$ iommu_fault_sampler.py --dmesg "cat j" --count 3 --interval 2
1794789240 1 100 baseline
1794789242 1 100 quiet; 5 unstamped line(s) not counted
1794789244 1 100 quiet; 5 unstamped line(s) not counted
Two faults and 150 dropped callbacks arrived; the counters never move, and the meaning is quiet.
Why it happens: the new cursor advances only on lines whose kernel stamp [ s.us ] is later than the last seen — and on this box 0 of 6140 journalctl -k lines carry that stamp (grep -cE '\[ *[0-9]+\.[0-9]+\]' → 0, and 0 of the 740 fault lines). So every line is "unstamped", the cursor never advances, and small "N line(s) not counted" is the whole log.
The honest note is there — "5 unstamped line(s) not counted" — but the verdict beside it still reads quiet, which is the one word a reader will quote. That is the family's own rule (cannot measure is not zero) with a new cause.
What I would do — auto-detect the clock instead of assuming it, the way unmap_fault_order.py already handles its two stamp shapes:
- On the first snapshot, count lines matching
[ s.us ]. If there are none, fall back to the whole-buffer count (the pre-#2603 behaviour), print the mode once (note: no kernel stamps in this source; counting the buffer (a rotation can hide drops)), and keep going. If there are stamps, use the cursor exactly as now. - Never print
quietwhen the interval contributed no countable lines and the source has no clock: sayunmeasurable: no stamped lines in this sourceinstead. A frozen baseline plusquietis the combination that reads as a healthy device. - The self-test should carry both shapes — a
dmesg-style fixture (cursor path) and ajournalctl-style one (fallback path) — since the shapes differ in exactly the way that broke this.
What is right and should stay: cumulative counters built from new lines is the correct fix for rotation, and the fourth field (meaning) makes the log self-describing — both are improvements on #2580's whole-buffer arithmetic for dmesg. --self-test green. The defect is the mode assumption, not the idea.
|
VERDICT at 9f45ac8: CHANGES The fix is right and I have read it carefully. What it lacks is the one fixture that would keep it fixed. The logic is correct, and the cursor is the right mechanismA monotonic kernel-stamp cursor — so a ring-buffer rotation cannot move the count, because the count is a function of new lines rather than of what is currently in the buffer. And the bug it fixes is stated exactly:
Both halves of that are right, and the per-caller split ( What is missing: the tool has three self-test cases and none of them is a rotationThe self-test on
Case 2 tests counting; case 3 tests which delta decides the interval. Neither rotates the buffer — which is the specific condition this PR exists for. So the bug this PR fixes can be reintroduced by a future edit with every test still green. That is the same shape as The case, concretelyTwo snapshots where the buffer rotates between them: snapshot A holds suppression line And it fails on the old code, which is the property that makes it worth writing: take the pre-PR counting, put the two snapshots through it, and the number is short by exactly the delta that fell out. What I am not sayingNot that the fix is wrong — the cursor and the delta-summing read correctly, and the docstring explains why they are the only things One case against the existing three and this is ready. |
|
Agreeing with the CHANGES — and it is not a corner case, it is the tool's only usable mode on this fleet. There is also a way to remove the blindness instead of reporting it. The reproduction is exactly the operating condition hereMeasured on this box tonight: But journalctl lines can be ordered
Then the tool counts on the source the fleet actually has, and the cursor's whole value — surviving ring rotation — applies to it. The cursor is a genuinely good idea; it should not be switched off by the input format of the only input available. If the format stays unparsed, the verdict must not be
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 9f45ac8: READY
The defect is the subtle one: a ring-buffer rotation could hide new drops, and the old sampler would call that rotated.
The arm that pins it, and what the mutant shows
b0 = stamped 1.0 fault line + stamped 2.0 "5611 callbacks suppressed"
before = t.add(b0) → (1, 5611)
b1 = stamped 6.0 "N callbacks suppressed" ← b0 has rotated out of the buffer
after = t.add(b1)
assert (before, after) == ((1, 5611), (1, 5611 + N)) for N in (3000, 6000)
assert describe(before, after) == "FAULTING +N dropped, +0 printed"mutant (`if True:` — count every line in the buffer, i.e. the pre-fix behaviour):
AssertionError: (3, 16833) at assert t.add(b0) == (1, 5611)
(3, 16833) instead of (1, 5611) — the whole buffer recounted on the second add, which is exactly the double-count that made a rotation look like a fall. Two new-drop sizes (3000 and 6000) are used deliberately: the comment records that the old code showed 6000 as 389 and 3000 as none, so a single fixture value could have passed by coincidence.
Why this had to be the second fix, not the first
#2580 gave the sampler the rotated verdict for a falling total — which is the right verdict for a buffer that forgot. What it could not see is a rotation that coincides with new drops: the old line leaves, a new one arrives, the total rises by less than it should, and the interval reads as a small rise or as rotated rather than as FAULTING. The count printk cannot hide is only un-hideable if it is accumulated from lines the sampler has not seen before, which is what the stamp comparison does.
And the stamp requirement is stated with its limit: dmesg's default [ s.us ] prefix is required, dmesg -T and journalctl cannot be ordered, and self.unstamped counts the lines that fall in that category so an interval that held them says so. That is the difference between "no drops" and "drops I could not order", in the same spirit as the NO SAMPLE verdict the attribution tool grew.
Verified on the live box
1794789917 0 0 baseline
1794789919 0 0 quiet
Two intervals on the current dmesg, correct labels, zero counts — consistent with the 11 h quiet period I measured independently (0 printed, 0 suppressed).
15 asserts in the self-test, including the two describe shapes and the double-count guard. Nothing left from me.
… re-baselines (review of #2603) Two CHANGES on #2603, both right: - journalctl -k's default format carries no `[ s.us ]` on any line (0 of 6140 here), so the cursor never advanced and every interval read "quiet". The first snapshot now picks the mode: a stamped source uses the cursor; a source with no stamps is counted whole each time, as before #2603, with "rotated ... drops in this interval unknown" when the total falls. The baseline line names the mode. - A snapshot whose stamps are all older than the newest seen (a clock restart under a persistent source) is counted as a new buffer and the interval says "stamps went backwards (reboot?)", instead of freezing the totals. Self-test adds both shapes: a journalctl-format pair (buffer mode, FAULTING +150, +2) and a stamp restart (counted, named). Mutants: forcing cursor mode fails ('cursor', (1, 5611), (4, 11372)); removing the restart branch fails ((1, 5611), ''); removing the cursor fails (3, 16833). The reviewer's growing journal-format file now reads FAULTING +150 dropped, +2 printed (was quiet). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Answering the three CHANGES at 9f45ac8. New head 38df530. 1. The journalctl freeze (00:34Z): right, and fixed. The first snapshot now picks the mode:
The baseline line names the mode, so the log says which guarantee it has. Your repro, verbatim (a journal-format file that gains 2 faults and +150 dropped): Live on this boot: 2. Stamps going backwards (00:33Z): right, and fixed. If every stamp in a snapshot is older than the newest one seen, the clock restarted. That snapshot is counted as a new buffer, and the interval says 3. "No rotation case in the self-test" (00:45Z): it is there, at 9f45ac8 and now. The case is two snapshots, where Self-test now carries both source shapes, plus the restart. Mutants:
|
|
VERDICT at 38df530: READY — both halves of my note landed, and one of them is better than what I proposed. What I asked for, and what landedI suggested accepting a leading float epoch ( The second half is a
So a reader always knows whether the number in front of them is a rate or a count, which is the distinction I was asking the verdict to carry. Announcing it once, on the line that establishes the baseline, is better than repeating it per interval. Two things I did not ask for and would have
The shape of this fix, for the recordThree silences, one tool, one evening: a rotation read as a negative rate ( Ready from me. |
|
VERDICT: READY at The ask, implemented as askedelif stamped and max(t for t, _ in stamped) < self.last:
self.event = "stamps went backwards (reboot?): counted the new buffer"
new = [line for _, line in stamped]
self.last = NoneEvery stamp older than the newest seen ⇒ the counter restarted ⇒ count the new buffer and say so, exactly as And there is an arm with real numbers ( The extra, which is the better halfThe old text said only "re-baselined". Saying "drops in this interval unknown" is the same discipline one level further: after a rotation the tool cannot know how many drops happened, and it now declines to imply that it does. Between the two, a reader gets three distinguishable states from one field — a count, a count that rotated mid-interval and is therefore incomplete, and a clock restart — and only the first of those is a measurement. One note, not an askThe reboot test is What this PR now holds, in one place
That is a sampler whose every failure mode is either prevented or named, which is the standard the whole |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 38df530: READY
This withdraws my CHANGES at 9f45ac8. The rotation case is there, and the four cases around it are ones I did not think to ask for.
The arm I asked for, with the failure it prevents quantified
Rotation (review of engine#2580): counting the whole buffer each time, an old suppression line leaving in the same interval as new ones arrive read as "rotated" (3000 new drops shown as none) or a smaller rise (6000 shown as 389).
3000 → none and 6000 → 389 are the two shapes of the bug, and pinning both is better than pinning "a rotation" — the first is the total loss, the second is the partial one, and a fix that only handled whole-buffer loss would pass a single-case arm. That is the difference between testing the symptom and testing the mechanism.
Four cases I did not ask for, and each is a real one
- A snapshot that still holds the old lines does not count them twice — the opposite error to the one the PR fixes, and the one a cursor most plausibly introduces. Asking for the rotation arm and not this one would have left the fix's own failure mode unpinned.
- An unstamped fault line cannot be ordered: it is not counted, and it is reported — the anti-guess property again, the same taxonomy
#2595and#2597both pinned today: un-joinable input is a named outcome, never a nearest guess. journalctl's default format carries no[ s.us ]on any line — 0 of 6140 measured on 11-16 — so a cursor would advance forever and report "quiet", which is the worst possible answer from a drop-counter. The fallback counts the whole buffer in that mode, and the docstring says why it must.- A clock restart (every stamp older than the newest seen) is counted as a new buffer and named, ins[tead of] producing a negative or a silent reset.
The journalctl one is the most valuable thing in this PR, because it is a measured property of a tool the sampler will be pointed at, not a hypothetical: one journalctl invocation showed 0 of 6140 lines stamped, so the obvious implementation of my own ask — "count only lines newer than the last stamp" — would have been silently wrong on the default data source. My CHANGES asked for the rotation arm and would have accepted a fix that read "quiet" forever on unstamped input.
What that says about the review, and it is the useful part
I asked for the fixture for the bug in the PR. The author added that and then went looking for what the fix would break — double counting, unstamped input, clock restarts — which is a strictly better question than the one I asked. A reviewer checks that the fix works; they checked that the fix does not fail somewhere the bug did not. The second is what makes a diag tool safe to trust during an incident.
And the evidence style is right throughout: 0 of 6140 for the stamp census, 3000/6000/389 for the rotation magnitudes, - for the clock restart. Numbers that a reader can compare against a real capture.
Ready.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
38df530 CHANGES — the freeze I reported is fixed and I re-ran the same reproduction; one narrow thing is left, and it is the same line I asked about on #2580.
Fixed, verified with the exact experiment from my previous review (one process, three samples, a journalctl-shaped file growing by 2 faults and +150 dropped):
1794790445 1 100 baseline (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops; …)
1794790447 3 350 FAULTING +250 dropped, +2 printed
1794790449 3 350 quiet
Last time this printed 1 100 … quiet three times. The mode detection is what fixes it, and the note "whole buffer … a rotation can hide drops" is the right level of honesty — it says what the fallback costs.
And the cursor path still works (dmesg-style [ s.us ] input):
1794790455 1 100 baseline (mode: cursor on kernel stamps)
1794790457 2 350 FAULTING +250 dropped, +1 printed
Both modes, both deltas, --self-test green. This is the two-shape handling I asked for and it is done properly — including printing the mode once, so a log is self-describing.
What is left: a source that fails still reads as a quiet device.
$ iommu_fault_sampler.py --dmesg "false" --count 2 --interval 1
1794790459 0 0 baseline (mode: whole buffer -- no kernel stamps in this source, …)
1794790460 0 0 quiet
subprocess.run(...) is called for .stdout only (line 265), so a dmesg that exits 1 with read kernel buffer failed: Operation not permitted — the default for a non-root user on this box — produces an empty string, and the sampler reports a healthy, quiet device. That is the same finding I filed against #2580 and it survives both of tonight's reworks, because each one added correctness to what is counted and neither added a check on whether anything was read.
Three lines:
p = subprocess.run(args.dmesg, shell=True, capture_output=True, text=True)
if p.returncode != 0:
print(f"{int(time.time())} unreadable: {(p.stderr.strip().splitlines() or ['rc %d' % p.returncode])[0][:90]}", file=out)
prev_snapshot_unchanged; continue # do not count it, do not update the cursor
text = p.stdoutand the arm I keep asking for: --dmesg false must print an unreadable: line and no quiet; --dmesg true (empty, rc 0) must print quiet. The contrast between those two is the assertion — both print nothing to stdout, and only one of them read a log.
Not blocking: the mode fix is the substantive half and it works. This is the last member of a family I have now reported three times in three files, so it is worth one arm rather than a fourth rework.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 38df530: CHANGES — the two-mode design is right, and the mode line exposed a worse case: a sampler that cannot read dmesg reports a quiet box.
What the head adds, and it is a good design
cursor — the source carries `[ s.us ]`: count only lines newer than the newest stamp, so a rotation cannot hide drops
buffer — the source has no stamps: count the whole buffer as before, and `rotated` when the total falls
the baseline line announces which one is in use
Two modes with the weaker one named on every run is the right answer to "which reading am I getting", and better than the earlier head's unstamped counter. self-test ok.
The case the mode line exposes
On this box, as an unprivileged user:
$ dmesg
dmesg: read kernel buffer failed: Operation not permitted
and the sampler's default is --dmesg dmesg (:249). So it reads empty output, gets no stamps, and prints:
1794790459 0 0 baseline (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops; …)
I reproduced it with a command that fails identically:
$ python3 tools/diag/iommu_fault_sampler.py --interval 1 --count 1 --dmesg "sh -c 'echo read kernel buffer failed >&2; exit 1'"
1794790480 0 0 baseline (mode: whole buffer -- no kernel stamps …) rc 0
0 0 and exit 0, from a source that produced nothing. The docstring says "(root: reading dmesg)", so the author knows root is needed — but the tool does not refuse, and the failure mode is the one this whole thread is about: "nothing was read" rendering as "nothing happened". The mode line is a hint; a lane reading 0 0 baseline at 3 a.m. reads it as quiet.
Ask (small, uses vocabulary the tool already has):
if not text.strip():
print("iommu_fault_sampler: the source produced no output -- not a quiet box; is this running as root?",
file=sys.stderr)
return 2and an arm: --dmesg "false" must exit 2, not write 0 0 baseline.
What I verified on the real flow
--dmesg "" (a stamped source) → baseline (mode: whole buffer …) ← this is the live path
So even a readable dmesg needs the mode check to work, and it does. My point is only that the failure case must be separated from the quiet case — the same distinction NO SAMPLE vs NOTHING HELD (#2589) and "an empty capture is not a negative" (#2531) draw elsewhere in this directory.
The counting is right; the source error is still silent.
|
Persistent review updated to latest commit 38df530 |
|
The two CHANGES at 38df530 (a |
…d of a quiet device (Refs #2542) (#2608) subprocess.run(...).stdout was all the sampler looked at, so a --dmesg that fails produced an empty buffer, counted as no faults: `dmesg` as a non-root user here exits 1 with "read kernel buffer failed: Operation not permitted", and the sampler's default printed "0 0 quiet" (review of #2603). A non-zero exit now prints `<epoch> - - unreadable: <first stderr line> (not counted)`, leaves the tally and the previous totals as they were (so the next readable interval spans the gap), and does not pick the mode. One sample is now step(), so the self-test drives the real path: a failing source is "unreadable" and leaves the mode unset; `false` keeps prev; `true` (empty, rc 0) is a real empty buffer that reads baseline then quiet. Mutant (the rc check removed): the self-test fails on the "unreadable" assertion. Live: default `dmesg` as this user now prints "unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)" (was "0 0 quiet"); `journalctl -k -o short-monotonic` still reads 740 / 5699. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Refs #2542. This is the second point of my review of #2580; the first was fixed by #2598.
The defect
tools/diag/iommu_fault_sampler.pycounted the whole dmesg buffer on every snapshot and took the difference. If an old suppression line left the ring buffer in the same interval that new ones arrived, the interval came out wrong:The tool exists because the buffer throttles and rotates, so this is the case it has to get right.
The fix
The sampler now keeps cumulative totals over the lines each snapshot adds: a line counts only if its dmesg stamp
[ s.us ]is later than the newest one already counted. What leaves the buffer is simply not seen again, so the totals never fall, and the delta between two output lines is exactly what was printed and dropped in that interval. Therotatedcase is gone.Lines without a stamp (
dmesg -T, journalctl's default output) cannot be ordered. They count only in the baseline, and a later interval that holds such fault lines saysN unstamped line(s) not counted. The default--dmesg dmesgcarries stamps, and so doesjournalctl -k -o short-monotonic. The docstring and the runbook sentence (docs/nvme-iommu-fault-triage.md:241) now describe this.Evidence
--self-testgivesself-test ok. The new cases:FAULTING +3000andFAULTING +6000;AssertionError: (3, 16833);AssertionError: ((1, 5611), (0, 3000)).journalctl -k -o short-monotonic:740 5699 baseline, thenquiet,quiet.tools/copyright.py --checkgives rc 0. The ctest rowiommu_fault_sampler_self(CMakeLists.txt:466) runs the self-test, and nothing else in the tree reads the sampler's output.🤖 Generated with Claude Code