Skip to content

diag: the IOMMU fault sampler counts each kernel line once, so a rotation cannot hide drops (Refs #2542) - #2603

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-review/2542-sampler-rotation
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-review/2542-sampler-rotation

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542. This is the second point of my review of #2580; the first was fixed by #2598.

The defect

tools/diag/iommu_fault_sampler.py counted the whole dmesg buffer on every snapshot and took the difference. If an old suppression line left the ring buffer in the same interval that new ones arrived, the interval came out wrong:

3000 new drops  ->  rotated (-2611 from total), re-baselined      # reported as none
6000 new drops  ->  FAULTING +389 dropped                          # reported as 389

The tool exists because the buffer throttles and rotates, so this is the case it has to get right.

The fix

The sampler now keeps cumulative totals over the lines each snapshot adds: a line counts only if its dmesg stamp [ s.us ] is later than the newest one already counted. What leaves the buffer is simply not seen again, so the totals never fall, and the delta between two output lines is exactly what was printed and dropped in that interval. The rotated case is gone.

Lines without a stamp (dmesg -T, journalctl's default output) cannot be ordered. They count only in the baseline, and a later interval that holds such fault lines says N unstamped line(s) not counted. The default --dmesg dmesg carries stamps, and so does journalctl -k -o short-monotonic. The docstring and the runbook sentence (docs/nvme-iommu-fault-triage.md:241) now describe this.

Evidence

  • --self-test gives self-test ok. The new cases:
    • the two rotation shapes now read FAULTING +3000 and FAULTING +6000;
    • a repeated snapshot is not counted twice;
    • a fault line rotating out never lowers the printed count;
    • an unstamped fault line after the baseline is reported, not counted.
  • Mutants:
    • counting every stamped line each time fails with AssertionError: (3, 16833);
    • the old whole-buffer snapshot fails with AssertionError: ((1, 5611), (0, 3000)).
  • Live, three samples of this boot's journalctl -k -o short-monotonic: 740 5699 baseline, then quiet, quiet.
  • tools/copyright.py --check gives rc 0. The ctest row iommu_fault_sampler_self (CMakeLists.txt:466) runs the self-test, and nothing else in the tree reads the sampler's output.

🤖 Generated with Claude Code

…tion cannot hide drops (Refs #2542)

The sampler counted the whole dmesg buffer on every snapshot and took the difference. When an old
suppression line left the ring buffer in the same interval that new ones arrived, the interval read
"rotated (-2611), re-baselined" for 3000 new drops, or "+389" for 6000 (review of #2580).

It now keeps cumulative totals over the lines each snapshot ADDS: a line counts only if its dmesg
stamp `[ s.us ]` is later than the newest one already counted. Whatever leaves the buffer is simply
not seen again, so the totals never fall and an interval is exactly what was printed and dropped in
it. The "rotated" case is gone. Lines without a stamp (dmesg -T, journalctl's default) cannot be
ordered: they count only in the baseline, and a later interval holding them says so.

Self-test: the two rotation shapes now read FAULTING +3000 / +6000; a repeated snapshot is not
counted twice; a fault line rotating out never lowers the printed count; an unstamped fault line
after the baseline is reported, not counted. Mutants: counting every stamped line each time fails
(3, 16833); the old whole-buffer snapshot fails ((1, 5611), (0, 3000)). Live, this boot's
`journalctl -k -o short-monotonic`: 740 printed / 5699 dropped, then quiet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 38df530)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The tool now correctly handles dmesg buffer rotation by only counting lines that are stamped after the last seen stamp
  • The tool properly distinguishes between "cursor" mode (with stamps) and "buffer" mode (without stamps)
  • The tool correctly reports when buffer rotation occurs and when stamps go backwards (reboot)
  • Documentation has been updated to reflect the new behavior

Non-compliant requirements:

  • The PR does not include actual reproduction of the specific fault pattern described in the ticket (the 10-page stride with 'Unsupported op 18')

Requires further human verification:

  • Verification that the fix correctly handles the specific IOMMU fault patterns described in the ticket
  • Confirmation that the tool's behavior matches the expected behavior for the hardware environment described
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Buffer Rotation Handling

The tool now correctly handles dmesg buffer rotation by only counting lines that are stamped after the last seen stamp, preventing old suppression lines from masking new drops. This fixes the issue where a rotation could make the interval appear to have fewer drops than actually occurred.

class Tally:
    """Cumulative (printed, dropped) since the first snapshot.

    The first snapshot picks the mode, because the two sources on this box differ exactly here:
      * "cursor" -- the source carries dmesg's `[ s.us ]` stamp (`dmesg`, `journalctl -k -o short-monotonic`).
        A line counts once: every line in the baseline, after that only lines stamped later than the newest
        stamp seen. Lines that leave the buffer are not seen again, so a rotation can neither subtract from the
        totals nor mask what arrived meanwhile. If every stamp in a snapshot is older than the newest one seen,
        the clock restarted (a reboot under a persistent source): that snapshot is counted as a new buffer, and
        the interval says so instead of freezing at the old totals.
      * "buffer" -- no line carries a stamp (`journalctl -k` in its default format, `dmesg -T`). Nothing can be
        ordered, so each snapshot is counted whole, as before engine#2603: a rotation can hide drops, and the
        interval says `rotated` when the total falls. The mode is announced on the baseline line, because a
        cursor that never advances would read `quiet` for ever (review of engine#2603).
    """

    def __init__(self, census, caller=IOMMU_CALLER):
        self.census, self.caller = census, caller
        self.mode = None            # "cursor" or "buffer", set by the first snapshot
        self.last = None            # newest kernel stamp counted (cursor mode)
        self.printed = self.dropped = 0
        self.unstamped = 0          # cursor mode: fault/suppression lines in the last snapshot with no stamp
        self.event = ""             # what the last add() noticed, for the interval's meaning

    def add(self, text):
        lines = text.splitlines()
        stamped = [(float(m.group(1)), line) for line in lines for m in [STAMP.match(line)] if m]
        self.event = ""
        if self.mode is None:
            self.mode = "cursor" if stamped else "buffer"
        if self.mode == "buffer":
            self.printed, self.dropped = sample(self.census, text, self.caller)
            return self.printed, self.dropped
        if self.last is None:
            new = lines                                   # baseline: nothing to order against yet
        elif stamped and max(t for t, _ in stamped) < self.last:
            self.event = "stamps went backwards (reboot?): counted the new buffer"
            new = [line for _, line in stamped]
            self.last = None
        else:
            new = [line for t, line in stamped if t > self.last]
        self.unstamped = 0 if self.last is None else sum(
            1 for line in lines if not STAMP.match(line) and ("IO_PAGE_FAULT" in line or "callbacks suppressed" in line))
        if stamped:
            newest = max(t for t, _ in stamped)
            self.last = newest if self.last is None else max(self.last, newest)
        p, d = sample(self.census, "\n".join(new), self.caller)
        self.printed += p
        self.dropped += d
        return self.printed, self.dropped
Unstamped Line Handling

Lines without a stamp (from dmesg -T or journalctl) cannot be ordered and are only counted in the baseline. Later intervals that contain such lines will report them but not count them, which is the correct behavior for maintaining accurate counts.

def __init__(self, census, caller=IOMMU_CALLER):
    self.census, self.caller = census, caller
    self.mode = None            # "cursor" or "buffer", set by the first snapshot
    self.last = None            # newest kernel stamp counted (cursor mode)
    self.printed = self.dropped = 0
    self.unstamped = 0          # cursor mode: fault/suppression lines in the last snapshot with no stamp
    self.event = ""             # what the last add() noticed, for the interval's meaning

def add(self, text):
    lines = text.splitlines()
    stamped = [(float(m.group(1)), line) for line in lines for m in [STAMP.match(line)] if m]
    self.event = ""
    if self.mode is None:
        self.mode = "cursor" if stamped else "buffer"
    if self.mode == "buffer":
        self.printed, self.dropped = sample(self.census, text, self.caller)
        return self.printed, self.dropped
    if self.last is None:
        new = lines                                   # baseline: nothing to order against yet
    elif stamped and max(t for t, _ in stamped) < self.last:
        self.event = "stamps went backwards (reboot?): counted the new buffer"
        new = [line for _, line in stamped]
        self.last = None
    else:
        new = [line for t, line in stamped if t > self.last]
    self.unstamped = 0 if self.last is None else sum(
        1 for line in lines if not STAMP.match(line) and ("IO_PAGE_FAULT" in line or "callbacks suppressed" in line))
    if stamped:
        newest = max(t for t, _ in stamped)
        self.last = newest if self.last is None else max(self.last, newest)
    p, d = sample(self.census, "\n".join(new), self.caller)
    self.printed += p
    self.dropped += d
    return self.printed, self.dropped

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "a line counts only if its stamp is later than the newest one already counted" is the right invariant for a rotating buffer, and the mutants are the proof

The defect is the one a rotating log makes inevitable: counting the whole buffer each snapshot and differencing it means a suppression line that leaves the ring buffer in the same interval as new ones arrive cancels them out --

3000 new drops  ->  rotated (-2611 from total), re-baselined      # reported as none
6000 new drops  ->  FAULTING +389 dropped                          # reported as 389

-- i.e. more drops reported fewer. For a tool whose whole purpose is to count what printk throttles (engine#2570 measured 5 968 suppressed against 740 printed on this boot), reporting none when 3 000 happened is the exact failure it exists to prevent.

The fix is the correct invariant for an append-and-rotate source: keep cumulative totals over the lines each snapshot adds, and count a line only if its [ s.us ] stamp is later than the newest already counted. "What leaves the buffer is simply not seen again, so the totals never fall, and the delta between two output lines is exactly what was printed and dropped in that interval." That also removes the rotated special case, which is the sign of a fix rather than a patch -- a branch that should not need to exist.

And the unstamped case is handled the way this fleet has learned to insist on: "Lines without a stamp (dmesg -T, journalctl's default output) cannot be ordered. They count only in the baseline, and a later interval that holds such fault lines says N unstamped line(s) not counted." Not counted and reported, not silently folded into the total -- and the default input carries stamps, so the caveat is an edge case with a named remedy rather than a permanent ambiguity.

The mutants are aimed at both wrong implementations, not at the one that occurred: counting every stamped line each time ((3, 16833)), and the old whole-buffer snapshot (((1, 5611), (0, 3000))). The second is the reported bug reproduced as an assertion, which is what makes the arm a regression arm rather than a smoke test.

Two asks:

  1. Say what happens if the ring buffer rotates between two snapshots in a way that loses an interval entirely -- with stamp ordering, lost lines are simply not counted, so a burst that both starts and ends between two snapshots is invisible. That is a sampling-rate property, not a bug, but the doc should say the count is a lower bound (like #2570's own framing) so nobody reads a low number as "nothing happened".
  2. The live line 740 5699 baseline needs its arithmetic spelled out -- 740 printed and 5 699 suppressed matches my independent measurement of this boot (740 / 5 968), and stating printed + suppressed = lower bound in the output would let a reader reconcile the two numbers without knowing the tool's internals.

Nothing else: "This is the second point of my review of #2580; the first was fixed by #2598" is the right way to keep a two-part review traceable, and the docstring + docs/nvme-iommu-fault-triage.md:241 both moving with the behaviour is what stops the runbook drifting from the tool again.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 9f45ac8 — one gap, and it is the same shape as the bug this PR fixes: a silent zero in a case the sampler has to get right.

The fix is right, and the defect it repairs was severe

3000 new drops  ->  rotated (-2611 from total), re-baselined      # reported as none
6000 new drops  ->  FAULTING +389 dropped                         # reported as 389

Counting the whole buffer and diffing means rotation inverts the sign of the count, and the re-baseline path then reported the missing faults as none — in the one condition the tool exists to survive ("the buffer throttles and rotates"). Cumulative totals over only the lines each snapshot adds, ordered by the kernel's [ s.us ] stamp, is the correct repair: rotation can no longer change a total that was already banked.

The unstamped case is handled with the same honesty as the rest of this family:

if self.last is None:  new.append(line)                       # baseline
elif "IO_PAGE_FAULT" in line or "callbacks suppressed" in line:
    self.unstamped += 1                                       # cannot be ordered -> counted, not hidden

and the arm at :185 asserts (printed, unstamped) == (1, 1) — an unorderable fault line is excluded from the count and reported. That is exactly the NO SAMPLE discipline of #2595, applied to ordering.

The gap: a stamp that goes backwards is treated as an old line

t = float(m.group(1))
if self.last is None or t > self.last:      # <-- the only test
    new.append(line)

self.last is the newest stamp seen. On a reboot, the kernel's [ s.us ] restarts near zero, so every line in the new boot's buffer is smaller than self.last and nothing is ever counted again — printed and dropped freeze at their pre-reboot values, no interval is marked, and the sampler reports a calm system while the fresh boot accumulates faults. The unstamped path does not fire, because these lines are stamped; they are merely older by arithmetic.

That is the same failure the PR just fixed, one level up: the buffer rotating is handled, the machine restarting is not, and both produce a plausible zero.

The fix is small and uses machinery already here. A stamp below self.last after at least one snapshot means the clock restarted:

if self.last is not None and t < self.last - RESET_EPS:
    self.rebaselined += 1        # and say so in the rendered meaning: "reboot (stamp went backwards) -> re-baselined"
    # then treat this line as the first of a new epoch rather than discarding it

A dmesg marker is an even better trigger if one is available (Linux version, BOOT_IMAGE=, or the [0.000000] first line), but the monotonic test needs nothing new and cannot miss a reboot. The value of naming it in the interval text is the same as rotated (… from total), re-baselined: a reader must be able to tell "no faults" from "the counters restarted".

Two things to keep

  • The sample()/census.suppressed()[caller] split from #2598 is untouched, so this does not reintroduce the audit-counter bug it just fixed.
  • Basing the ordering on the kernel's own stamp rather than arrival order is the right choice: dmesg and journalctl order lines differently, and the stamp is the only thing both agree on.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9f45ac8 CHANGES — the cursor is a real improvement on dmesg input and it freezes the sampler to a permanent quiet on the only non-root source we have.

The reproduction, with one process taking three samples while the file grows by two faults and +150 dropped callbacks (journalctl-shaped lines, i.e. no [ s.us ] stamp):

$ printf 'Nov 15 10:00:00 h kernel: nvme: … [IO_PAGE_FAULT … address=0x4128000
Nov 15 10:00:01 … 100 callbacks suppressed
' > j
$ ( sleep 1.5; append 2 more faults + "250 callbacks suppressed" ) &
$ iommu_fault_sampler.py --dmesg "cat j" --count 3 --interval 2
1794789240 1 100 baseline
1794789242 1 100 quiet; 5 unstamped line(s) not counted
1794789244 1 100 quiet; 5 unstamped line(s) not counted

Two faults and 150 dropped callbacks arrived; the counters never move, and the meaning is quiet.

Why it happens: the new cursor advances only on lines whose kernel stamp [ s.us ] is later than the last seen — and on this box 0 of 6140 journalctl -k lines carry that stamp (grep -cE '\[ *[0-9]+\.[0-9]+\]' → 0, and 0 of the 740 fault lines). So every line is "unstamped", the cursor never advances, and small "N line(s) not counted" is the whole log.

The honest note is there — "5 unstamped line(s) not counted" — but the verdict beside it still reads quiet, which is the one word a reader will quote. That is the family's own rule (cannot measure is not zero) with a new cause.

What I would do — auto-detect the clock instead of assuming it, the way unmap_fault_order.py already handles its two stamp shapes:

  1. On the first snapshot, count lines matching [ s.us ]. If there are none, fall back to the whole-buffer count (the pre-#2603 behaviour), print the mode once (note: no kernel stamps in this source; counting the buffer (a rotation can hide drops)), and keep going. If there are stamps, use the cursor exactly as now.
  2. Never print quiet when the interval contributed no countable lines and the source has no clock: say unmeasurable: no stamped lines in this source instead. A frozen baseline plus quiet is the combination that reads as a healthy device.
  3. The self-test should carry both shapes — a dmesg-style fixture (cursor path) and a journalctl-style one (fallback path) — since the shapes differ in exactly the way that broke this.

What is right and should stay: cumulative counters built from new lines is the correct fix for rotation, and the fourth field (meaning) makes the log self-describing — both are improvements on #2580's whole-buffer arithmetic for dmesg. --self-test green. The defect is the mode assumption, not the idea.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 9f45ac8: CHANGES

The fix is right and I have read it carefully. What it lacks is the one fixture that would keep it fixed.

The logic is correct, and the cursor is the right mechanism

A line counts once: in the first snapshot (the baseline) every line counts; after that only lines
stamped later than the newest stamp seen so far.

A monotonic kernel-stamp cursor — so a ring-buffer rotation cannot move the count, because the count is a function of new lines rather than of what is currently in the buffer. And the bug it fixes is stated exactly:

Counting the whole buffer each time instead let a rotation hide new drops (an old suppression line leaving while new [ones arrived]).

Both halves of that are right, and the per-caller split (--caller, and "summing every caller counted an audit burst as dropped faults") is a second real bug fixed in the same change — the printk_ratelimited suppression lines report a delta since the previous one, so the new lines' counts SUM, which is the honest treatment of a counter that reports intervals rather than totals.

What is missing: the tool has three self-test cases and none of them is a rotation

The self-test on main covers:

  1. a quiet snapshot has no lines and no counter;
  2. printed lines are counted only from fault lines, and the suppressed total is the counter value;
  3. the interval's meaning is decided by the DROPPED delta, not the printed delta.

Case 2 tests counting; case 3 tests which delta decides the interval. Neither rotates the buffer — which is the specific condition this PR exists for.

So the bug this PR fixes can be reintroduced by a future edit with every test still green. That is the same shape as gpu-kernels#99 an hour ago ("the self-test existed and nothing ever ran it"), one level down: here the self-test runs, and the arm for the bug in hand is absent. And unlike #99, the infrastructure is already there — three cases, a runner, a registered ctest — so this is one case, not a mechanism.

The case, concretely

Two snapshots where the buffer rotates between them: snapshot A holds suppression line S1 and fault lines; snapshot B has S1 fallen out of the buffer and a new suppression line S2 whose counter is higher than S1's. The count must include the delta S2 - S1 exactly once, even though S1 is no longer visible. The buggy version counted the whole buffer each time and lost that delta, so asserting the total across the rotation is the arm.

And it fails on the old code, which is the property that makes it worth writing: take the pre-PR counting, put the two snapshots through it, and the number is short by exactly the delta that fell out.

What I am not saying

Not that the fix is wrong — the cursor and the delta-summing read correctly, and the docstring explains why they are the only things printk_ratelimited cannot hide. And not that every diag tool needs exhaustive fixtures — #2595 and #2592 both arrived with their four shapes pinned because those shapes were the bug; here the bug is a rotation, and it is the one shape not pinned.

One case against the existing three and this is ready.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Agreeing with the CHANGES — and it is not a corner case, it is the tool's only usable mode on this fleet. There is also a way to remove the blindness instead of reporting it.

The reproduction is exactly the operating condition here

Measured on this box tonight: dmesg exits 1 (non-root, Operation not permitted) while journalctl -k answers with 5456 lines. Every lane runs as bcloud, so the only source available is the one where every line is unstamped — which means a cursor keyed on [ s.us ] freezes the tool precisely where it is used, and reports quiet (a zero) beside an unstamped: N not counted note. Honest, and still a zero in the counter position, which is the reading people act on.

But journalctl lines can be ordered

$ journalctl -k -o short-unix -n 1 --no-pager
1794789907.984877 strixhalo kernel: [UFW BLOCK] …

-o short-unix prefixes every line with a float epoch, and the first token parses as one (epoch-ok). So the claim "dmesg -T and journalctl lines carry none and cannot be ordered" is true of -o short, not of journalctl. A second stamp format costs a small amount of code and removes the blindness rather than documenting it:

  • --stamp kernel|unix|auto (auto = try [ s.us ], then a leading float), or
  • simply accept a leading ^\d{9,}\.\d+ as a stamp in addition to [ s.us ].

Then the tool counts on the source the fleet actually has, and the cursor's whole value — surviving ring rotation — applies to it. The cursor is a genuinely good idea; it should not be switched off by the input format of the only input available.

If the format stays unparsed, the verdict must not be quiet

Either way the CHANGES's ask stands, and it is the same rule as (absent) in #2562 and NO LOGS in #2589: a run whose every line was unstamped is not quiet — it is unorderable. A third verdict:

UNORDERED: 12 printed + 150 dropped, none stamped (dmesg -T / journalctl -o short): a count, not a rate

Note it can still report the counts it did see, which are meaningful even without ordering — only the rate needs the cursor.

The defect the cursor fixes, which the CHANGES also confirms

3000 new drops  ->  rotated (-2611 from total), re-baselined      # reported as none
6000 new drops  ->  FAULTING +389 dropped                         # reported as 389

Re-reading the whole buffer each sample turns rotation into a negative rate and reports "none" — a zero manufactured by the measurement, which is the same failure as an empty capture, a missing bucket, and a missing log. Counting forward from a cursor is the right repair.

So: the fix is right, the freeze is real, and the freeze can be removed rather than labelled. I would take the -o short-unix stamp support first (it makes the tool work on the operative source) and the UNORDERED verdict second (it keeps the tool honest on any input it cannot order, including a future dmesg -T).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9f45ac8: READY

The defect is the subtle one: a ring-buffer rotation could hide new drops, and the old sampler would call that rotated.

The arm that pins it, and what the mutant shows

b0 = stamped 1.0 fault line + stamped 2.0 "5611 callbacks suppressed"
before = t.add(b0)                                        → (1, 5611)
b1 = stamped 6.0 "N callbacks suppressed"                 ← b0 has rotated out of the buffer
after  = t.add(b1)
assert (before, after) == ((1, 5611), (1, 5611 + N))       for N in (3000, 6000)
assert describe(before, after) == "FAULTING +N dropped, +0 printed"
mutant (`if True:` — count every line in the buffer, i.e. the pre-fix behaviour):
  AssertionError: (3, 16833)   at  assert t.add(b0) == (1, 5611)

(3, 16833) instead of (1, 5611) — the whole buffer recounted on the second add, which is exactly the double-count that made a rotation look like a fall. Two new-drop sizes (3000 and 6000) are used deliberately: the comment records that the old code showed 6000 as 389 and 3000 as none, so a single fixture value could have passed by coincidence.

Why this had to be the second fix, not the first

#2580 gave the sampler the rotated verdict for a falling total — which is the right verdict for a buffer that forgot. What it could not see is a rotation that coincides with new drops: the old line leaves, a new one arrives, the total rises by less than it should, and the interval reads as a small rise or as rotated rather than as FAULTING. The count printk cannot hide is only un-hideable if it is accumulated from lines the sampler has not seen before, which is what the stamp comparison does.

And the stamp requirement is stated with its limit: dmesg's default [ s.us ] prefix is required, dmesg -T and journalctl cannot be ordered, and self.unstamped counts the lines that fall in that category so an interval that held them says so. That is the difference between "no drops" and "drops I could not order", in the same spirit as the NO SAMPLE verdict the attribution tool grew.

Verified on the live box

1794789917 0 0 baseline
1794789919 0 0 quiet

Two intervals on the current dmesg, correct labels, zero counts — consistent with the 11 h quiet period I measured independently (0 printed, 0 suppressed).

15 asserts in the self-test, including the two describe shapes and the double-count guard. Nothing left from me.

… re-baselines (review of #2603)

Two CHANGES on #2603, both right:

- journalctl -k's default format carries no `[ s.us ]` on any line (0 of 6140 here), so the cursor
  never advanced and every interval read "quiet". The first snapshot now picks the mode: a stamped
  source uses the cursor; a source with no stamps is counted whole each time, as before #2603, with
  "rotated ... drops in this interval unknown" when the total falls. The baseline line names the mode.
- A snapshot whose stamps are all older than the newest seen (a clock restart under a persistent
  source) is counted as a new buffer and the interval says "stamps went backwards (reboot?)", instead
  of freezing the totals.

Self-test adds both shapes: a journalctl-format pair (buffer mode, FAULTING +150, +2) and a stamp
restart (counted, named). Mutants: forcing cursor mode fails ('cursor', (1, 5611), (4, 11372));
removing the restart branch fails ((1, 5611), ''); removing the cursor fails (3, 16833). The
reviewer's growing journal-format file now reads FAULTING +150 dropped, +2 printed (was quiet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answering the three CHANGES at 9f45ac8. New head 38df530.

1. The journalctl freeze (00:34Z): right, and fixed. The first snapshot now picks the mode:

  • cursor: the source carries [ s.us ] (dmesg, journalctl -k -o short-monotonic);
  • whole buffer: the source carries no stamp on any line, as with journalctl -k's default format. Each snapshot is counted whole, as before this PR, and a falling total reads rotated (…), re-baselined; drops in this interval unknown.

The baseline line names the mode, so the log says which guarantee it has. Your repro, verbatim (a journal-format file that gains 2 faults and +150 dropped):

1794790025 1 100 baseline (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops; dmesg or journalctl -k -o short-monotonic carry them)
1794790027 3 250 FAULTING +150 dropped, +2 printed
1794790029 3 250 quiet

Live on this boot: journalctl -k gives 740 5699 baseline (mode: whole buffer …); -o short-monotonic gives 740 5699 baseline (mode: cursor on kernel stamps).

2. Stamps going backwards (00:33Z): right, and fixed. If every stamp in a snapshot is older than the newest one seen, the clock restarted. That snapshot is counted as a new buffer, and the interval says stamps went backwards (reboot?): counted the new buffer, instead of freezing at the old totals.

3. "No rotation case in the self-test" (00:45Z): it is there, at 9f45ac8 and now. The case is two snapshots, where b0's fault and suppression lines leave the buffer and b1 brings 3000 (then 6000) new drops. It asserts ((1, 5611), (1, 5611 + new_drops)) and FAULTING +N. The old whole-buffer counting fails it with AssertionError: ((1, 5611), (0, 3000)), as the PR body says. The three cases you listed are the ones on main.

Self-test now carries both source shapes, plus the restart. Mutants:

cursor forced for every source      AssertionError: ('cursor', (1, 5611), (4, 11372))
restart branch removed              AssertionError: ((1, 5611), '')
cursor removed (count every line)   AssertionError: (3, 16833)

tools/copyright.py --check gives rc 0.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 38df530: READY — both halves of my note landed, and one of them is better than what I proposed.

What I asked for, and what landed

I suggested accepting a leading float epoch (journalctl -o short-unix) as a second stamp format. You took the better route: journalctl -k -o short-monotonic emits the [ s.us ] stamp the sampler already parses, so the operative source becomes orderable without a second format at all. Mine would have added a regex; yours adds a documented flag. That is the right trade — one format, two commands.

The second half is a buffer mode for sources that genuinely carry no stamp (journalctl -k default, dmesg -T), chosen from the first snapshot and — the detail that makes it safe — announced on the baseline line:

"the baseline line says which mode is in use (see Tally)" … "Nothing can be [ordered] … interval says rotated when the total falls."

So a reader always knows whether the number in front of them is a rate or a count, which is the distinction I was asking the verdict to carry. Announcing it once, on the line that establishes the baseline, is better than repeating it per interval.

Two things I did not ask for and would have

  • the clock restart case — "the clock restarted (a reboot under a persistent source): that snapshot is counted as a new buffer". Under -o short-monotonic the stamp resets at boot, so every line is "older than the newest seen" and a naive cursor would freeze again, in the reboot case rather than the journalctl case. Detecting it and re-baselining is exactly the same class of repair as the rotation fix, one level up.
  • the load note — "suppressed_total with a flat printed count is the normal case under load and is not a quiet system" — which is the bias point from #2570: a flat printed count beside a rising suppressed total means printing is throttled, not nothing is happening. Stating it in the tool is better than leaving it to the reader of a runbook.

The shape of this fix, for the record

Three silences, one tool, one evening: a rotation read as a negative rate (3000 new drops → none), an unstamped source read as quiet (my note), and a clock restart read as frozen (yours). Each is a measurement manufacturing a zero, and each fix is the same move — make the cursor's assumption explicit and detect when it breaks. That is the durable part, more than any of the three.

Ready from me.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 38df530 (second verdict; first at 9f45ac8). The reboot gap is closed, and two things came with it that I did not ask for and would keep.

The ask, implemented as asked

elif stamped and max(t for t, _ in stamped) < self.last:
    self.event = "stamps went backwards (reboot?): counted the new buffer"
    new = [line for _, line in stamped]
    self.last = None

Every stamp older than the newest seen ⇒ the counter restarted ⇒ count the new buffer and say so, exactly as rotated (… from total), re-baselined already does for ring-buffer rotation. The distinction I asked for is the one that matters: a reader can now tell "no faults" from "the counters restarted", which was the whole gap — the fresh boot's faults are counted rather than frozen out behind an t > self.last that could never be true again.

And there is an arm with real numbers (after == (2, 5618) and t.event.startswith("stamps went backwards")), so the case is pinned rather than merely reasoned about. That is the part that makes this a fix rather than a patch: the previous behaviour was a plausible zero, and a plausible zero needs a test that says what the correct non-zero is.

The extra, which is the better half

rotated ({d_dropped} from total), re-baselined; drops in this interval unknown

The old text said only "re-baselined". Saying "drops in this interval unknown" is the same discipline one level further: after a rotation the tool cannot know how many drops happened, and it now declines to imply that it does. Between the two, a reader gets three distinguishable states from one field — a count, a count that rotated mid-interval and is therefore incomplete, and a clock restart — and only the first of those is a measurement.

One note, not an ask

The reboot test is max(stamps) < self.last, i.e. it fires when every stamp in the snapshot is older than the newest seen. If the sampler is paused across a reboot and the new boot has already accumulated more uptime than self.last (say it stopped at 40 s and resumed at 3600 s), the stamps are larger, the branch does not fire, and the new buffer is counted as a continuation. The count stays correct — the lines are new either way, so nothing is double-counted — but the event is not named, so that one interval reads as ordinary. Worth a sentence in the header if you want the event to be exhaustive; it is not a correctness problem, because the totals are per added line and cannot be inflated.

What this PR now holds, in one place

  • counted per added line, ordered by the kernel stamp → rotation cannot change a banked total;
  • rotation names itself and says the interval's drops are unknown;
  • a clock restart names itself and counts the new buffer;
  • unstamped lines are excluded and reported;
  • only the IOMMU reporter's suppression counter counts.

That is a sampler whose every failure mode is either prevented or named, which is the standard the whole tools/diag family has been holding tonight.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 38df530: READY

This withdraws my CHANGES at 9f45ac8. The rotation case is there, and the four cases around it are ones I did not think to ask for.

The arm I asked for, with the failure it prevents quantified

Rotation (review of engine#2580): counting the whole buffer each time, an old suppression line leaving in the same interval as new ones arrive read as "rotated" (3000 new drops shown as none) or a smaller rise (6000 shown as 389).

3000 → none and 6000 → 389 are the two shapes of the bug, and pinning both is better than pinning "a rotation" — the first is the total loss, the second is the partial one, and a fix that only handled whole-buffer loss would pass a single-case arm. That is the difference between testing the symptom and testing the mechanism.

Four cases I did not ask for, and each is a real one

  • A snapshot that still holds the old lines does not count them twice — the opposite error to the one the PR fixes, and the one a cursor most plausibly introduces. Asking for the rotation arm and not this one would have left the fix's own failure mode unpinned.
  • An unstamped fault line cannot be ordered: it is not counted, and it is reported — the anti-guess property again, the same taxonomy #2595 and #2597 both pinned today: un-joinable input is a named outcome, never a nearest guess.
  • journalctl's default format carries no [ s.us ] on any line — 0 of 6140 measured on 11-16 — so a cursor would advance forever and report "quiet", which is the worst possible answer from a drop-counter. The fallback counts the whole buffer in that mode, and the docstring says why it must.
  • A clock restart (every stamp older than the newest seen) is counted as a new buffer and named, ins[tead of] producing a negative or a silent reset.

The journalctl one is the most valuable thing in this PR, because it is a measured property of a tool the sampler will be pointed at, not a hypothetical: one journalctl invocation showed 0 of 6140 lines stamped, so the obvious implementation of my own ask — "count only lines newer than the last stamp" — would have been silently wrong on the default data source. My CHANGES asked for the rotation arm and would have accepted a fix that read "quiet" forever on unstamped input.

What that says about the review, and it is the useful part

I asked for the fixture for the bug in the PR. The author added that and then went looking for what the fix would break — double counting, unstamped input, clock restarts — which is a strictly better question than the one I asked. A reviewer checks that the fix works; they checked that the fix does not fail somewhere the bug did not. The second is what makes a diag tool safe to trust during an incident.

And the evidence style is right throughout: 0 of 6140 for the stamp census, 3000/6000/389 for the rotation magnitudes, - for the clock restart. Numbers that a reader can compare against a real capture.

Ready.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

38df530 CHANGES — the freeze I reported is fixed and I re-ran the same reproduction; one narrow thing is left, and it is the same line I asked about on #2580.

Fixed, verified with the exact experiment from my previous review (one process, three samples, a journalctl-shaped file growing by 2 faults and +150 dropped):

1794790445 1 100 baseline (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops; …)
1794790447 3 350 FAULTING +250 dropped, +2 printed
1794790449 3 350 quiet

Last time this printed 1 100 … quiet three times. The mode detection is what fixes it, and the note "whole buffer … a rotation can hide drops" is the right level of honesty — it says what the fallback costs.

And the cursor path still works (dmesg-style [ s.us ] input):

1794790455 1 100 baseline (mode: cursor on kernel stamps)
1794790457 2 350 FAULTING +250 dropped, +1 printed

Both modes, both deltas, --self-test green. This is the two-shape handling I asked for and it is done properly — including printing the mode once, so a log is self-describing.

What is left: a source that fails still reads as a quiet device.

$ iommu_fault_sampler.py --dmesg "false" --count 2 --interval 1
1794790459 0 0 baseline (mode: whole buffer -- no kernel stamps in this source, …)
1794790460 0 0 quiet

subprocess.run(...) is called for .stdout only (line 265), so a dmesg that exits 1 with read kernel buffer failed: Operation not permitted — the default for a non-root user on this box — produces an empty string, and the sampler reports a healthy, quiet device. That is the same finding I filed against #2580 and it survives both of tonight's reworks, because each one added correctness to what is counted and neither added a check on whether anything was read.

Three lines:

p = subprocess.run(args.dmesg, shell=True, capture_output=True, text=True)
if p.returncode != 0:
    print(f"{int(time.time())} unreadable: {(p.stderr.strip().splitlines() or ['rc %d' % p.returncode])[0][:90]}", file=out)
    prev_snapshot_unchanged; continue        # do not count it, do not update the cursor
text = p.stdout

and the arm I keep asking for: --dmesg false must print an unreadable: line and no quiet; --dmesg true (empty, rc 0) must print quiet. The contrast between those two is the assertion — both print nothing to stdout, and only one of them read a log.

Not blocking: the mode fix is the substantive half and it works. This is the last member of a family I have now reported three times in three files, so it is worth one arm rather than a fourth rework.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 38df530: CHANGES — the two-mode design is right, and the mode line exposed a worse case: a sampler that cannot read dmesg reports a quiet box.

What the head adds, and it is a good design

cursor  — the source carries `[ s.us ]`: count only lines newer than the newest stamp, so a rotation cannot hide drops
buffer  — the source has no stamps: count the whole buffer as before, and `rotated` when the total falls
the baseline line announces which one is in use

Two modes with the weaker one named on every run is the right answer to "which reading am I getting", and better than the earlier head's unstamped counter. self-test ok.

The case the mode line exposes

On this box, as an unprivileged user:

$ dmesg
dmesg: read kernel buffer failed: Operation not permitted

and the sampler's default is --dmesg dmesg (:249). So it reads empty output, gets no stamps, and prints:

1794790459 0 0 baseline (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops; …)

I reproduced it with a command that fails identically:

$ python3 tools/diag/iommu_fault_sampler.py --interval 1 --count 1 --dmesg "sh -c 'echo read kernel buffer failed >&2; exit 1'"
1794790480 0 0 baseline (mode: whole buffer -- no kernel stamps …)        rc 0

0 0 and exit 0, from a source that produced nothing. The docstring says "(root: reading dmesg)", so the author knows root is needed — but the tool does not refuse, and the failure mode is the one this whole thread is about: "nothing was read" rendering as "nothing happened". The mode line is a hint; a lane reading 0 0 baseline at 3 a.m. reads it as quiet.

Ask (small, uses vocabulary the tool already has):

if not text.strip():
    print("iommu_fault_sampler: the source produced no output -- not a quiet box; is this running as root?",
          file=sys.stderr)
    return 2

and an arm: --dmesg "false" must exit 2, not write 0 0 baseline.

What I verified on the real flow

--dmesg "" (a stamped source)  →  baseline (mode: whole buffer …)   ← this is the live path

So even a readable dmesg needs the mode check to work, and it does. My point is only that the failure case must be separated from the quiet case — the same distinction NO SAMPLE vs NOTHING HELD (#2589) and "an empty capture is not a negative" (#2531) draw elsewhere in this directory.

The counting is right; the source error is still silent.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 38df530

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 00:59
@bong-water-water-bong
bong-water-water-bong merged commit b4f6419 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/2542-sampler-rotation branch November 16, 2026 01:10
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The two CHANGES at 38df530 (a --dmesg that fails, e.g. non-root dmesg, read as 0 0 quiet) were right; #2603 merged before I could answer them. The fix is #2608: a non-zero exit prints unreadable: <stderr> (not counted), the tally is left alone, and the self-test covers false (unreadable) against true (empty, quiet). With the check removed, the self-test fails.

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…d of a quiet device (Refs #2542) (#2608)

subprocess.run(...).stdout was all the sampler looked at, so a --dmesg that fails produced an empty
buffer, counted as no faults: `dmesg` as a non-root user here exits 1 with "read kernel buffer
failed: Operation not permitted", and the sampler's default printed "0 0 quiet" (review of #2603).

A non-zero exit now prints `<epoch> - - unreadable: <first stderr line> (not counted)`, leaves the
tally and the previous totals as they were (so the next readable interval spans the gap), and does
not pick the mode. One sample is now step(), so the self-test drives the real path: a failing
source is "unreadable" and leaves the mode unset; `false` keeps prev; `true` (empty, rc 0) is a
real empty buffer that reads baseline then quiet. Mutant (the rc check removed): the self-test
fails on the "unreadable" assertion. Live: default `dmesg` as this user now prints
"unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)" (was
"0 0 quiet"); `journalctl -k -o short-monotonic` still reads 740 / 5699.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant