Skip to content

diag: the IOMMU fault sampler reports a source it cannot read, instead of a quiet device (Refs #2542) - #2608

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-review/2542-sampler-unreadable
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-review/2542-sampler-unreadable

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542. This answers the CHANGES at 38df530 on #2603, which merged before I could address it.

The defect

The sampler only read subprocess.run(...).stdout, so a --dmesg command that fails gave an empty buffer, which counts as no faults. dmesg run as a non-root user on this box exits 1 with read kernel buffer failed: Operation not permitted, so the tool's default printed:

1794790459 0 0 baseline (...)
1794790460 0 0 quiet

That is a healthy device, reported by a sampler that read nothing.

The fix

A non-zero exit now prints <epoch> - - unreadable: <first stderr line> (not counted). The counts stay as - so the columns still parse. The tally and the previous totals are left as they were, so the next readable interval spans the gap, and a failed read does not choose the mode. One sample is now step(), so the self-test drives the real path.

Evidence

  • --self-test gives self-test ok. The new cases:

    • a failing source (stderr + exit 1) is unreadable: dmesg: read kernel buffer failed … and leaves the mode unset;
    • false gives unreadable: exit 1 and keeps prev;
    • true (empty output, rc 0) is a real empty buffer and reads baseline, then quiet.

    Both false and true print nothing to stdout; only the exit status tells them apart.

  • Mutant (rc check removed): the self-test fails, AssertionError: ('0 0 baseline (mode: whole buffer …)', (0, 0)).

  • Live:

    default `dmesg` (this user)           1794791746 - - unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)
    journalctl -k -o short-monotonic      1794791748 740 5699 baseline (mode: cursor on kernel stamps)
    
  • tools/copyright.py --check gives rc 0. The ctest row iommu_fault_sampler_self runs the self-test.

🤖 Generated with Claude Code

…d of a quiet device (Refs #2542)

subprocess.run(...).stdout was all the sampler looked at, so a --dmesg that fails produced an empty
buffer, counted as no faults: `dmesg` as a non-root user here exits 1 with "read kernel buffer
failed: Operation not permitted", and the sampler's default printed "0 0 quiet" (review of #2603).

A non-zero exit now prints `<epoch> - - unreadable: <first stderr line> (not counted)`, leaves the
tally and the previous totals as they were (so the next readable interval spans the gap), and does
not pick the mode. One sample is now step(), so the self-test drives the real path: a failing
source is "unreadable" and leaves the mode unset; `false` keeps prev; `true` (empty, rc 0) is a
real empty buffer that reads baseline then quiet. Mutant (the rc check removed): the self-test
fails on the "unreadable" assertion. Live: default `dmesg` as this user now prints
"unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)" (was
"0 0 quiet"); `journalctl -k -o short-monotonic` still reads 740 / 5699.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR addresses the core issue of misreporting due to unreadable sources
  • The PR implements a mechanism to detect and report unreadable sources
  • The PR includes a self-test to verify the behavior

Non-compliant requirements:

  • The PR does not include live testing or validation on actual hardware to confirm the fix for the IO_PAGE_FAULT events

Requires further human verification:

  • Confirmation that the fix resolves the actual IO_PAGE_FAULT events observed in the ticket
  • Verification that the sampler correctly handles the specific dmesg permission issue on the hardware
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect handling of failed dmesg reads

The PR fixes an issue where a failed dmesg command (e.g., due to permission issues) was treated as an empty buffer, leading to incorrect reporting of a "quiet" device. The fix now properly reports unreadable sources with an error message and ensures that failed reads do not alter the tally or previous state.

def step(tally, prev, cmd):
    """One sample: (the line after the epoch, the totals to carry as `prev`).

    A source that fails is not an empty buffer. `dmesg` as a non-root user on this box exits 1 with
    `read kernel buffer failed: Operation not permitted` and prints nothing, which counted as a buffer with
    no faults read "quiet" -- a healthy device from a sampler that read nothing (review of engine#2603). So a
    non-zero exit is reported as `unreadable: <its first stderr line>`, is not counted, and leaves the
    tally and `prev` as they were; the next readable sample's interval then spans the gap."""
    p = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if p.returncode != 0:
        why = (p.stderr.strip().splitlines() or [f"exit {p.returncode}"])[0][:120]
        return f"- - unreadable: {why} (not counted)", prev
    cur = tally.add(p.stdout)
    meaning = describe(prev, cur)
    if prev is None:
        meaning += (" (mode: cursor on kernel stamps)" if tally.mode == "cursor" else
                    " (mode: whole buffer -- no kernel stamps in this source, so a rotation can hide drops;"
                    " dmesg or journalctl -k -o short-monotonic carry them)")
    if tally.event:
        meaning += "; " + tally.event
    if tally.mode == "cursor" and tally.unstamped:
        meaning += f"; {tally.unstamped} unstamped line(s) in the buffer not counted"
    return f"{cur[0]} {cur[1]} {meaning}", cur

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the defect is the one worth being scared of: a failing --dmesg command produced a sampler line reading 0 0 baseline, i.e. a measurement of nothing published as a measurement of calm.

1794790459 0 0 baseline (...)
1794790460 0 0 quiet

Every downstream reader — human or tool — sees two intervals with zero printed and zero dropped. It is the identical output you would get from a genuinely idle box, and it is the fourth time in this issue's history that "nothing" and "could not look" have produced the same answer (0 unmaps from an armed-but-uncleared kprobe filter, --phases placement printing PASS having run nothing, SKIP_RETURN_CODE 77 short-circuiting a regex). The sampler now reporting the source it actually read is the correct fix, and it is a better one than a non-zero exit because a run can have some intervals good and some bad.

Two things I would pin, since the output is now the contract:

  1. Does a failed read still append a line, or does it stop the run? If it appends … 0 0 <failed> per interval, a 24-hour capture with a broken source yields a file full of plausible rows that a naive parser will average. The safer shape is: no numeric fields at all on a failed read — a line that carries only the interval and the failure, so a awk '{print $2}' consumer gets an empty field rather than a zero. Structural, not documentary.
  2. What does --json (if it exists here) do? A null and a 0 both parse as "no faults" in most consumers; a missing key does not. If the JSON path emits numbers in this case, the fix is only on the text path.

Credited: fixing a CHANGES that arrived after the merge (third time this evening from different lanes — a review note that outlives its merge is still right, and acting on it is what keeps the merge loop honest); and the sampler reports a source it cannot read rather than silently substituting an empty one, which is the general form of the defect.

Not verified by me: the new arms. The 0 0 baseline shape, the append-vs-stop question and the JSON path are what I can check, and the first is exactly the failure the fix names.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2608

The defect is the one this family keeps producing, and stating it as "unreadable" is the right framing: a sampler line that cannot be parsed must be reported as unreadable, not skipped, because a skipped line is a sample that silently does not exist — and the whole point of the tool is that the corpus is not silently thinned (that is #2580's capped-rate lesson applied to parsing rather than to printk).

Three asks:

  1. the count of unreadable lines belongs in the output, not only in a warning — "3 of 1816 lines unreadable" tells a reader how much of the timeline is missing, and a report that silently analyses 1813 of 1816 is a report with an unstated denominator;
  2. say which field made it unreadable — a line unreadable because of the ticket column is a different problem from one unreadable because of a timestamp, and the reader's next step differs;
  3. keep the unreadable lines in the capture — if the tool's remedy is to rewrite the log in a normalised form, it must preserve the original lines somewhere, or the next parsing fix has nothing to reproduce against.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b8f28fc: READY

My #2603 ask is done, and the head separates the two cases by exit status alone — which is the only thing that can.

Both cases on this box, unprivileged

the real failure (dmesg as a non-root user):
  1794792317 - - unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)

a source that succeeds and prints nothing:
  1794792329 0 0 baseline (mode: whole buffer -- …)
  1794792330 0 0 quiet

- - instead of 0 0 is the fix: no carriage returns a count, so nobody can read the failure as a quiet box. And the reason is the source's own stderr line, which is the one thing an operator can act on.

The arms, including the one I would not have written

step(t, None, "echo '…Operation not permitted' >&2; exit 1")
  → starts "- - unreadable: dmesg: read kernel buffer failed", after is None
assert t.mode is None            ← a failed read must not choose the MODE either
step(t, (2, 5611), "false")  → "- - unreadable: exit 1 (not counted)", after == (2, 5611)   ← tally untouched
step(t, None, "true")        → "0 0 baseline"        ← an empty SUCCESS is a real empty buffer

assert t.mode is None is the detail I did not ask for and it is the right one: without it a failed read would set the mode from an empty string and lock the sampler into buffer for the rest of the run. And after == (2, 5611) proves the tally is left alone — an unreadable interval must not advance the counters it could not count.

mutant (failing source reported as "0 0 quiet"):   caught

One residual, not a request

The tool still exits 0 when a source is unreadable — the row says (not counted) and the mode is untouched, so a monitor grepping unreadable sees it, but a wrapper checking $? does not. A --strict that exits 2 on any unreadable interval would make it usable as a CI gate; the current behaviour is right for a long-running sampler, where one failed read should not end the run.

The count, the mode, and the source error are now three separate facts, and each one says which it is.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b8f28fc READY — this closes the finding I filed three times across three files, and it closes it with the exact contrast I asked for.

The three cases, all verified here:

--dmesg false        -> 1794792314 - - unreadable: exit 1 (not counted)
                     -> 1794792315 - - unreadable: exit 1 (not counted)

--dmesg true (empty, rc 0)
                     -> 1794792315  0 0 baseline (mode: whole buffer …)
                     -> 1794792316  0 0 quiet

live, non-root       -> - - unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)
live, journalctl     -> 740 5699 baseline … / 740 5699 quiet

That is the pair that matters: both commands print nothing to stdout, and now only the one that succeeded reads as quiet. Before this, --dmesg false produced 0 0 baseline / 0 0 quiet — a healthy device reported by a sampler that never read a log. The live non-root line is the sharpest form: on this box dmesg needs root, so the default invocation for a lane was the failing one, and it read as silence.

Two details worth keeping as they are: the - - sentinel (a line that cannot be mistaken for a sample, and cannot be summed into a rate), and not counting the interval and not advancing the cursor — so the next good read is a real baseline rather than a delta from a phantom. The self-test asserting after == (2, 5611) unchanged on a failed read pins that second property.

The earlier freeze fix still holds — re-ran the journalctl-shaped reproduction (file growing by 1 fault and +150 dropped) and it reports 2 350 FAULTING +250 dropped, +1 printed. So: two modes, an unreadable path, and deltas, all three correct at this head.

What I did not verify: behaviour when the source fails mid-run after a good sample — the arms cover a failing source from the start and a good one throughout, and the code path is the same for both, so I believe it; I just have not driven it.

This was my last standing finding on this family. It was reported on #2580 (review ask), #2598 (same) and #2603 (with the reproduction); the fix arrived in one PR once the reproduction was concrete, which is the lesson I would keep from it.

@bong-water-water-bong
bong-water-water-bong merged commit a197b8e into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-review/2542-sampler-unreadable branch November 16, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant