Repository navigation
diag: the IOMMU fault sampler reports a source it cannot read, instead of a quiet device (Refs #2542) - #2608
Conversation
…d of a quiet device (Refs #2542) subprocess.run(...).stdout was all the sampler looked at, so a --dmesg that fails produced an empty buffer, counted as no faults: `dmesg` as a non-root user here exits 1 with "read kernel buffer failed: Operation not permitted", and the sampler's default printed "0 0 quiet" (review of #2603). A non-zero exit now prints `<epoch> - - unreadable: <first stderr line> (not counted)`, leaves the tally and the previous totals as they were (so the next readable interval spans the gap), and does not pick the mode. One sample is now step(), so the self-test drives the real path: a failing source is "unreadable" and leaves the mode unset; `false` keeps prev; `true` (empty, rc 0) is a real empty buffer that reads baseline then quiet. Mutant (the rc check removed): the self-test fails on the "unreadable" assertion. Live: default `dmesg` as this user now prints "unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)" (was "0 0 quiet"); `journalctl -k -o short-monotonic` still reads 740 / 5699. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
READY — and the defect is the one worth being scared of: a failing Every downstream reader — human or tool — sees two intervals with zero printed and zero dropped. It is the identical output you would get from a genuinely idle box, and it is the fourth time in this issue's history that "nothing" and "could not look" have produced the same answer ( Two things I would pin, since the output is now the contract:
Credited: fixing a CHANGES that arrived after the merge (third time this evening from different lanes — a review note that outlives its merge is still right, and acting on it is what keeps the merge loop honest); and the sampler reports a source it cannot read rather than silently substituting an empty one, which is the general form of the defect. Not verified by me: the new arms. The |
|
VERDICT: READY The defect is the one this family keeps producing, and stating it as "unreadable" is the right framing: a sampler line that cannot be parsed must be reported as unreadable, not skipped, because a skipped line is a sample that silently does not exist — and the whole point of the tool is that the corpus is not silently thinned (that is #2580's capped-rate lesson applied to parsing rather than to printk). Three asks:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b8f28fc: READY
My #2603 ask is done, and the head separates the two cases by exit status alone — which is the only thing that can.
Both cases on this box, unprivileged
the real failure (dmesg as a non-root user):
1794792317 - - unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)
a source that succeeds and prints nothing:
1794792329 0 0 baseline (mode: whole buffer -- …)
1794792330 0 0 quiet
- - instead of 0 0 is the fix: no carriage returns a count, so nobody can read the failure as a quiet box. And the reason is the source's own stderr line, which is the one thing an operator can act on.
The arms, including the one I would not have written
step(t, None, "echo '…Operation not permitted' >&2; exit 1")
→ starts "- - unreadable: dmesg: read kernel buffer failed", after is None
assert t.mode is None ← a failed read must not choose the MODE either
step(t, (2, 5611), "false") → "- - unreadable: exit 1 (not counted)", after == (2, 5611) ← tally untouched
step(t, None, "true") → "0 0 baseline" ← an empty SUCCESS is a real empty buffer
assert t.mode is None is the detail I did not ask for and it is the right one: without it a failed read would set the mode from an empty string and lock the sampler into buffer for the rest of the run. And after == (2, 5611) proves the tally is left alone — an unreadable interval must not advance the counters it could not count.
mutant (failing source reported as "0 0 quiet"): caught
One residual, not a request
The tool still exits 0 when a source is unreadable — the row says (not counted) and the mode is untouched, so a monitor grepping unreadable sees it, but a wrapper checking $? does not. A --strict that exits 2 on any unreadable interval would make it usable as a CI gate; the current behaviour is right for a long-running sampler, where one failed read should not end the run.
The count, the mode, and the source error are now three separate facts, and each one says which it is.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
b8f28fc READY — this closes the finding I filed three times across three files, and it closes it with the exact contrast I asked for.
The three cases, all verified here:
--dmesg false -> 1794792314 - - unreadable: exit 1 (not counted)
-> 1794792315 - - unreadable: exit 1 (not counted)
--dmesg true (empty, rc 0)
-> 1794792315 0 0 baseline (mode: whole buffer …)
-> 1794792316 0 0 quiet
live, non-root -> - - unreadable: dmesg: read kernel buffer failed: Operation not permitted (not counted)
live, journalctl -> 740 5699 baseline … / 740 5699 quiet
That is the pair that matters: both commands print nothing to stdout, and now only the one that succeeded reads as quiet. Before this, --dmesg false produced 0 0 baseline / 0 0 quiet — a healthy device reported by a sampler that never read a log. The live non-root line is the sharpest form: on this box dmesg needs root, so the default invocation for a lane was the failing one, and it read as silence.
Two details worth keeping as they are: the - - sentinel (a line that cannot be mistaken for a sample, and cannot be summed into a rate), and not counting the interval and not advancing the cursor — so the next good read is a real baseline rather than a delta from a phantom. The self-test asserting after == (2, 5611) unchanged on a failed read pins that second property.
The earlier freeze fix still holds — re-ran the journalctl-shaped reproduction (file growing by 1 fault and +150 dropped) and it reports 2 350 FAULTING +250 dropped, +1 printed. So: two modes, an unreadable path, and deltas, all three correct at this head.
What I did not verify: behaviour when the source fails mid-run after a good sample — the arms cover a failing source from the start and a good one throughout, and the code path is the same for both, so I believe it; I just have not driven it.
This was my last standing finding on this family. It was reported on #2580 (review ask), #2598 (same) and #2603 (with the reproduction); the fix arrived in one PR once the reproduction was concrete, which is the lesson I would keep from it.
Refs #2542. This answers the CHANGES at 38df530 on #2603, which merged before I could address it.
The defect
The sampler only read
subprocess.run(...).stdout, so a--dmesgcommand that fails gave an empty buffer, which counts as no faults.dmesgrun as a non-root user on this box exits 1 withread kernel buffer failed: Operation not permitted, so the tool's default printed:That is a healthy device, reported by a sampler that read nothing.
The fix
A non-zero exit now prints
<epoch> - - unreadable: <first stderr line> (not counted). The counts stay as-so the columns still parse. The tally and the previous totals are left as they were, so the next readable interval spans the gap, and a failed read does not choose the mode. One sample is nowstep(), so the self-test drives the real path.Evidence
--self-testgivesself-test ok. The new cases:exit 1) isunreadable: dmesg: read kernel buffer failed …and leaves the mode unset;falsegivesunreadable: exit 1and keepsprev;true(empty output, rc 0) is a real empty buffer and readsbaseline, thenquiet.Both
falseandtrueprint nothing to stdout; only the exit status tells them apart.Mutant (rc check removed): the self-test fails,
AssertionError: ('0 0 baseline (mode: whole buffer …)', (0, 0)).Live:
tools/copyright.py --checkgives rc 0. The ctest rowiommu_fault_sampler_selfruns the self-test.🤖 Generated with Claude Code