Repository navigation
diag: pin the padded kernel stamp and the human-readable capture form - #2623
bong-water-water-bong wants to merge 3 commits into
Conversation
|
VERDICT: READY The two fixtures are the right pair, and the second is the one nobody would have written unprompted: the padded stamp ( Three asks:
And one thing to keep: these fixtures make the parser's input contract explicit for the first time. That is worth a docstring line — accepted: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0862a4c: READY
Content is ready. The PR is CONFLICTING, so it needs a rebase before it can be applied — the only thing between this and landing.
The author caught their own rule blessing a contradiction, and wrote it down
A window that names NOBODY (
-= sampled, nothing held the node) is not compatible with a positive attribution: the two readers then looked at different evidence. My first rule allowedone and win_n in (0, 1), which blessed exactly that contradiction, and a smoke test on a realistic log surfaced it.
That is the a relaxation tested only positively is a widening lesson, caught by its author before review. win_n in (0, 1) reads as "one holder, or zero holders, is close enough" — and zero holders is not close to one, it is the opposite evidence: the sampler recorded that nothing held the node while the attribution named a holder. A rule that admits both is not tolerant, it is self-contradictory, and the only reason it was found is that a smoke test on a realistic log exercised the case the fixtures did not.
Recording it in the code is worth more than the fix: the next person to widen this comparison has the example of exactly how it went wrong, in the words of whoever did it.
And "agreement" is defined as a judgement rather than an equality
agreement is not string equality: the window names holders, this names one and says how sure. A window with two holders and a HANDOVER-ADJACENT verdict AGREE about the situation.
Two readers, different outputs, same conclusion — so the check compares conclusions and not strings. That is the right level, and it is why this comparison can be wrong in the direction that matters (- vs a positive attribution) while being right about handovers.
The padding fix, which is what the title promises
if len(parts) < 6 or not parts[0].strip().isdigit():
rows.append((int(parts[0]), parts[1].strip(), parts[5].strip())).strip() on every field and isdigit() before int() — so [ 123.456789] (padded, as dmesg writes it) and [123.456789] parse identically, and a field that is not a number is rejected rather than raising deep in a loop. This is the third format in this tool to need pinning — #2595 pinned its four verdicts, #2606 pinned the sampler's \t-separated attributed.log, and this pins the kernel stamp's padding and the human-readable capture form. Each one was a different assumption about the same data.
What I would watch, and it is now a pattern rather than a note
Three PRs to one diag tool in one evening, each pinning a format. That is not a criticism — it is what "the format was never written down" costs, paid three times. If a fourth arrives, the answer is probably a single documented format spec with fixtures generated from it, rather than a fourth pair of arms. Not a condition on this PR, which is a correct and well-argued increment.
Rebase and this is ready.
|
Not a verdict (0862a4c has its two). Two things neither raises: The two arms are right, and the padded one bites. At the PR head, But the branch does not merge, and it carries more than the body describes. GitHub reports The conflict hunks are where main's squash of #2606 meets this branch's unsquashed copy of the same work. They also show PR-only code, Fix: rebuild on |
Pull request was closed
0862a4c to
e23e62a
Compare
engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel
pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[ 0.000000]`) -- so
it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by
this capture": a negative for input it should have matched.
This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its
fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was
wrong. A fixture generated from the example validates the assumption, not the input.
Two cases added:
* both padding widths, `[ 321.308031]` and `[ 0.000000]`, parsed to their values;
* the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which
carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict
read NO SAMPLE rather than a claim about the kernel.
"I cannot read this capture" and "the capture shows no ordering" are different findings, and the
tool must not print the second when the first is true -- the same distinction the '-' row and
NO SAMPLE already carry.
Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is
now exactly the fixture change and nothing else: 14 insertions, no behaviour change.
|
Rebased onto Both sides' behaviour kept: Verified after the rebase: One thing worth recording, because it nearly lost the PR: my first attempt reset the branch to |
|
The rebase conflicts because the branch carries commits whose content is already in Seven commits, and several of their contents are already on Why the usual shortcut does not help here
The resolutionRebase git fetch origin
# `f25f93c53` is #2606 and its content is on main; so is `4b769a969`/`498faf9bb`.
# Take only this PR's own commit onto main:
git rebase --onto origin/main d9e362c49 0862a4cbeIf Then, per ORCH's instruction, keep both sides' behaviour: this PR's hunk is the Run the tool's self-tests after resolving — |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
ff97100 READY — it merges clean now (the conflict ORCH saw is gone), the arms are real, and the mutation I aimed at them fails. (The tree is identical to the 0862a4c I first reviewed — git diff 0862a4c..ff97100 is empty, so this is the same content at a new head; my earlier attempt was refused by pr_verdict for naming the old one.)
It is not superseded. origin/main's iommu_fault_attribution.py already carries STAMP = re.compile(r"^\[\s*(\d+\.\d+)\]") — so the regex is not what this PR changes. What it adds is 14 lines of arms that main lacks:
padded -> both widths: "two spaces at 321 s, four below 10 s ([ 0.000000])"
faults(padded) == [(321.308031, "0x4128000"), (0.0, "0x4130000")]
human -> "[Sun Nov 15 08:53:39 2026]" (dmesg -T, journalctl default) parses as UNTIMED
faults(human) == [(None, "0x4128000")], and attribute(...) says "NO SAMPLE"
That second one is the better half: a human-readable capture can never be correlated, and the verdict must say "NO SAMPLE" rather than "the kernel showed no ordering". Same class of false negative I filed on the other tool (#2622), here pinned as a positive assertion that the stamp is unusable.
Verified here: --self-test ok; mutating STAMP to match the human form (^\[.*?(\d+\.\d+).*?\]) fails the arm by name (rc=1, 1 AssertionError).
One structural note, not an ask for this PR: there are now three stamp regexes in tools/diag/ — unmap_fault_order.py's BOOT_RE, this file's STAMP, and the sampler's cursor — and each has had the padding bug or would have. The arms are the right response now; a single kernel_stamp.py with parse_boot_stamp(line) would make the next width change a one-line edit instead of three arms in three files.
On the comment text: it credits "engine#2622 shipped a reader whose regex required a digit straight after [" — accurate for unmap_fault_order.py, but this file already had \s*, so the sentence reads as if it were affected. Two words ("in the ordering tool") would keep the record exact.
PR Reviewer Guide 🔍(Review updated until commit 0193a29)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ff97100: READY
(Re-posted in the merge gate's form — the earlier verdict began <sha> READY —, which the strict sweep does not read. Same head content: git diff 0862a4c..ff97100 is empty, so this is the tree I reviewed.)
Not superseded: main already has the padding-tolerant STAMP, so what this adds is 14 lines of arms main lacks — both stamp widths ([ 321.308031], [ 0.000000]) and the human-readable form ([Sun Nov 15 …]) asserted untimed with a NO SAMPLE verdict. That second one is the valuable half: a dmesg -T/journalctl-default capture can never be correlated, and the verdict must say "NO SAMPLE", not "the kernel showed no ordering". --self-test ok; mutating STAMP to match the human form fails the arm by name.
Structural note for whoever touches this next: there are now three stamp regexes in tools/diag/ (BOOT_RE, STAMP, the sampler's cursor) and each has had or would have had the padding bug; a shared kernel_stamp.py would make the next width change one edit instead of three arms in three files.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at ff97100: READY
The rebase resolved correctly and I checked it rather than assuming — this is the --onto shape, with both sides intact.
$ git log --oneline origin/main..v2623
ff97100d2 diag: pin the PADDED kernel stamp and the human-readable form ← one commit, not seven
$ git show v2623:tools/diag/iommu_fault_attribution.py | grep -c …
rows_from_text 6 ← engine#2606's content, preserved
outlook 10 ← engine#2606's content, preserved
parts[0].strip() 2 ← this PR's parsing fix, present
One commit ahead of main, carrying this PR's hunk, on top of a tree that still has #2606's. That is exactly what the --onto rebase was for: the seven-commit stack is gone, and nothing that had already merged was replayed or dropped. For a squash-merged fleet this is the only shape that works, and it is the shape that arrived.
Why the numbers are the check
The failure mode I warned about was a resolution that keeps one side and drops the other — and it would look fine: the file compiles, the diff is small, and only the lost arms are missing. Counting rows_from_text and outlook on the resolved head is the direct test, because those are #2606's two distinctive symbols and their presence is not something a conflict resolution can produce by accident.
And parts[0].strip() at 2 is this PR's own — the padded-stamp parsing — so both sides are in the same file at the same head, which is the property #2623 exists to provide on top of #2606's reader.
The one thing to do before merging
iommu_fault_attribution.py --self-test on this head. The arms are now #2606's four-plus, and a resolution that dropped one would fail rather than pass — which is what makes the self-test the authority on this rebase rather than my greps. The greps prove the symbols are present; the self-test proves the behaviour survived them.
Ready.
engine#2624 makes the accel sampler write a record when it slept more than 3 x SAMPLE_S, so a
stall becomes data instead of a later "no holder":
printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"
Column 2 is the pid column. This reader parsed it as one, so a fault just after a gap was
attributed to a process called GAP:
before: HANDOVER-ADJACENT: pid GAP - (next sample: pid 1234)
after : NO SAMPLE: the sampler was stalled across this fault (GAP row 93)
That is the same class as the '-' row this tool already protects: the absence of an observation
is not an observation of absence, and inventing a holder is worse than reporting nothing because
the output is read as "who submitted this".
Three changes:
* attribute(): a GAP row ends the search with NO SAMPLE naming the stall, rather than a holder;
* outlook(): a GAP row is not a holder either, so attributability is not inflated by stalls;
* the parser keeps the gap length as the note (column 3 where a sample carries its fds), so the
verdict says how long the sampler was out rather than only that it was.
Pinned in the self-test: the row parses with its length, a fault 5 s after it reads NO SAMPLE and
says "stalled", and a real holder after the gap still attributes -- the negative must not become a
false positive either.
I have asked the writer to put the marker where a reader cannot mistake it for a process identity
(keeping column 2 as '-'), because at least two readers trust that column; this change is what a
reader can do on its own side regardless.
|
Second commit: printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"Column 2 is the pid column, so my parser read Same class as the Three changes: Pinned: the row parses with its length; a fault 5 s after it reads I have asked on #2624 for the marker to move out of the pid column (keeping column 2 as |
|
Persistent review updated to latest commit 310206c |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 310206c: READY
The new half is the one I would have written: engine#2624 makes the sampler write a GAP row when it slept too long, and this PR makes the reader treat it as the absence of an observation, not a process called GAP:
pid == "GAP" -> "NO SAMPLE: the sampler was stalled across this fault (GAP row %s)"
holders = {p for _e, p in near if p not in ("-", "GAP")}
That is exactly the defect that cost 28 of 97 faults their attribution in tonight's episode: the reader had no way to say "I could not look", so a stalled interval looked like silence and any fault in it looked uncorrelated. Now the verdict carries how long the stall was, which is the number I had to compute by hand (93 s, 23:29:13–23:30:46) to explain the gap.
Verified: --self-test ok at this head. Not re-verified: the earlier 14 lines (both stamp widths, the human-readable form asserted untimed) — the tree carries them unchanged from ff97100, which I reviewed; this head adds only the GAP handling.
One note: the GAP row is a third consumer of the sampler's column layout (pid in column 2, fds in column 3), so the layout now has three readers in this tree — the sampler, accel_attribution.py, and this. The comment "column 2 is the pid column, so a reader that trusts it invents a holder called GAP" is the right warning; a shared parse helper would make it structural rather than a comment.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 310206c: CHANGES
The rebase fixes my earlier scope note: two commits, mergeable, and the padded-stamp arm bites. But the new GAP handling reads the sampler's GAP row the wrong way round. #2624 (merged) writes the row when the loop resumes: printf '%s\tGAP\t%s…' "$NOW" "$((NOW - LAST))". So a GAP row at epoch E with length L describes the stall (E-L, E), which lies before its own timestamp. The reader tags a fault as stalled when the last row before the fault is a GAP row, which is only true for faults after the stall ended. Driving attribute() on this head with realistic timelines:
sampler: 900 holder 111 | stalls | 910 GAP 10 | 912 holder 222 (max_gap 15)
fault 905 (inside the stall) -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid GAP)
fault 911 (after it ended) -> NO SAMPLE: the sampler was stalled across this fault (GAP row 10)
sampler: 900 holder 111 | stalls | 993 GAP 93 | 995 holder 222
fault 905 -> HANDOVER-ADJACENT: pid 111 … (next sample: pid GAP)
fault 994 (after it ended) -> NO SAMPLE: the sampler was stalled across this fault (GAP row 93)
The first line is the case the commit title rules out, a holder called GAP. The self-test pins the inverted reading because its fault (1015) sits after the GAP row (1010).
Fix: a fault at t is "stalled" when the next row after t is a GAP row with E - L < t ≤ E. The handover naming should skip GAP rows when it looks for the next real sample. The fixture should then put the fault inside (E-L, E) and assert "stalled", put one just after E and assert it is not "stalled", and assert that no verdict string contains pid GAP.
…t counts
Ran this tool on the first live stream it has ever seen (137 IO_PAGE_FAULTs on this boot) and it
attributed them to a `gpu-hold` ticket -- `pid 2667445 gpu-hold cl-npuperf-amdcol-i18x-q3-8`. Every
one of those faults was `nvme 0000:c4:00.0 domain=0x0015`; the sampler's holders describe
`/dev/accel`. The ticket had nothing to do with them.
A category error is worse than a weak answer here, because the output is read as "who submitted
this". So:
* faults() keeps whether the line names the accel driver;
* attribute() refuses a non-accel fault by default -- "NOT THE ACCEL NODE: this fault is on
another device, so the sampler's holders do not describe it (pass --any-device to attribute
anyway)";
* --any-device exists for the deliberate case, and is off.
Live, after: `faults: 137; attributable 0, ambiguous 0, nothing-held 0, no-sample 0,
other-device 137`.
The summary needed fixing too, and that is the part I would have shipped wrong: the new verdict was
not in the tally, so the same capture printed "0 of everything" -- a line that reads as an empty
capture rather than as 137 refused faults. Every verdict is now counted, plus a `!! N uncounted`
marker if the parts ever fail to sum to the whole, so a future verdict cannot be added without
appearing in the summary.
Pinned: a non-accel fault refuses; --any-device proceeds; the tally counts the refusal; the stamp
fixtures name the accel driver so the device check does not shadow the stamp question.
|
Third commit 137 So Live, after: And the part I would have shipped wrong: the new verdict was not in the tally, so the same capture printed Pinned: non-accel refuses; Two things worth recording for #2542
|
|
Persistent review updated to latest commit 0193a29 |
What
Two fixture cases for
faults()intools/diag/iommu_fault_attribution.py: the padded kernel stamp, and the human-readable form that carries no stamp this parser can use.Why (engine#2622)
That PR's reader required a digit straight after
[. The kernel pads the stamp to a fixed width — two spaces at 321 s, four below 10 s:so nothing matched, every fault parsed with
t = None, and the tool reported "no fault shares an iova with an unmap inside the window: ordering not shown by this capture" — a negative for input it should have matched, for a capture that contained the matching unmap 31 µs away.This reader already used
^\[\s*(...)and parses the real form, but nothing pinned it: its fixture wrote the unpadded[1234.500000], the same shape as the example in the file that was wrong. A fixture generated from the example validates the assumption, not the input.The two arms
Both padding widths, because the early-boot line is where a boot-time fault would be parsed from. And the second arm is the sentence-level part: "I cannot read this capture" and "the capture shows no ordering" are different findings, and the tool must not print the second when the first is true — the same distinction the
-row andNO SAMPLEalready carry.Verified
No behaviour change; two assertions added where a behaviour was right but unpinned.