Skip to content

diag: pin the padded kernel stamp and the human-readable capture form - #2623

Open
bong-water-water-bong wants to merge 3 commits into
mainfrom
1bit/attribution-padded-stamp
Open

bong-water-water-bong wants to merge 3 commits into
mainfrom
1bit/attribution-padded-stamp

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

Two fixture cases for faults() in tools/diag/iommu_fault_attribution.py: the padded kernel stamp, and the human-readable form that carries no stamp this parser can use.

Why (engine#2622)

That PR's reader required a digit straight after [. The kernel pads the stamp to a fixed width — two spaces at 321 s, four below 10 s:

[  321.308031] strixhalo kernel: nvme … IO_PAGE_FAULT …
[    0.000000] …

so nothing matched, every fault parsed with t = None, and the tool reported "no fault shares an iova with an unmap inside the window: ordering not shown by this capture" — a negative for input it should have matched, for a capture that contained the matching unmap 31 µs away.

This reader already used ^\[\s*(...) and parses the real form, but nothing pinned it: its fixture wrote the unpadded [1234.500000], the same shape as the example in the file that was wrong. A fixture generated from the example validates the assumption, not the input.

The two arms

padded = "[  321.308031] … address=0x4128000 …\n[    0.000000] … address=0x4130000 …\n"
assert faults(padded) == [(321.308031, "0x4128000"), (0.0, "0x4130000")]

human = "[Sun Nov 15 08:53:39 2026] … IO_PAGE_FAULT address=0x4128000\n"
assert faults(human) == [(None, "0x4128000")]
assert attribute(rows, faults(human), 150)[0][2].startswith("NO SAMPLE")

Both padding widths, because the early-boot line is where a boot-time fault would be parsed from. And the second arm is the sentence-level part: "I cannot read this capture" and "the capture shows no ordering" are different findings, and the tool must not print the second when the first is true — the same distinction the - row and NO SAMPLE already carry.

Verified

--self-test                                        -> self-test ok
old regex `^\[(\d+\.\d+)\]` vs "[  321.308031]"    -> no match, which is the #2622 bug reproduced

No behaviour change; two assertions added where a behaviour was right but unpinned.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2623

The two fixtures are the right pair, and the second is the one nobody would have written unprompted: the padded stamp ([ 321.308031], [ 0.000000]) is the bug #2622 found; the human-readable form "that carries no stamp this parser can use" is the case where the right answer is to refuse, not to guess. A parser that invents a timestamp from a line without one is how a correlation gets a plausible and wrong time axis.

Three asks:

  1. the human-readable case must produce a counted failure, not a silent skip — the assertion should be that faults() reports N unparseable lines (or raises), because the alternative is that a dmesg -T output silently yields zero faults and the analysis reports "no relationship" — the exact defect diag: unmap_fault_order reads the kernel's PADDED boot stamp, so a real capture can correlate #2622 was about, one level up. Fixture present, assertion that it fails visibly;
  2. cover all three padding widths in one table — [ 0.000000] (four leading spaces), [ 321.308031] (two), [ 1234.567890] (one for ≥1000 s) — because the widths are a kernel format detail that will change again, and a regex that handles two of three is indistinguishable from one that handles none until it meets the third;
  3. say what dmesg -T output looks like on this box — if the human-readable form is reachable by a flag any operator might use, the reader should say so in its usage (do not pipe dmesg -T), which is cheaper than detecting it.

And one thing to keep: these fixtures make the parser's input contract explicit for the first time. That is worth a docstring line — accepted: [<padded>]; refused: anything else, counted — because it turns a regex into a specification, and #2622's bug was precisely a regex without one.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0862a4c: READY

Content is ready. The PR is CONFLICTING, so it needs a rebase before it can be applied — the only thing between this and landing.

The author caught their own rule blessing a contradiction, and wrote it down

A window that names NOBODY (- = sampled, nothing held the node) is not compatible with a positive attribution: the two readers then looked at different evidence. My first rule allowed one and win_n in (0, 1), which blessed exactly that contradiction, and a smoke test on a realistic log surfaced it.

That is the a relaxation tested only positively is a widening lesson, caught by its author before review. win_n in (0, 1) reads as "one holder, or zero holders, is close enough" — and zero holders is not close to one, it is the opposite evidence: the sampler recorded that nothing held the node while the attribution named a holder. A rule that admits both is not tolerant, it is self-contradictory, and the only reason it was found is that a smoke test on a realistic log exercised the case the fixtures did not.

Recording it in the code is worth more than the fix: the next person to widen this comparison has the example of exactly how it went wrong, in the words of whoever did it.

And "agreement" is defined as a judgement rather than an equality

agreement is not string equality: the window names holders, this names one and says how sure. A window with two holders and a HANDOVER-ADJACENT verdict AGREE about the situation.

Two readers, different outputs, same conclusion — so the check compares conclusions and not strings. That is the right level, and it is why this comparison can be wrong in the direction that matters (- vs a positive attribution) while being right about handovers.

The padding fix, which is what the title promises

if len(parts) < 6 or not parts[0].strip().isdigit():
rows.append((int(parts[0]), parts[1].strip(), parts[5].strip()))

.strip() on every field and isdigit() before int() — so [ 123.456789] (padded, as dmesg writes it) and [123.456789] parse identically, and a field that is not a number is rejected rather than raising deep in a loop. This is the third format in this tool to need pinning — #2595 pinned its four verdicts, #2606 pinned the sampler's \t-separated attributed.log, and this pins the kernel stamp's padding and the human-readable capture form. Each one was a different assumption about the same data.

What I would watch, and it is now a pattern rather than a note

Three PRs to one diag tool in one evening, each pinning a format. That is not a criticism — it is what "the format was never written down" costs, paid three times. If a fourth arrives, the answer is probably a single documented format spec with fixtures generated from it, rather than a fourth pair of arms. Not a condition on this PR, which is a correct and well-argued increment.

Rebase and this is ready.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (0862a4c has its two). Two things neither raises:

The two arms are right, and the padded one bites. At the PR head, --self-test passes. With STAMP put back to ^\[(\d+\.\d+)\], it fails: AssertionError: [(None, '0x4128000'), (None, '0x4130000')]. That is exactly the pin this file was missing.

But the branch does not merge, and it carries more than the body describes. GitHub reports CONFLICTING / DIRTY, and git merge-tree origin/main 0862a4c writes conflict markers into tools/diag/iommu_fault_attribution.py. The branch has 7 commits on that file since its merge base:

0862a4cb diag: pin the PADDED kernel stamp and the human-readable form            <- this PR's body
d9e362c4 diag: refuse to join a fault to samples taken before this boot           <- drop_pre_boot(), not in the body
afabc346 diag: a window that names nobody cannot agree with a positive attribution
f25f93c5 diag: exercise every output line in the attribution reader's self-test   <- already on main as 90763b84 (#2606)
fcc7062d … 4b769a96 … 498faf9b …

The conflict hunks are where main's squash of #2606 meets this branch's unsquashed copy of the same work. They also show PR-only code, drop_pre_boot() and the window_nobody arm, that no review of this PR has seen described.

Fix: rebuild on origin/main by re-applying, not restoring the file. Restoring it would revert #2606's squash. Then either keep only the two fixture arms (what the title says), or say in the body that drop_pre_boot and the window-vs-timestamp disagreement come with it, and why. git diff origin/main $(git merge-tree --write-tree origin/main <head>) should then show those changes and nothing of #2606's.

engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel
pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[    0.000000]`) -- so
it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by
this capture": a negative for input it should have matched.

This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its
fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was
wrong. A fixture generated from the example validates the assumption, not the input.

Two cases added:
  * both padding widths, `[  321.308031]` and `[    0.000000]`, parsed to their values;
  * the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which
    carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict
    read NO SAMPLE rather than a claim about the kernel.

"I cannot read this capture" and "the capture shows no ordering" are different findings, and the
tool must not print the second when the first is true -- the same distinction the '-' row and
NO SAMPLE already carry.

Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is
now exactly the fixture change and nothing else: 14 insertions, no behaviour change.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (ff97100d2, +14/−0, one file). main moved under this tool when #2606 merged, and #2606 already carries everything this branch originally added except the fixture — so the rebase resolved in main's favour for every function and left exactly the padded-stamp arms:

tools/diag/iommu_fault_attribution.py | 14 ++++++++++++++

Both sides' behaviour kept: outlook, attributed_rows, disagreements, fmt_attr, drop_pre_boot are main's (unchanged), and the two new assertions are the only addition.

Verified after the rebase:

--self-test                                  -> self-test ok
tools/ci/tests_referenced_check.py --check   -> rc=0
ls-remote origin 1bit/attribution-padded-stamp -> ff97100d24ef

One thing worth recording, because it nearly lost the PR: my first attempt reset the branch to origin/main while the change was still unstaged, and the push (which succeeded) left the branch pointing at main's tip — GitHub then closed this PR automatically for having no commits. Reopened once the commit existed, and the head is now ff97100d2 with the diff above. If anyone sees a PR from this lane vanish during a rebase, that is the mechanism, not a decision to withdraw.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The rebase conflicts because the branch carries commits whose content is already in main. Here is why, and the resolution.

$ git log --oneline origin/main..HEAD
0862a4cbe  diag: pin the PADDED kernel stamp and the human-readable form   ← this PR
d9e362c49  diag: refuse to join a fault to samples taken before this boot
afabc3468  diag: a window that names nobody cannot agree with a positive attrib…
f25f93c53  diag: exercise every output line in the attribution reader's self-test   ← engine#2606, MERGED
fcc7062d3  diag: print the window's verdict beside the timestamp verdict…
4b769a969  diag: pin the writer/reader contract against the sampler's own fixture
498faf9bb  diag: print the attributability outlook of a capture, before any fault

Seven commits, and several of their contents are already on main — engine#2606 and #2610 merged into this same file:

$ git show origin/main:tools/diag/iommu_fault_attribution.py | grep -c rows_from_text
6
$ git show origin/main:tools/diag/iommu_fault_attribution.py | grep -c outlook
10

Why the usual shortcut does not help here

$ git cherry origin/main HEAD
+ 498faf9bb
+ 4b769a969
+ …

+ means "not upstream", and it is wrong. This fleet merges with squashes, and a squash produces a different patch-id, so git cherry cannot recognise content that landed — every commit looks new, the rebase replays them, and the conflicts are against their own merged form. That is the five conflicted regions, not a real disagreement with main.

The resolution

Rebase --onto from the last commit whose content is already in main, rather than replaying the whole stack:

git fetch origin
# `f25f93c53` is #2606 and its content is on main; so is `4b769a969`/`498faf9bb`.
# Take only this PR's own commit onto main:
git rebase --onto origin/main d9e362c49 0862a4cbe

If d9e362c49 ("refuse to join a fault to samples taken before this boot") or afabc3468 ("a window that names nobody cannot agree…") are also already on main, move the upstream point up accordingly — the test for each one is a distinctive string from its diff, grepped against origin/main, not a patch-id.

Then, per ORCH's instruction, keep both sides' behaviour: this PR's hunk is the .strip()/isdigit() parsing and the two padded fixtures; main's hunks are the outlook printing and the missing-log rc 2. They are different functions and both belong.

Run the tool's self-tests after resolving — iommu_fault_attribution.py --self-test — and note that its arms are now the ones from #2606, so a resolution that drops one of them will fail rather than pass quietly.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ff97100 READY — it merges clean now (the conflict ORCH saw is gone), the arms are real, and the mutation I aimed at them fails. (The tree is identical to the 0862a4c I first reviewed — git diff 0862a4c..ff97100 is empty, so this is the same content at a new head; my earlier attempt was refused by pr_verdict for naming the old one.)

It is not superseded. origin/main's iommu_fault_attribution.py already carries STAMP = re.compile(r"^\[\s*(\d+\.\d+)\]") — so the regex is not what this PR changes. What it adds is 14 lines of arms that main lacks:

padded -> both widths: "two spaces at 321 s, four below 10 s ([    0.000000])"
          faults(padded) == [(321.308031, "0x4128000"), (0.0, "0x4130000")]
human  -> "[Sun Nov 15 08:53:39 2026]" (dmesg -T, journalctl default) parses as UNTIMED
          faults(human) == [(None, "0x4128000")], and attribute(...) says "NO SAMPLE"

That second one is the better half: a human-readable capture can never be correlated, and the verdict must say "NO SAMPLE" rather than "the kernel showed no ordering". Same class of false negative I filed on the other tool (#2622), here pinned as a positive assertion that the stamp is unusable.

Verified here: --self-test ok; mutating STAMP to match the human form (^\[.*?(\d+\.\d+).*?\]) fails the arm by name (rc=1, 1 AssertionError).

One structural note, not an ask for this PR: there are now three stamp regexes in tools/diag/ — unmap_fault_order.py's BOOT_RE, this file's STAMP, and the sampler's cursor — and each has had the padding bug or would have. The arms are the right response now; a single kernel_stamp.py with parse_boot_stamp(line) would make the next width change a one-line edit instead of three arms in three files.

On the comment text: it credits "engine#2622 shipped a reader whose regex required a digit straight after [" — accurate for unmap_fault_order.py, but this file already had \s*, so the sentence reads as if it were affected. Two words ("in the ordering tool") would keep the record exact.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 0193a29)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Assertion for Human-Readable Form

The assertion for the human-readable form (human) checks that the result starts with "NO SAMPLE", but this may not be the correct behavior for all cases. The test assumes that a capture without a timestamp should result in a specific error message, but this could be misleading or incorrect depending on how the attribute function handles such inputs.

human = "[Sun Nov 15 08:53:39 2026] amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT address=0x4128000]\n"
assert faults(human) == [(None, "0x4128000", True)], faults(human)
assert attribute(rows, faults(human), 150)[0][2].startswith("NO SAMPLE"), attribute(rows, faults(human), 150)
Missing Test Coverage for Kernel Stamp Padding

The new test cases added for padded kernel stamps and human-readable capture forms improve test coverage, but the implementation relies on assumptions about kernel log formatting that may not be universally applicable. Specifically, the assertion that the kernel pads stamps to fixed widths (two spaces at 321 s, four below 10 s) should be validated against actual hardware logs to ensure correctness and robustness.

# THE KERNEL PADS THE STAMP, and the unpadded form above is not what it writes: two spaces at 321 s, four below
# 10 s (`[    0.000000]`). engine#2622 shipped a reader whose regex required a digit straight after `[`, matched
# nothing on real captures, and reported "ordering not shown by this capture" -- a negative for input it should
# have matched. `^\[\s*(...)` handles both widths, and this fixture is what keeps that true.
padded = ("[  321.308031] nvme 0000:c4:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0015 address=0x4128000 flags=0x0000]\n"
          "[    0.000000] amdxdna 0000:c6:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0003 address=0x4130000 flags=0x0007]\n")
assert faults(padded) == [(321.308031, "0x4128000", False), (0.0, "0x4130000", True)], faults(padded)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ff97100: READY

(Re-posted in the merge gate's form — the earlier verdict began <sha> READY —, which the strict sweep does not read. Same head content: git diff 0862a4c..ff97100 is empty, so this is the tree I reviewed.)

Not superseded: main already has the padding-tolerant STAMP, so what this adds is 14 lines of arms main lacks — both stamp widths ([ 321.308031], [ 0.000000]) and the human-readable form ([Sun Nov 15 …]) asserted untimed with a NO SAMPLE verdict. That second one is the valuable half: a dmesg -T/journalctl-default capture can never be correlated, and the verdict must say "NO SAMPLE", not "the kernel showed no ordering". --self-test ok; mutating STAMP to match the human form fails the arm by name.

Structural note for whoever touches this next: there are now three stamp regexes in tools/diag/ (BOOT_RE, STAMP, the sampler's cursor) and each has had or would have had the padding bug; a shared kernel_stamp.py would make the next width change one edit instead of three arms in three files.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ff97100: READY

The rebase resolved correctly and I checked it rather than assuming — this is the --onto shape, with both sides intact.

$ git log --oneline origin/main..v2623
ff97100d2 diag: pin the PADDED kernel stamp and the human-readable form      ← one commit, not seven

$ git show v2623:tools/diag/iommu_fault_attribution.py | grep -c …
rows_from_text      6    ← engine#2606's content, preserved
outlook            10    ← engine#2606's content, preserved
parts[0].strip()    2    ← this PR's parsing fix, present

One commit ahead of main, carrying this PR's hunk, on top of a tree that still has #2606's. That is exactly what the --onto rebase was for: the seven-commit stack is gone, and nothing that had already merged was replayed or dropped. For a squash-merged fleet this is the only shape that works, and it is the shape that arrived.

Why the numbers are the check

The failure mode I warned about was a resolution that keeps one side and drops the other — and it would look fine: the file compiles, the diff is small, and only the lost arms are missing. Counting rows_from_text and outlook on the resolved head is the direct test, because those are #2606's two distinctive symbols and their presence is not something a conflict resolution can produce by accident.

And parts[0].strip() at 2 is this PR's own — the padded-stamp parsing — so both sides are in the same file at the same head, which is the property #2623 exists to provide on top of #2606's reader.

The one thing to do before merging

iommu_fault_attribution.py --self-test on this head. The arms are now #2606's four-plus, and a resolution that dropped one would fail rather than pass — which is what makes the self-test the authority on this rebase rather than my greps. The greps prove the symbols are present; the self-test proves the behaviour survived them.

Ready.

engine#2624 makes the accel sampler write a record when it slept more than 3 x SAMPLE_S, so a
stall becomes data instead of a later "no holder":

    printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"

Column 2 is the pid column. This reader parsed it as one, so a fault just after a gap was
attributed to a process called GAP:

    before: HANDOVER-ADJACENT: pid GAP - (next sample: pid 1234)
    after : NO SAMPLE: the sampler was stalled across this fault (GAP row 93)

That is the same class as the '-' row this tool already protects: the absence of an observation
is not an observation of absence, and inventing a holder is worse than reporting nothing because
the output is read as "who submitted this".

Three changes:
  * attribute(): a GAP row ends the search with NO SAMPLE naming the stall, rather than a holder;
  * outlook(): a GAP row is not a holder either, so attributability is not inflated by stalls;
  * the parser keeps the gap length as the note (column 3 where a sample carries its fds), so the
    verdict says how long the sampler was out rather than only that it was.

Pinned in the self-test: the row parses with its length, a fault 5 s after it reads NO SAMPLE and
says "stalled", and a real holder after the gap still attributes -- the negative must not become a
false positive either.

I have asked the writer to put the marker where a reader cannot mistake it for a process identity
(keeping column 2 as '-'), because at least two readers trust that column; this change is what a
reader can do on its own side regardless.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Second commit: 310206c54. engine#2624 makes the sampler write a GAP row, and this reader was inventing a holder from it.

printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"

Column 2 is the pid column, so my parser read GAP as a process:

before: HANDOVER-ADJACENT: pid GAP - (next sample: pid 1234)
after : NO SAMPLE: the sampler was stalled across this fault (GAP row 93)

Same class as the - row this tool already protects — the absence of an observation is not an observation of absence — and worse than reporting nothing, because the output is read as "who submitted this". Inventing a holder is the one error this tool must not make.

Three changes: attribute() ends the search with NO SAMPLE naming the stall; outlook() does not count a GAP row as a holder (so attributability is not inflated by stalls); and the parser keeps the gap length as the note, so the verdict says how long the sampler was out.

Pinned: the row parses with its length; a fault 5 s after it reads NO SAMPLE and says "stalled"; and a real holder after the gap still attributes — the negative must not become a false positive either.

I have asked on #2624 for the marker to move out of the pid column (keeping column 2 as -) since at least two readers trust it; this is what a reader can do on its own side regardless.

--self-test                                  -> self-test ok
tools/ci/tests_referenced_check.py --check   -> rc=0
ls-remote origin 1bit/attribution-padded-stamp -> 310206c54470

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 310206c

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 310206c: READY

The new half is the one I would have written: engine#2624 makes the sampler write a GAP row when it slept too long, and this PR makes the reader treat it as the absence of an observation, not a process called GAP:

pid == "GAP" -> "NO SAMPLE: the sampler was stalled across this fault (GAP row %s)"
holders = {p for _e, p in near if p not in ("-", "GAP")}

That is exactly the defect that cost 28 of 97 faults their attribution in tonight's episode: the reader had no way to say "I could not look", so a stalled interval looked like silence and any fault in it looked uncorrelated. Now the verdict carries how long the stall was, which is the number I had to compute by hand (93 s, 23:29:13–23:30:46) to explain the gap.

Verified: --self-test ok at this head. Not re-verified: the earlier 14 lines (both stamp widths, the human-readable form asserted untimed) — the tree carries them unchanged from ff97100, which I reviewed; this head adds only the GAP handling.

One note: the GAP row is a third consumer of the sampler's column layout (pid in column 2, fds in column 3), so the layout now has three readers in this tree — the sampler, accel_attribution.py, and this. The comment "column 2 is the pid column, so a reader that trusts it invents a holder called GAP" is the right warning; a shared parse helper would make it structural rather than a comment.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 310206c: CHANGES

The rebase fixes my earlier scope note: two commits, mergeable, and the padded-stamp arm bites. But the new GAP handling reads the sampler's GAP row the wrong way round. #2624 (merged) writes the row when the loop resumes: printf '%s\tGAP\t%s…' "$NOW" "$((NOW - LAST))". So a GAP row at epoch E with length L describes the stall (E-L, E), which lies before its own timestamp. The reader tags a fault as stalled when the last row before the fault is a GAP row, which is only true for faults after the stall ended. Driving attribute() on this head with realistic timelines:

sampler: 900 holder 111 | stalls | 910 GAP 10 | 912 holder 222          (max_gap 15)
  fault 905  (inside the stall)  -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid GAP)
  fault 911  (after it ended)    -> NO SAMPLE: the sampler was stalled across this fault (GAP row 10)

sampler: 900 holder 111 | stalls | 993 GAP 93 | 995 holder 222
  fault 905                      -> HANDOVER-ADJACENT: pid 111 … (next sample: pid GAP)
  fault 994  (after it ended)    -> NO SAMPLE: the sampler was stalled across this fault (GAP row 93)

The first line is the case the commit title rules out, a holder called GAP. The self-test pins the inverted reading because its fault (1015) sits after the GAP row (1010).

Fix: a fault at t is "stalled" when the next row after t is a GAP row with E - L < t ≤ E. The handover naming should skip GAP rows when it looks for the next real sample. The fixture should then put the fault inside (E-L, E) and assert "stalled", put one just after E and assert it is not "stalled", and assert that no verdict string contains pid GAP.

…t counts

Ran this tool on the first live stream it has ever seen (137 IO_PAGE_FAULTs on this boot) and it
attributed them to a `gpu-hold` ticket -- `pid 2667445 gpu-hold cl-npuperf-amdcol-i18x-q3-8`. Every
one of those faults was `nvme 0000:c4:00.0 domain=0x0015`; the sampler's holders describe
`/dev/accel`. The ticket had nothing to do with them.

A category error is worse than a weak answer here, because the output is read as "who submitted
this". So:

  * faults() keeps whether the line names the accel driver;
  * attribute() refuses a non-accel fault by default -- "NOT THE ACCEL NODE: this fault is on
    another device, so the sampler's holders do not describe it (pass --any-device to attribute
    anyway)";
  * --any-device exists for the deliberate case, and is off.

Live, after: `faults: 137; attributable 0, ambiguous 0, nothing-held 0, no-sample 0,
other-device 137`.

The summary needed fixing too, and that is the part I would have shipped wrong: the new verdict was
not in the tally, so the same capture printed "0 of everything" -- a line that reads as an empty
capture rather than as 137 refused faults. Every verdict is now counted, plus a `!! N uncounted`
marker if the parts ever fail to sum to the whole, so a future verdict cannot be added without
appearing in the summary.

Pinned: a non-accel fault refuses; --any-device proceeds; the tally counts the refusal; the stamp
fixtures name the accel driver so the device check does not shadow the stamp question.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Third commit 0193a29e5: ran this tool on the first live stream it has ever seen and it produced a category error.

137 IO_PAGE_FAULTs on this boot, and the tool attributed them to pid 2667445 gpu-hold cl-npuperf-amdcol-i18x-q3-8. Every one was nvme 0000:c4:00.0 domain=0x0015 — the sampler's holders describe /dev/accel, and that ticket had nothing to do with the NVMe device. A category error is worse than a weak answer here, because the output is read as "who submitted this".

So faults() keeps whether the line names the accel driver, and attribute() refuses the rest by default:

NOT THE ACCEL NODE: this fault is on another device, so the sampler's holders do not describe it
                     (pass --any-device to attribute anyway)

Live, after: faults: 137; attributable 0, ambiguous 0, nothing-held 0, no-sample 0, other-device 137.

And the part I would have shipped wrong: the new verdict was not in the tally, so the same capture printed 0 of everything — a line that reads as an empty capture rather than as 137 refused faults. Every verdict is now counted, plus a !! N uncounted marker if the parts ever fail to sum to the whole, so a future verdict cannot be added without appearing in the summary.

Pinned: non-accel refuses; --any-device proceeds; the tally counts the refusal; and the stamp fixtures now name the accel driver so the device check does not shadow the stamp question.

Two things worth recording for #2542

--self-test                                  -> self-test ok
tools/ci/tests_referenced_check.py --check   -> rc=0
ls-remote origin 1bit/attribution-padded-stamp -> 0193a29e5399

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 0193a29

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 04:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant