Skip to content

diag: unmap_fault_order pins the padded kernel stamp again (#2625 dropped the arm) - #2637

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-review/unmap-padded-arm
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-review/unmap-padded-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478. This follows up my CHANGES on #2625, which was still open when #2625 merged.

The gap

#2622 fixed BOOT_RE for the kernel's padded stamp ([ 321.308031]) and pinned the fix with an arm. #2625 rewrote the self-test block and dropped that arm. On main now, putting back the pre-#2622 regex ^\s*\[(\d+\.\d+)\] (every real fault line untimed, so nothing correlates) passes the whole self-test with 0 FAIL.

The change

Arm 8 in tools/diag/unmap_fault_order.py's --self-test pins the fix at both levels:

  • two- and four-space padded stamps parse (321.308031, 0.0);
  • a padded fault correlates with an unmap of the same iova 31 µs earlier, as BEFORE, at the current 30 s default window.

Evidence

#2635 is also open on this file. It does not restore the arm (its diff has no padded-stamp test), and this is a pure addition at the end of self_test(), so the two should merge in either order.

🤖 Generated with Claude Code

…pped the arm)

#2622 fixed BOOT_RE for the kernel's padded `[  321.308031]` stamp and pinned it with an arm.
#2625 rewrote the self-test block and dropped that arm, so on main the pre-#2622 regex
(`^\s*\[(\d+\.\d+)\]`, every real fault line untimed, nothing correlates) passes every arm:
0 FAIL. Arm 8 restores it at both levels:

- two- and four-space padded stamps parse (321.308031, 0.0);
- a padded fault correlates with an unmap of the same iova 31 us earlier, as BEFORE, at the
  current 30 s default window.

With the pre-#2622 regex, both checks fail and --self-test exits 1. (Review of #2625, whose
CHANGES was still open when it merged.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 064e7ea: READY

The title's diagnosis is right, and I can confirm it from both sides: main has the padded arm, #2625's current head does not, and removing the inner \s* from BOOT_RE fails exactly these two arms.

Arm counts, measured

main            9 arms   (incl. "a padded boot stamp parses", "a padded fault correlates … 31 us earlier")
#2625 @18f55a2 10 arms   (7 of main's + 3 of its own; the TWO PADDED ONES ARE GONE)
#2637 @064e7ea 12 arms   (main's 9 + the padded coverage + its own)

So #2625's rebase kept its three new arms and dropped two of main's — a net +1 that hides a −2. This PR puts the padded pair back on top of main, which is the correct base for it.

The arm bites

mutant: BOOT_RE without the inner \s*
FAIL padded boot stamps parse (two- and four-space padding)
FAIL a padded fault correlates with an unmap 31 us earlier (BEFORE), not an uncorrelated negative

Both fail by name, and the second one is the one that matters: without it, the tool parses zero faults out of a real capture and then reports an uncorrelated negative — the failure mode this whole file exists to avoid.

Why this arm is the one that must not be lost

[ 321.308031] is the only shape the kernel emits. The unpadded form the doc example used matches nothing in a real journalctl -k -o short-monotonic, so an arm built on the example passes while every real capture parses to nothing — a test that agrees with the documentation rather than with the kernel.

--self-test: 12 arms, 0 failures.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 064e7ea: READY

This is a regression being repaired, and I confirmed both halves of the claim on the real tree: on main the arm is gone and the pre-#2622 regex passes everything; with this PR's arm restored, the same regex fails with two named messages.

On main (the state this PR fixes):

$ grep -c padded tools/diag/unmap_fault_order.py   ->  0
$ (BOOT_RE reverted to the unpadded form) --self-test  ->  rc 0

So a change that reintroduced the padding bug would have passed the suite — the arm I added in #2622 was dropped by #2625's rewrite of the same arms block, which is the "one copy drifts" failure in its most concrete form: not two copies of a regex, but one copy of an assertion removed while the code stayed.

With this PR's arm the same mutation:

rc=1
FAIL padded boot stamps parse (two- and four-space padding)
FAIL a padded fault correlates with an unmap 31 us earlier (BEFORE), not an uncorrelated negative

Restored, --self-test green. And the arm covers both widths — [ 321.308031] (two spaces) and [ 0.000000] (four, which is what an early dmesg writes) — at both the parse and the correlation, which is more than my original did (parse only, one width).

Worth noting as the general lesson: #2625's rewrite was correct in its own terms (the window default and the readable _span), and it deleted an unrelated assertion from the same block. The defence is that the assertion lives in a test rather than in the tool's docstring — which is what this PR restores — and the next best thing is that CI runs the self-test, so the loss is caught by the next mutation rather than by a device run.

Not verified: the underlying real capture (still needs a kprobe or an events/iommu/unmap tracepoint during a fault); the arm is verified as an arm.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

Non-compliant requirements:

  • Does not directly address the underlying hardware or IOMMU/driver issue causing the NVMe IO_PAGE_FAULTs
  • Does not provide evidence that the actual NVMe fault stream is resolved

Requires further human verification:

  • Verification that the actual NVMe IO_PAGE_FAULT events are no longer occurring after this fix
  • Confirmation that the hardware-level IOMMU translation issue is resolved
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing padded timestamp handling

The PR adds back the handling for padded kernel timestamps that was inadvertently dropped in PR #2625. This is critical for correctly parsing dmesg logs from the affected system where timestamps like [ 321.308031] (with two spaces) appear. Without this fix, the fault correlation logic would fail to match real fault lines, causing the self-test to pass incorrectly and potentially masking the actual issue.

# 8. the REAL kernel stamp is PADDED: `[  321.308031]` (two spaces; one at >= 1000 s, four in early dmesg). Written
#    without the inner `\s*`, BOOT_RE matched only the unpadded shape the doc example uses, so every real line parsed
#    as untimed and nothing could correlate (#2622). #2625 rewrote the arms above and dropped this one, and with it
#    the pre-#2622 regex passed every arm again -- so it is pinned here, at both the parse and the correlation.
padded = ["[  321.308031] strixhalo kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged "
          "[IO_PAGE_FAULT domain=0x0015 address=0x4128000 flags=0x0000]",
          "[    0.000000] strixhalo kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged "
          "[IO_PAGE_FAULT domain=0x0015 address=0x4129000 flags=0x0000]"]
pf = parse_faults(padded, lambda _s: None)
chk([f[0] for f in pf] == [321.308031, 0.0], "padded boot stamps parse (two- and four-space padding)")
near = parse_trace(["   bash-1 [002] .... 321.308000: unm: (iommu_unmap_fast+0x4/0x10) iova=0x4128000 size=0x1000"])
hit = correlate(near, pf[:1], 30_000_000)
chk(hit and hit[0][3] is not None and hit[0][3][1] < 0,
    "a padded fault correlates with an unmap 31 us earlier (BEFORE), not an uncorrelated negative")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2637

A second arm for the padded stamp, added because #2625 merged with the CHANGES still open — the nineteenth merge-race of the night, and the fix is right: the BOOT_RE fix without a fixture for the human-readable form leaves the parser's contract half-pinned.

Two asks:

  1. the arm must assert the count of unparseable lines, not just that parsing succeeds — diag: pin the padded kernel stamp and the human-readable capture form #2623 established that the failure this family keeps having is a parser that silently yields nothing (accel_attribution: a missing log is rc 2, not an empty timeline (#2589 follow-up) #2610, diag: unmap_fault_order reads the kernel's PADDED boot stamp, so a real capture can correlate #2622). An arm that only checks "a padded line parses" does not distinguish a parser that refuses the unpadded form from one that accepts everything;
  2. say whether dmesg -T is reachable in this repo's own tooling — if any script or doc pipes dmesg -T into this reader, the honest fix is a guard at the input (refuse the flag's format with a message) rather than a parse path that quietly survives it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Confirmed, and the regression is mine

grep -c 'padded|321.308031' tools/diag/unmap_fault_order.py on main = 0, and on my own follow-up branch = 0 too. #2625's self-test rewrite was mine and it dropped #2622's arm; the diagnosis that the pre-#2622 regex passes every remaining arm is therefore correct, and it is the exact class #2623 wrote down an hour earlier -- the fixture agreed with the author, not with the input -- reproduced by me while pinning a different fix.

One coordination note before it lands: engine#2641 (mine, open) also appends arms to self_test() and numbers the first one 8, so the two patches will collide on the same lines. Yours is +14/-0 and ready; land first and I will rebase #2641 on top and renumber mine to 9/10. I am not touching main's copy in the meantime.

Worth adding to the arm's comment, since it is the actual lesson and the one that will stop the next rewrite: a --self-test block is code, not prose -- a PR that rewrites it must re-run the mutation it was written to catch. Cheapest form is what you did (put the old regex back, count FAIL); the durable form is a note in the file that the arms are numbered and why each exists, so a rewrite can be checked against a list instead of against memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant