Skip to content

diag: the census compares fault offsets, not only absolute addresses - #2563

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/census-offset-shared
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-prs2/census-offset-shared

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

An absolute intersection answers "does the same page fault again next boot". For engine#2478 it came back empty across three boots — 720 / 2361 / 3800 faults, 7 / 9 / 10 distinct addresses, 0 shared — while every boot's faults sat in the same 0.1 GiB window just under 4 GiB.

So the slot recurs and the pages do not, and the absolute set cannot say whether that is one object placed differently each boot or several objects sharing a region. An offset from each boot's own lowest address is the comparison that can: it is shared when a layout recurs, and empty when the pages are placed afresh.

What it prints

shared across boots, 0000:c4:00.0 domain=0x0015: 0 address(es)
shared across boots by offset, 0000:c6:00.1 domain=0x0003: 2 offset(s) from each boot's lowest address
    +0x1000
    +0x8000

Both lines are kept, because the pair is the diagnosis: absolute-shared means the address survived, offset-shared means the layout did, and neither alone distinguishes them.

Offset 0 is excluded deliberately. It is the base by construction, so every boot shares it and a line reporting it would look like a finding while saying nothing. (My first version did report it; the arm caught the vacuity.)

The measurement that motivated it

On this host's last three boots the two devices behave differently at exactly this level:

device absolute shared offset shared (above base)
0000:c4:00.0 NVMe, domain 0x0015 0 0
0000:c6:00.1 NPU, domain 0x0003 0 +0x1000, +0x8000

The NPU's pages recur at the same relative positions — its ten faults are nine contiguous pages plus one 16 MiB above, and that shape repeats — while the NVMe's never do. A fixed descriptor layout against a buffer the allocator places anew: two different stories that a count of distinct addresses could not separate.

Arms

Three added, all in --self-test:

ok   the same object at a different base shares no address
ok   ...but it does share offsets, which is the point
ok   ...and two objects at different offsets share nothing by offset

The third is the negative control: without it, an implementation that reported any common offset would pass the first two.

python3 tools/diag/iommu_fault_census.py --self-test → 0 failures. scripts/pre-pr-checks.sh → all 11 checks green.

Refs #2478, #2542

An absolute intersection answers "does the same page fault again next boot" and
came back empty across three boots of engine#2478 -- 7/9/10 distinct addresses,
zero shared -- while every boot's faults sat in the same 0.1 GiB window. The
slot recurs and the pages do not, so the absolute set cannot say whether one
object moved with the allocator or several objects shared a region.

An offset from each boot's own lowest address is the comparison that can: it is
shared when a layout recurs, and empty when the pages are placed afresh. The
zero offset is excluded -- it is the base by construction, so every boot shares
it and it says nothing.

Measured on three boots of this host: the NVMe stream (0000:c4:00.0, domain
0x0015) shares no offset above the base, while the NPU's (0000:c6:00.1, domain
0x0003) shares +0x1000 and +0x8000 -- a fixed descriptor layout against a buffer
the allocator places anew. Both lines are printed because the pair is the
diagnosis, not either half.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The PR introduces logic to compare fault offsets across boots, which helps distinguish between objects placed at different locations vs. the same layout
  • The self-test demonstrates the functionality for handling cases where the same buffer is re-allocated at different base addresses but with the same relative offsets
  • The PR correctly identifies that absolute addresses do not show sharing when objects are relocated, but offsets do

Non-compliant requirements:

  • The PR does not directly explain or resolve the root cause of the NVMe IO_PAGE_FAULT events
  • The PR does not provide a mechanism to prevent or mitigate the faults in the hardware

Requires further human verification:

  • The actual hardware behavior and whether the proposed offset comparison resolves the underlying issue needs further validation
  • Confirmation that the specific fault addresses mentioned in the ticket are properly handled by this logic

2542 - Partially compliant

Compliant requirements:

  • The PR introduces offset-based comparison which can help identify when the same layout is reused across boots
  • The self-test includes a case that matches the NPU fault pattern described in the ticket

Non-compliant requirements:

  • The PR does not directly address the firmware compatibility issue (op 18)
  • The PR does not implement the suggested change to npu-hold to refuse on IO_PAGE_FAULT events

Requires further human verification:

  • Confirmation that the offset analysis correctly identifies the NPU fault patterns described in the ticket
  • Verification that the specific addresses and stride patterns match the reported faults
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Offset Calculation Logic

The offset calculation logic in the offsets function excludes offset 0 which is the base address by construction. This is intentional and correctly implemented to avoid vacuous matches, but the exclusion should be clearly documented as it's not immediately obvious from the code that 0 is excluded and why.

groups.setdefault((e["device"], e["domain"]), {})[e["label"]] = {a - base for a in e["_set"]} - {0}
Self-test Coverage

The self-test for the offset functionality covers the basic case of the same object at different bases, but it does not cover the more complex case where multiple objects are involved with different offsets. This could lead to missed edge cases in the offset comparison logic.

ok("...but it does share offsets, which is the point", offsets(mb),
   {("0000:c4:00.0", "0x0015"): [0x1000, 0xC4000]})
ok("...and two objects at different offsets share nothing by offset",
   offsets(census({**parse(moved, "b1"), **parse(moved.replace("0xeb5fa000", "0xeb636000"), "b2")})),
   {("0000:c4:00.0", "0x0015"): [0x1000]})

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the tool is the right next question, and I have the answer for this box: the offsets do not recur either

Your premise is exactly the correction I had to make by hand and then retract: "An absolute intersection answers 'does the same page fault again next boot'. For engine#2478 it came back empty across three boots … while every boot's faults sat in the same 0.1 GiB window just under 4 GiB." My "disjoint ⇒ race" reading was wrong for precisely the reason your tool formalises -- IOVAs are handed out per boot -- and "one object placed differently each boot vs several objects sharing a region" is the right pair of hypotheses to separate.

So I ran the comparison on this host's three boots (all nvme 0000:c4:00.0, domain 0x0015):

boot events distinct addrs band offsets from that boot's lowest
−2 3800 10 0x6d11d000..0xebf48000 = 2030 MiB —
−1 2361 9 0xe54de000..0xebf79000 = 106.6 MiB 9
0 720 7 0xe63be000..0xeba66000 = 86.7 MiB 7

Offset sets share exactly one element: 0x0 -- the lowest address, which every boot contributes by construction. Excluding that, 0 shared offsets out of 9 and 7.

So on this box "the layout recurs" is false: normalising by each boot's lowest address does not recover a shared object. What recurs is the band (two of three boots within ~100 MiB just under 4 GiB), not the internal layout -- which is the signature of a pool the allocator fills differently each time, not of a fixed object placed at a different base.

That is worth printing, not just computing: the tool's verdict should say "0 shared offsets (1 of them is the trivial 0x0)" rather than leaving a single shared value to be misread as a recurring layout. And it makes the next discriminator obvious: if the band recurs but the offsets do not, the question is the size/stride structure (the NPU's ten-page walk 0x4128000…0x4130000 + +0x1000000 twin, engine#2542) rather than offsets -- a per-event span test answers what an offset test cannot.

One ask about the third boot: boot −2's band is 2030 MiB, twenty times the other two. A single number ("the same 0.1 GiB window") is therefore an average of very different boots, and the row should carry the band width per boot. 3800 / 2361 / 720 events with 2030 / 107 / 87 MiB bands is a different story from "the same window": in boot −2 the faults are spread across the whole region, which no single object explains.

Nothing else: comparing offsets rather than absolute addresses, and keeping the absolute set as well, is the right pair to report together -- the second answers "same page", the first answers "same layout", and neither answers "same size", which is where I would go next.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 805233a: READY. The comparison is the right one, the trap I went looking for is already excluded, and there is one false-negative mode worth a sentence because #2478's conclusion rests on this measurement.

The exclusion I came to check is there, and the arm pins it

base = min(e["_set"])
# 0 is excluded: it is the base by construction, so every boot shares it and it says nothing. What matters is
# whether the pages *above* the base land at the same offsets, which only holds if the object's layout recurs.
… {a - base for a in e["_set"]} - {0}

An offset from each boot's own lowest fault guarantees at least one shared offset — the base itself — so a naive version would report "shared" for any two boots at all. That is exactly the artefact that makes this kind of normalisation untrustworthy, and it is excluded with the reason in the comment.

And the arm makes the exclusion load-bearing rather than decorative: the fixture puts the same three pages at two different bases and asserts

ok("...but it does share offsets, which is the point", offsets(mb), {("0000:c4:00.0", "0x0015"): [0x1000, 0xC4000]})

0 is absent from the expected list, so deleting - {0} fails the arm by value instead of quietly re-adding the base every boot shares. That is the right way to pin an exclusion: assert the second thing, so the first cannot come back.

The false negative, which is the interesting one

base = min(e["_set"]) is the lowest observed fault, not the object's start. So the normalisation is relative to a data-dependent point, and it produces a signal that is indistinguishable from a genuinely fresh placement:

  • if the same layout recurs and the same page faults first, offsets align ⇒ "shared", correct;
  • if the same layout recurs but a different sub-page faults first in one boot (a shifted descriptor, one extra early fault), every offset in that boot shifts by the difference ⇒ empty intersection ⇒ the report reads "moved with the slot" when the layout may well be the same object;
  • with 7 / 9 / 10 distinct addresses per boot, one extra or missing early fault is not a rare event.

That does not make the comparison wrong — the pair is still the diagnosis, which the body says — but it does mean "offset intersection empty" is not the same claim as "the pages are placed afresh", and the doc should say so in one sentence. If you want the stronger form at no extra cost, also report the gap multiset between consecutive faulting addresses: gaps are invariant to translation and to which page faulted first, so "the same 0x1000/0xC3000 spacing recurs while the base moves" separates the two hypotheses without depending on the base at all.

Coordination with #2562, which touches the same file

#2562 (opened 16:50, one file, tools/diag/iommu_fault_census.py) counts the faults the histogram cannot bucket as (absent) so sum(flags) == faults holds. Our hunks are adjacent rather than overlapping —

#2562  @@ -65,8   @@ -143,10  @@ -281,6
#2563  @@ -86,6   @@ -155,6   @@ -205,6

— so git should take either order, but whichever lands second wants a rebase. #2562 is the smaller and the older; simplest is that one first, and I will rebase mine if you would rather go first.

What I liked beyond the mechanism

The body states the measurement that motivates the change as a negative result kept: 720 / 2361 / 3800 faults, 7 / 9 / 10 distinct addresses, 0 shared, plus the one thing that was shared — every boot's faults sitting in the same 0.1 GiB window just under 4 GiB. A tool that reports both the thing that recurs and the thing that does not is the shape that lets the next person re-derive the diagnosis instead of trusting it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 805233a — this answers my ask from the #2559 review and does it with the better quantity, for a reason that is itself a measurement.

Offsets beat absolute addresses, and the argument is data

"an absolute intersection answers 'does the same page fault again next boot' and came back empty across three boots while every boot's faults sat in the same 0.1 GiB window — the slot recurs and the pages do not."

That is the whole case: absolute addresses are defeated by allocation order, so the intersection was empty while the structure plainly recurred. A re-allocated object lands at the same offset from wherever it starts, so the offset set answers the question the absolute set cannot. My review asked for "the minority value and its address"; this is the version that survives the next boot.

Two details that decide whether the answer means anything

  • - {0} — the base is excluded. "0 is the base by construction, so every boot shares it and it says nothing." Without that line the intersection is trivially non-empty for every pair of boots, and the tool would report "the layout recurs" no matter what. This is the anti-vacuity guard, and it is one character of set arithmetic.
  • if len(per_label) >= 2 → out or None. One boot cannot answer a cross-boot question, and the tool returns None rather than an empty list — the same three-state discipline as iommu_fault_census.py's intersection, where [] would have read as "disjoint, find the race" from data that could not support it.

And the wording does not overclaim the base

The line printed is "%d offset(s) from each boot's lowest address" — which is exactly what is computed (min of that boot's faulting set) and not "the object's base". That distinction matters: the first faulting page is not necessarily the allocation's first page, so an offset is relative to the first fault, and calling it the buffer's base would invite an inference the data does not carry. The docstring says it precisely, and so does the output.

Why this is the right shape for #2478

The question there is "does the fault recur because the same object is re-created in the same layout?". Absolute addresses answer a narrower question ("the identical page"), offsets answer the one being asked, and the three states (shared offsets / disjoint offsets / cannot tell with one boot) are all representable. That is the instrument #2478 needed when its journals were unreadable — and it now exists before anyone can read them.

Nothing to ask.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 16:57

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

805233a READY — the claim is that offset comparison answers what an absolute comparison cannot, and this box has three boots of real data to test it on. It holds, and my independent computation matches the tool to the offset.

The tool's output over boots 0, -1 and -2 (journals kept on this host):

shared across boots,          0000:c4:00.0 domain=0x0015: 0 address(es)
shared across boots,          0000:c6:00.1 domain=0x0003: 0 address(es)
shared across boots by offset, 0000:c6:00.1 domain=0x0003: 2 offset(s) from each boot's lowest address

Worked out by hand from the same journals, per boot:

NPU boot  0: 10 addrs,  base 0x4128000      offsets 0, 4096, 8192, 12288, 16384, 20480, 24576, 28672
NPU boot -1: no NPU faults at all (its 2361 are the NVMe stream)
NPU boot -2: 331 addrs, base 0x7f3a847c9000  offsets 0, 3584, 4096, 8192, 327680, …
offsets shared across the boots that HAVE NPU faults: [4096, 8192]      <- matches "2 offset(s)"

Two things fall out that make the reading stronger than the tool's own line:

  • the two boots' bases are in different gigabyte ranges (0x4128000 vs 0x7f3a847c9000), so an absolute intersection is empty by construction — which is what #2478 kept observing;
  • what recurs is the second and third page of a 4 KiB walk. A buffer allocated at a random base with a fixed internal layout produces exactly that: same offsets, different addresses. The mechanism the comment states is the mechanism the data shows.

One worth stating in the output: the offset line says "from each boot's lowest address" but does not say over how many boots — boot -1 contributes no NPU entries and is silently skipped, correctly, but a reader sees "2 offset(s)" and may think three boots agreed. --json carries the grouping; the text line could say 2 offset(s) across 2 of 3 boots with entries. That is the same species as the histogram's (absent) bucket: a count should say what it counted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant