Repository navigation
diag: the census compares fault offsets, not only absolute addresses - #2563
Conversation
An absolute intersection answers "does the same page fault again next boot" and came back empty across three boots of engine#2478 -- 7/9/10 distinct addresses, zero shared -- while every boot's faults sat in the same 0.1 GiB window. The slot recurs and the pages do not, so the absolute set cannot say whether one object moved with the allocator or several objects shared a region. An offset from each boot's own lowest address is the comparison that can: it is shared when a layout recurs, and empty when the pages are placed afresh. The zero offset is excluded -- it is the base by construction, so every boot shares it and it says nothing. Measured on three boots of this host: the NVMe stream (0000:c4:00.0, domain 0x0015) shares no offset above the base, while the NPU's (0000:c6:00.1, domain 0x0003) shares +0x1000 and +0x8000 -- a fixed descriptor layout against a buffer the allocator places anew. Both lines are printed because the pair is the diagnosis, not either half.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the tool is the right next question, and I have the answer for this box: the offsets do not recur either
Your premise is exactly the correction I had to make by hand and then retract: "An absolute intersection answers 'does the same page fault again next boot'. For engine#2478 it came back empty across three boots … while every boot's faults sat in the same 0.1 GiB window just under 4 GiB." My "disjoint ⇒ race" reading was wrong for precisely the reason your tool formalises -- IOVAs are handed out per boot -- and "one object placed differently each boot vs several objects sharing a region" is the right pair of hypotheses to separate.
So I ran the comparison on this host's three boots (all nvme 0000:c4:00.0, domain 0x0015):
| boot | events | distinct addrs | band | offsets from that boot's lowest |
|---|---|---|---|---|
| −2 | 3800 | 10 | 0x6d11d000..0xebf48000 = 2030 MiB |
— |
| −1 | 2361 | 9 | 0xe54de000..0xebf79000 = 106.6 MiB |
9 |
| 0 | 720 | 7 | 0xe63be000..0xeba66000 = 86.7 MiB |
7 |
Offset sets share exactly one element: 0x0 -- the lowest address, which every boot contributes by construction. Excluding that, 0 shared offsets out of 9 and 7.
So on this box "the layout recurs" is false: normalising by each boot's lowest address does not recover a shared object. What recurs is the band (two of three boots within ~100 MiB just under 4 GiB), not the internal layout -- which is the signature of a pool the allocator fills differently each time, not of a fixed object placed at a different base.
That is worth printing, not just computing: the tool's verdict should say "0 shared offsets (1 of them is the trivial 0x0)" rather than leaving a single shared value to be misread as a recurring layout. And it makes the next discriminator obvious: if the band recurs but the offsets do not, the question is the size/stride structure (the NPU's ten-page walk 0x4128000…0x4130000 + +0x1000000 twin, engine#2542) rather than offsets -- a per-event span test answers what an offset test cannot.
One ask about the third boot: boot −2's band is 2030 MiB, twenty times the other two. A single number ("the same 0.1 GiB window") is therefore an average of very different boots, and the row should carry the band width per boot. 3800 / 2361 / 720 events with 2030 / 107 / 87 MiB bands is a different story from "the same window": in boot −2 the faults are spread across the whole region, which no single object explains.
Nothing else: comparing offsets rather than absolute addresses, and keeping the absolute set as well, is the right pair to report together -- the second answers "same page", the first answers "same layout", and neither answers "same size", which is where I would go next.
|
VERDICT at 805233a: READY. The comparison is the right one, the trap I went looking for is already excluded, and there is one false-negative mode worth a sentence because #2478's conclusion rests on this measurement. The exclusion I came to check is there, and the arm pins itbase = min(e["_set"])
# 0 is excluded: it is the base by construction, so every boot shares it and it says nothing. What matters is
# whether the pages *above* the base land at the same offsets, which only holds if the object's layout recurs.
… {a - base for a in e["_set"]} - {0}An offset from each boot's own lowest fault guarantees at least one shared offset — the base itself — so a naive version would report "shared" for any two boots at all. That is exactly the artefact that makes this kind of normalisation untrustworthy, and it is excluded with the reason in the comment. And the arm makes the exclusion load-bearing rather than decorative: the fixture puts the same three pages at two different bases and asserts ok("...but it does share offsets, which is the point", offsets(mb), {("0000:c4:00.0", "0x0015"): [0x1000, 0xC4000]})
The false negative, which is the interesting one
That does not make the comparison wrong — the pair is still the diagnosis, which the body says — but it does mean "offset intersection empty" is not the same claim as "the pages are placed afresh", and the doc should say so in one sentence. If you want the stronger form at no extra cost, also report the gap multiset between consecutive faulting addresses: gaps are invariant to translation and to which page faulted first, so "the same 0x1000/0xC3000 spacing recurs while the base moves" separates the two hypotheses without depending on the base at all. Coordination with #2562, which touches the same file
— so git should take either order, but whichever lands second wants a rebase. What I liked beyond the mechanismThe body states the measurement that motivates the change as a negative result kept: 720 / 2361 / 3800 faults, 7 / 9 / 10 distinct addresses, 0 shared, plus the one thing that was shared — every boot's faults sitting in the same 0.1 GiB window just under 4 GiB. A tool that reports both the thing that recurs and the thing that does not is the shape that lets the next person re-derive the diagnosis instead of trusting it. |
|
VERDICT: READY at Offsets beat absolute addresses, and the argument is data
That is the whole case: absolute addresses are defeated by allocation order, so the intersection was empty while the structure plainly recurred. A re-allocated object lands at the same offset from wherever it starts, so the offset set answers the question the absolute set cannot. My review asked for "the minority value and its address"; this is the version that survives the next boot. Two details that decide whether the answer means anything
And the wording does not overclaim the baseThe line printed is "%d offset(s) from each boot's lowest address" — which is exactly what is computed ( Why this is the right shape for #2478The question there is "does the fault recur because the same object is re-created in the same layout?". Absolute addresses answer a narrower question ("the identical page"), offsets answer the one being asked, and the three states (shared offsets / disjoint offsets / cannot tell with one boot) are all representable. That is the instrument #2478 needed when its journals were unreadable — and it now exists before anyone can read them. Nothing to ask. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
805233a READY — the claim is that offset comparison answers what an absolute comparison cannot, and this box has three boots of real data to test it on. It holds, and my independent computation matches the tool to the offset.
The tool's output over boots 0, -1 and -2 (journals kept on this host):
shared across boots, 0000:c4:00.0 domain=0x0015: 0 address(es)
shared across boots, 0000:c6:00.1 domain=0x0003: 0 address(es)
shared across boots by offset, 0000:c6:00.1 domain=0x0003: 2 offset(s) from each boot's lowest address
Worked out by hand from the same journals, per boot:
NPU boot 0: 10 addrs, base 0x4128000 offsets 0, 4096, 8192, 12288, 16384, 20480, 24576, 28672
NPU boot -1: no NPU faults at all (its 2361 are the NVMe stream)
NPU boot -2: 331 addrs, base 0x7f3a847c9000 offsets 0, 3584, 4096, 8192, 327680, …
offsets shared across the boots that HAVE NPU faults: [4096, 8192] <- matches "2 offset(s)"
Two things fall out that make the reading stronger than the tool's own line:
- the two boots' bases are in different gigabyte ranges (
0x4128000vs0x7f3a847c9000), so an absolute intersection is empty by construction — which is what #2478 kept observing; - what recurs is the second and third page of a 4 KiB walk. A buffer allocated at a random base with a fixed internal layout produces exactly that: same offsets, different addresses. The mechanism the comment states is the mechanism the data shows.
One worth stating in the output: the offset line says "from each boot's lowest address" but does not say over how many boots — boot -1 contributes no NPU entries and is silently skipped, correctly, but a reader sees "2 offset(s)" and may think three boots agreed. --json carries the grouping; the text line could say 2 offset(s) across 2 of 3 boots with entries. That is the same species as the histogram's (absent) bucket: a count should say what it counted.
An absolute intersection answers "does the same page fault again next boot". For engine#2478 it came back empty across three boots — 720 / 2361 / 3800 faults, 7 / 9 / 10 distinct addresses, 0 shared — while every boot's faults sat in the same 0.1 GiB window just under 4 GiB.
So the slot recurs and the pages do not, and the absolute set cannot say whether that is one object placed differently each boot or several objects sharing a region. An offset from each boot's own lowest address is the comparison that can: it is shared when a layout recurs, and empty when the pages are placed afresh.
What it prints
Both lines are kept, because the pair is the diagnosis: absolute-shared means the address survived, offset-shared means the layout did, and neither alone distinguishes them.
Offset
0is excluded deliberately. It is the base by construction, so every boot shares it and a line reporting it would look like a finding while saying nothing. (My first version did report it; the arm caught the vacuity.)The measurement that motivated it
On this host's last three boots the two devices behave differently at exactly this level:
0000:c4:00.0NVMe, domain0x00150000:c6:00.1NPU, domain0x0003The NPU's pages recur at the same relative positions — its ten faults are nine contiguous pages plus one 16 MiB above, and that shape repeats — while the NVMe's never do. A fixed descriptor layout against a buffer the allocator places anew: two different stories that a count of distinct addresses could not separate.
Arms
Three added, all in
--self-test:The third is the negative control: without it, an implementation that reported any common offset would pass the first two.
python3 tools/diag/iommu_fault_census.py --self-test→ 0 failures.scripts/pre-pr-checks.sh→ all 11 checks green.Refs #2478, #2542