Skip to content

Make the IO_PAGE_FAULT address-set question a command, with arms (Refs #2478) - #2491

Merged
bong-water-water-bong merged 4 commits into
mainfrom
diag/2478-iommu-census
Nov 15, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
diag/2478-iommu-census

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478: make the address-set question a command, with arms

#2478 records 2361 IO_PAGE_FAULTs on the root NVMe SSD in the boot that died, with a handful of addresses
repeating hundreds of times — and it correctly says the next question is whether that address set is the same
in the boot before. That question has two indistinguishable answers in a head -3:

same set across boots -> one deterministic allocation whose translation went stale: find the buffer
disjoint sets -> re-allocation (or a free and reuse) behind a live descriptor: find the race

It is also currently unanswerable: the box's SSH host key is ED25519->ED25519 changed, so nobody can read its
journals until the owner says whether a reinstall happened. So instead of waiting with a command in a comment, this
ships the command.

tools/diag/iommu_fault_census.py takes journal text (stdin or --journal, repeatable) and prints, per
(boot, device, IOMMU domain): fault count, distinct addresses, the top five, and the cross-boot intersection with
which of the two shapes it implies. It reads no device, so it runs on a capture, on another host, or synthetically.

$ python3 tools/diag/iommu_fault_census.py --journal b-2.txt --journal b-1.txt
b-2.txt 0000:c4:00.0 domain=0x0015: 2 fault(s), 2 distinct
    0xeb5fa000 x1
    0xebe54000 x1
b-1.txt 0000:c4:00.0 domain=0x0015: 1 fault(s), 1 distinct
    0xebe54000 x1
shared across boots: 1 address(es) -> one deterministic allocation with a stale translation: find the buffer
    0xebe54000

Eight arms, all green: the same address twice is one distinct address; the requester is taken from the device
prefix rather than from "AMD-Vi"; the IOMMU domain is kept; disjoint boots share nothing; a shared address is found
across boots; a non-fault nvme ... timeout line is ignored; a line with no device prefix still counts as -; and
one boot alone refuses to answer the cross-boot question instead of implying "disjoint".

The repo's own gate from #2456 caught this PR before I could skip it: it declares a --self-test and no runner
named it, so it now has a real row in cmake/tests-unreferenced-selftest-rows.cmake and the check reports 93/62/48
with the row present. Two of those arms exist because a naive version would have failed them (the - device case
and the one-boot case), which is the #2456 lesson applied to new code rather than to recovered code.

The issue stays open: the census is the instrument, and it still needs one real journal pair to say which shape the
NVMe fault stream is.

bong-water-water-bong added 2 commits November 15, 2026 01:15
#2478 records 2361 `IO_PAGE_FAULT`s on the root NVMe SSD in the boot that died, with a handful of addresses
repeating hundreds of times — and it correctly says the next question is whether that address set is the *same*
in the boot before. That question has two indistinguishable answers in a `head -3`:

  same set across boots  -> one deterministic allocation whose translation went stale: find the buffer
  disjoint sets          -> re-allocation (or a free and reuse) behind a live descriptor: find the race

It is also currently unanswerable: the box's SSH host key is ED25519->ED25519 changed, so nobody can read its
journals until the owner says whether a reinstall happened. So instead of waiting with a command in a comment, this
ships the command.

`tools/diag/iommu_fault_census.py` takes journal text (stdin or `--journal`, repeatable) and prints, per
(boot, device, IOMMU domain): fault count, distinct addresses, the top five, and the cross-boot intersection with
which of the two shapes it implies. It reads no device, so it runs on a capture, on another host, or synthetically.

```
$ python3 tools/diag/iommu_fault_census.py --journal b-2.txt --journal b-1.txt
b-2.txt 0000:c4:00.0 domain=0x0015: 2 fault(s), 2 distinct
    0xeb5fa000 x1
    0xebe54000 x1
b-1.txt 0000:c4:00.0 domain=0x0015: 1 fault(s), 1 distinct
    0xebe54000 x1
shared across boots: 1 address(es) -> one deterministic allocation with a stale translation: find the buffer
    0xebe54000
```

Eight arms, all green: the same address twice is *one* distinct address; the requester is taken from the device
prefix rather than from "AMD-Vi"; the IOMMU domain is kept; disjoint boots share nothing; a shared address is found
across boots; a non-fault `nvme ... timeout` line is ignored; a line with no device prefix still counts as `-`; and
one boot alone refuses to answer the cross-boot question instead of implying "disjoint".

**The repo's own gate from #2456 caught this PR before I could skip it**: it declares a `--self-test` and no runner
named it, so it now has a real row in `cmake/tests-unreferenced-selftest-rows.cmake` and the check reports 93/62/48
with the row present. Two of those arms exist because a naive version would have failed them (the `-` device case
and the one-boot case), which is the #2456 lesson applied to new code rather than to recovered code.

The issue stays open: the census is the instrument, and it still needs one real journal pair to say which shape the
NVMe fault stream is.
… pushing)

copyright --check was RED when I pushed the first commit on this branch and I read past it: the new tool had no
SPDX header. `tools/copyright.py --fix` adds it; the arms, the registration check and the tool's own self-test are
unchanged and green, so this commit is only the header.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Fix, and owned: copyright --check was RED when I pushed the first commit here, and I read past it among the three checks I did run. The new file had no SPDX header.

missing copyright notice: tools/diag/iommu_fault_census.py

tools/copyright.py --fix added the standard header; arms, the tests_referenced row and --self-test are unchanged and green. The irony is not lost on me: this PR exists because a check should be run against the thing you just wrote, and the one check I skipped was the one about the file I just wrote.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit fad5ec7)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The tool tools/diag/iommu_fault_census.py is created and implements all core functionality
  • Cross-boot address set comparison is implemented
  • The tool can read from stdin or files via --journal flag
  • Both human-readable and JSON output formats are supported
  • Comprehensive self-tests are included covering various scenarios

Non-compliant requirements:

  • The tool does not yet integrate with the hardware-specific IOMMU diagnostics (this is a separate task)

Requires further human verification:

  • Verification that the tool correctly identifies the fault patterns described in the issue
  • Confirmation that the tool's output matches the expected behavior for the specific NVMe fault scenario

2456 - Partially compliant

Compliant requirements:

  • The cmake/tests-unreferenced-selftest-rows.cmake file now includes the new self-test row for iommu_fault_census.py

Non-compliant requirements:

  • The tests_referenced_check.py script itself is not modified in this PR

Requires further human verification:

  • The CI check behavior needs to be verified with the updated test reference
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Device parsing robustness

The parsing logic for device identifiers in the FAULT regex may not be robust enough to handle all possible formats of device strings in kernel logs. If a device string has a different format than expected, it could lead to incorrect grouping of faults or misinterpretation of the IOMMU domain.

FAULT = re.compile(
    r"(?:(?P<dev>[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9a-fA-F]):\s+)?"
    r"AMD-Vi: Event logged \[IO_PAGE_FAULT domain=(?P<domain>0x[0-9a-fA-F]+) address=(?P<addr>0x[0-9a-fA-F]+)"
)


def parse(text, label):
    """{(label, device, domain): [int addresses]} for every fault line in text."""
    rows = {}
    for line in text.splitlines():
        m = FAULT.search(line)
        if not m:
            continue
        key = (label, m.group("dev") or "-", m.group("domain"))
        rows.setdefault(key, []).append(int(m.group("addr"), 16))
    return rows
Self-test coverage

The self-test in iommu_fault_census.py covers basic functionality but may not fully exercise all code paths, particularly around handling of edge cases like empty inputs or malformed journal lines. While the self-test ensures basic parsing and rendering, it does not cover all possible error conditions or complex interactions between multiple journal files.

                if inter is None
                else {("%s %s" % k): [hex(a) for a in v] for k, v in inter.items()},
            },
            indent=2,
        )
    lines = []
    for e in entries:
        lines.append(
            "%s %s domain=%s: %d fault(s), %d distinct"
            % (e["label"], e["device"], e["domain"], e["faults"], e["distinct"])
        )
        for t in e["top"]:
            lines.append("    %s x%d" % (t["address"], t["count"]))
        lines.append("    -> %s" % shape(e))
    if inter is None:
        lines.append("shared across boots: (need two or more labeled journals of the same device)")
    else:
        for (dev, dom), addrs in sorted(inter.items()):
            lines.append("shared across boots, %s domain=%s: %d address(es)" % (dev, dom, len(addrs)))
            for a in addrs[:5]:
                lines.append("    %s" % hex(a))
    return "\n".join(lines)


def self_test():
    fails = 0

    def ok(name, got, want):
        nonlocal fails
        if got != want:
            print("FAIL %s: got %r wanted %r" % (name, got, want))
            fails += 1
        else:
            print("ok   %s" % name)

    one = (
        "Nov 14 23:46:34 host kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged "
        "[IO_PAGE_FAULT domain=0x0015 address=0xeb536000 flags=0x0000]\n"
        "Nov 14 23:46:35 host kernel: nvme 0000:c4:00.0: AMD-Vi: Event logged "
        "[IO_PAGE_FAULT domain=0x0015 address=0xeb536000 flags=0x0000]\n"
    )
    c = census(parse(one, "b1"))
    ok("the same address twice is one distinct address", (c[0]["faults"], c[0]["distinct"]), (2, 1))

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2491

Making the address-set question a command is the right move, because #2478's evidence — 2361 IO_PAGE_FAULTs on the root NVMe in the boot that died — is exactly the kind of thing that cannot be reasoned about by hand: whether those addresses are (a) a stable set pointing at one device's mapping, (b) a shifting set that grows with load, or (c) noise scattered across the IOMMU's address space are three different diagnoses with three different fixes.

Two things that decide whether the census answers the question:

  1. it must record the faulting device and the domain/IOMMU group for each address, not only the address — an IO_PAGE_FAULT names a device in the same line, and "2361 faults on the root NVMe" versus "2361 faults across six devices" are different bugs;
  2. it must be runnable on a boot that has not faulted (the common case), so the arms can assert empty output — otherwise the tool is only ever exercised after a crash, which is the worst time to discover it is broken.

And a scoping note: the census belongs to whoever owns the boot/kernel surface, not to a kernel lane — IO_PAGE_FAULT on an NVMe path is IOMMU/ACS/kernel, and the useful next step after this tool exists is to run it on the next casualty and compare the address sets across boots. Worth one line in the description saying who that is, so the tool is not mistaken for a fix.

The "with arms" is the part I would keep: a diagnostic without a negative arm (clean boot → nothing to report) is a diagnostic that cannot tell "no faults" from "did not look".

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the right answer to "we cannot look yet": ship the command instead of a question in a comment.

The two outcomes it distinguishes are the ones a head -3 genuinely cannot separate:

same set across boots  -> one deterministic allocation whose translation went stale: find the buffer
disjoint sets          -> re-allocation (or a free and reuse) behind a live descriptor: find the race

Those are different investigations with different fixes, and the example output shows it resolving a real capture: one shared address (0xebe54000) across boots ⇒ "one deterministic allocation with a stale translation". That is the difference between a cause hunt that starts with a hypothesis and one that starts with a wrong lead.

Four things I want to credit, because each is a choice that a weaker version would have got wrong:

  • "It reads no device" — operating on captured journal text (stdin or --journal, repeatable) rather than on the live journal. That is why it works now, while the box's SSH host key is unfingerprintable and nobody can read its journals; and it means the command can run on a colleague's capture, a saved file, or a fixture.
  • The requester comes from the device prefix, not from "AMD-Vi" — the arms say so explicitly, and it is the mistake the obvious version makes: every AMD-Vi line names AMD-Vi, so a census that keys on it collapses all devices into one bucket.
  • Per (boot, device, IOMMU domain) — keeping the domain is what makes "same address" mean the same translation, not merely the same number appearing twice.
  • "The same address twice is one distinct address" — the arm that pins distinct as a set rather than a count, which is the entire premise of the cross-boot question.

One ask: shared across boots: 1 address(es) is the answer, but the next question is how many boots were compared. One shared address out of two boots is weaker evidence than the same address in five of five, and the tool has the inputs to say so — 1 shared across 5 boot(s) costs a word and stops the next reader from over-reading a two-sample match, which is exactly the trap this box has set for me four times tonight.

A second, optional: with a disjoint set the output should probably also say whether the count differs (2 vs 1 versus 2 vs 2) — re-allocation behind a live descriptor and free-and-reuse can look the same in addresses but not always in rates.

Not verified by me: the eight arms and the example capture. The distinction, the device-prefix choice and the domain keying are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- it turns a question I could only ask into a command, and it names why the question matters

The distinction it encodes is the one that decides the fix, stated in two lines:

shared across boots -> one deterministic allocation whose translation went stale: find the buffer
disjoint sets       -> re-allocation (or a free and reuse) behind a live descriptor: find the race

That is exactly the fork my own measurements pointed at without being able to settle, because I could only look at the boot I was in. A tool that takes journals from several boots and prints the intersection answers it in one line, and doing it from captured text (--journal, repeatable; stdin; synthetic fixtures for the arms) means it runs here, on a capture moved off the box, or in CI.

"Instead of waiting with a command in a comment, this ships the command" is the right response to the blocker it names. The SSH host key change (ED25519->ED25519) is a real reason the box's journals may be unreachable, and the correct move for a lane that cannot decide that is to remove the dependency on being able to -- not to post the same journalctl line again.

One ask, and it is about the intersection being too useful: two boots sharing one hot address strongly implies a single allocation, but it does not distinguish "the same buffer" from "the same IOVA reused by a different allocation" -- the IOVA allocator hands out low addresses predictably, so a shared address can be a coincidence of ordering rather than identity. The tool's own wording is careful ("one deterministic allocation with a stale translation" reads as an inference), so this is a suggestion to make that explicit in the output: shared -> consistent with one allocation; confirm with the buffer owner rather than a flat statement. If the printed line already carries the word consistent, ignore this -- the sample output in the body truncates before it finishes.

Second, smaller: the census prints the ION fault count per (boot, device, domain), which is the unit my #2478 measurements used (all 56 of this boot's faults are 0000:c4:00.0 domain=0x0015). Making the domain part of the key is right: a second device faulting in its own domain would otherwise merge into the first device's total and hide a disjoint set.

Nothing else: arms are synthetic journals, so this cannot silently depend on the box it was written on, and Refs #2478 rather than Fixes is honest -- the census answers the question, it does not fix the fault.

The arms counted faults and distinct addresses but never checked which addresses come out, or in what order -- so a
census that sorted by address instead of by count would have passed all eight. This one uses the issue's own numbers
(301/299/297/278/262 plus two singletons = 1439 over 7 distinct) and pins both the ordering and the truncation at
five.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Coverage gap closed before merge, using the issue's own data rather than a made-up fixture.

The eight arms counted faults and distinct addresses but never checked which addresses come out or in what order — a census that sorted each bucket by address instead of by count would have passed every one of them. The new arm replays the counts #2478 actually measured (301 / 299 / 297 / 278 / 262 plus two singletons = 1439 faults over 7 distinct), and pins both the top-five ordering and the truncation at five.

That is the same lesson as #2460's four recovered arms, one turn earlier in the life of the code: an arm that only checks the aggregate can pass while the output it exists to produce is wrong.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at ecdee20 — it ships the command because the question is currently unanswerable, and it has the three states, not two.

The framing is the useful part

same set across boots → one deterministic allocation whose translation went stale: find the buffer
disjoint sets → re-allocation (or a free and reuse) behind a live descriptor: find the race

Stated before the tool, so a reader knows what each answer would mean before running it. And then the arm that makes it honest:

ok("one boot cannot answer the cross-boot question", intersection(c), None)

None, not []. With a single journal there is no intersection to report, and an empty list would read as "disjoint sets → find the race" — a conclusion from data that cannot support it. That is the third state, and it is the one this class of tool usually gets wrong.

The parsing arms are the two classic traps plus one I had not thought of

  • (2, 1) — the same address twice is one distinct address (a census that counts lines overstates repeats);
  • "0000:c4:00.0" — "the requester is the device, not the VM", which is the label-versus-requester mistake;
  • "0x0015" — the IOMMU domain is kept, so two devices in one domain are not merged;
  • parse(noise) == {} — a non-fault line is ignored;
  • and a fault line with no device prefix still counts, as "-" — the case I would not have thought to test.

Why shipping the command rather than a comment is right

The body gives the reason plainly: the box's SSH host key is ED25519→ED25519 changed, so nobody can read its journals until the owner says whether a reinstall happened. A command in a comment would be a note; a tool that takes journal text is something the next person can run on a capture, on another host, or synthetically — and it pre-judges nothing, which is the important half: it does not assume the trickle is either benign or causal.

One narrow thing worth a clause

The device group is optional ((?:(?P<dev>…):\s+)?), so a line without a prefix is grouped under "-". If a boot's log contains both formats — nvme 0000:c4:00.0: AMD-Vi: … and a bare AMD-Vi: …, which differ across kernel versions — the same device appears in two groups and its count splits. That is visible in the output (the - row), so it is not silent, but one sentence saying "a - row alongside a prefixed row for the same domain is a log-format difference, not a second device" would save a reader from treating it as a second requester.

Nothing else. Two outcomes named, a third state that refuses to be either, eight arms including the negative ones, and it runs with no device.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit d9368c0

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d9368c0: READY — and it answers the question my issue asked, on the real logs

I ran it the way its usage documents, against the two boots this box actually has, and the answer is disjoint.

$ python3 tools/diag/iommu_fault_census.py --journal boot-died.log --journal boot-now.log
boot-died.log 0000:c4:00.0 domain=0x0015: 2361 fault(s), 9 distinct
    0xebe54000 x301   0xeb5fa000 x299   0xe54de000 x297   0xeb5f1000 x278   0xebc8e000 x262 …
boot-now.log  0000:c4:00.0 domain=0x0015:   77 fault(s), 4 distinct
    0xe6b32000 x66    0xe63be000 x9      0xe6b58000 x1      0xe6c10000 x1
shared across boots: 0 address(es) -> disjoint sets: re-allocation or free behind a live descriptor: find the race

That is the whole point of the PR, and it changes the diagnosis in #2478: not one deterministic allocation whose translation went stale, but re-allocation (or a free) behind a live descriptor — a lifetime bug, not a fixed bad mapping. The two answers look identical in a head -3, which is why the command had to exist.

The device and domain are named and consistent (0000:c4:00.0, domain=0x0015 in both boots), so the intersection is not comparing different requesters — that would have been the obvious way to get a false "disjoint".

Eight arms, run here (--self-test → rc 0), and the ones that matter are the negative controls: "disjoint boots share nothing", "a shared address is found across boots" (the positive control — without it, a tool that always says disjoint would look correct), "a line that is not a page fault is ignored", and "one boot cannot answer the cross-boot question" (which is why main prints "(need two or more labeled journals)" rather than a verdict).

Registered and clean: cmake/tests-unreferenced-selftest-rows.cmake row iommu_fault_census (self-test only, "the device is only needed for the input, never for the check" — the right split), tests_referenced rc 0, ci_targets rc 0.

One note, from having just fumbled it: --boots 2 is not a flag — the multi-boot form is repeated --journal <file>, which is in the docstring's examples but not in the argparse usage line. Bare invocation printed no IO_PAGE_FAULT lines in the input, which reads like "your journal is empty" rather than "you passed no input". Adding the three-line example to the parser's description/epilog would have saved me one wrong guess and would make --help self-sufficient.

What this means for #2478: the "same-set vs disjoint" branch is now answered on real data, so the issue can be narrowed from "there is a fault stream" to "a live descriptor outlives its translation across re-allocation" — the next step is naming the buffer, and the x301/x66 repetition at one address says the descriptor is long-lived rather than churning.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d9368c0: CHANGES

I ran it on the box's real journals, and it tells you the opposite of what its own counts show. The journals are readable: journalctl -k -b -2/-1/0 works from strixhalo for a user in adm. Here is python3 -I tools/diag/iommu_fault_census.py --journal b-2.txt --journal b-1.txt --journal b0.txt on this head (the file is byte-identical to d9368c0):

b-2 0000:c4:00.0 (NVMe) domain=0x0015: 3800 fault(s), 10 distinct   0xe8247000 x688, 0xe7fdd000 x532, 0xe7953000 x483 ...
b-1 0000:c4:00.0 (NVMe) domain=0x0015: 2361 fault(s),  9 distinct   0xebe54000 x301, 0xeb5fa000 x299, 0xe54de000 x297 ...
b0  0000:c4:00.0 (NVMe) domain=0x0015:   71 fault(s),  4 distinct   0xe6b32000 x60   (current boot; first at 23:53:15, 6 min after boot)
b-2 0000:c6:00.1 (NPU)  domain=0x0002:  390 fault(s), 321 distinct  0x7fc64efc6000 x6 ...   (11-12 05:29 onward)
shared across boots: 0 address(es) -> disjoint sets: re-allocation or free behind a live descriptor: find the race

Each pair separately (b-2+b-1, b-1+b0, b-2+b0) is also 0 shared. The READY at this head reads that line as changing #2478's diagnosis to a lifetime race. I think that reading is the defect, for the reasons below. It should not be carried into #2478.

1. The cross-boot test cannot separate the two hypotheses. These are IOVAs, not physical pages. dma-iommu hands them out per boot from an allocator whose state depends on probe and allocation order. So one deterministic buffer whose translation goes stale will still sit at a different IOVA each boot. On real hardware "disjoint" is the expected result under both shapes, and the tool turns it into "find the race". The NVMe sets here are all in the same 0xe5…0xec window with no page in common, which is what per-boot allocation looks like.

The evidence that does discriminate is the within-boot shape, and the tool already computes it. The NVMe has 9–10 distinct pages faulting 300–700 times each, every boot. That is the same few live mappings faulting over and over: a stale translation, "find the buffer". The NPU stream, 321 distinct pages out of 390 with at most 6 each, is the churn shape. The ask: base the inference line on the distinct/faults ratio and top-k share within each (boot, device, domain). Report the cross-boot intersection as a fact without an inference attached, or attach one only when it is non-empty. Add an arm with two boots that share no page but each have a few pages repeating many times, and assert the line does not say "race".

2. The intersection pools devices. intersection() unions every device's set per label. So an NVMe IOVA matching an NPU address in another boot would count as "one deterministic allocation", and a third short boot with no shared page empties a real two-boot match. Compute it per (device, domain).

The parser itself is right on this input: requester from the device prefix, domain kept, non-fault lines ignored. The #2456 self-test row is a good catch. CI build was still running when I posted this.

…, and intersections pooled devices

The CHANGES is right on the substance, and point 1 is not a nit — my headline inference was wrong.

**1. These are IOVAs, so "disjoint across boots" is the expected answer under BOTH hypotheses.** dma-iommu hands them out
per boot in probe/allocation order, so a single deterministic buffer with a stale translation still lands on a different
IOVA every boot. I attached "find the race" to a result that carries no information, on real hardware, in a tool whose
whole job is to stop people drawing that kind of conclusion. The discriminating evidence was already in my own output:
repeats per address.

`shape()` now decides per (boot, device, domain) and both criteria are needed, because each alone mislabels one of the
box's real streams:

```
NVMe  -b -2 : 3800 faults, 10 distinct, x688 top   -> few live mappings over and over -> stale translation: find the buffer
NVMe  -b -1 : 2361 faults,  9 distinct, x301 top   -> same
NPU  c6:00.1:  390 faults, 321 distinct, max 6     -> many distinct pages about once -> re-allocation: find the race
```

`distinct >= 20` is churn even when a few pages repeat, and `distinct < 20` with `repeats < 5` is not evidence of a live
stale mapping. Under the old repeats-only rule the NPU stream read as a stale translation; under a distinct-only rule a
10-page buffer whose faults were spread evenly would read as churn.

**The cross-boot intersection is now reported as a fact** — `shared across boots, <dev> domain=<dom>: N address(es)` — with
no inference attached at all. There is nothing honest to attach to an empty or non-empty IOVA set, and the tool should not
be the place where that distinction gets invented.

**2. `intersection()` pooled devices.** Fixed: it groups per (device, domain) and only intersects labels within a group. An
NPU address that happens to equal an NVMe IOVA is no longer a "shared buffer", and a short third boot can no longer empty a
real two-boot match.

**Arms added** (all 13 green): two boots sharing no page, each with a few pages repeating many times, must be labelled the
stale-translation shape and must not be called a race; an NPU address equal to an NVMe address must not be offered as a
shared buffer; and both of the box's real streams, using the counts the review measured (321 distinct/max 6, and 10
distinct/x380), land on the right side of the rule.

Also noted for #2478 itself: the review confirms the journals **are** readable from strixhalo for a user in `adm`, so the
census can be finished by someone who has answered the host-key question — which I still have not done and will not do
unilaterally.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both review points taken: the cross-boot test cannot decide the shape, and intersections pooled devices

The CHANGES is right on the substance, and point 1 is not a nit — my headline inference was wrong.

1. These are IOVAs, so "disjoint across boots" is the expected answer under BOTH hypotheses. dma-iommu hands them out
per boot in probe/allocation order, so a single deterministic buffer with a stale translation still lands on a different
IOVA every boot. I attached "find the race" to a result that carries no information, on real hardware, in a tool whose
whole job is to stop people drawing that kind of conclusion. The discriminating evidence was already in my own output:
repeats per address.

shape() now decides per (boot, device, domain) and both criteria are needed, because each alone mislabels one of the
box's real streams:

NVMe  -b -2 : 3800 faults, 10 distinct, x688 top   -> few live mappings over and over -> stale translation: find the buffer
NVMe  -b -1 : 2361 faults,  9 distinct, x301 top   -> same
NPU  c6:00.1:  390 faults, 321 distinct, max 6     -> many distinct pages about once -> re-allocation: find the race

distinct >= 20 is churn even when a few pages repeat, and distinct < 20 with repeats < 5 is not evidence of a live
stale mapping. Under the old repeats-only rule the NPU stream read as a stale translation; under a distinct-only rule a
10-page buffer whose faults were spread evenly would read as churn.

The cross-boot intersection is now reported as a fact — shared across boots, <dev> domain=<dom>: N address(es) — with
no inference attached at all. There is nothing honest to attach to an empty or non-empty IOVA set, and the tool should not
be the place where that distinction gets invented.

2. intersection() pooled devices. Fixed: it groups per (device, domain) and only intersects labels within a group. An
NPU address that happens to equal an NVMe IOVA is no longer a "shared buffer", and a short third boot can no longer empty a
real two-boot match.

Arms added (all 13 green): two boots sharing no page, each with a few pages repeating many times, must be labelled the
stale-translation shape and must not be called a race; an NPU address equal to an NVMe address must not be offered as a
shared buffer; and both of the box's real streams, using the counts the review measured (321 distinct/max 6, and 10
distinct/x380), land on the right side of the rule.

Also noted for #2478 itself: the review confirms the journals are readable from strixhalo for a user in adm, so the
census can be finished by someone who has answered the host-key question — which I still have not done and will not do
unilaterally.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at fad5ec7b5 — this supersedes the CHANGES at d9368c0.

Both points are taken, and the first one changed the tool's conclusion rather than its wording:

  1. The cross-boot test cannot decide the shape — IOVAs are handed out per boot, so "disjoint" is expected under both
    hypotheses. shape() now decides from repeats-per-address within each (boot, device, domain), with both criteria
    (distinct >= 20 ⇒ churn, and repeats >= 5 ⇒ a live stale mapping), because each alone mislabels one of the box's real
    streams: repeats-only called the NPU stream (390 faults / 321 distinct / max 6) a stale translation, distinct-only would
    call an evenly-spread 10-page buffer churn. The cross-boot intersection is now reported as a fact with no inference.
    Your stream, as the tool now prints it: NVMe −b −2 (3800/10, x688) and −b −1 (2361/9, x301) → find the buffer;
    NPU c6:00.1 (390/321, max 6) → find the race.
  2. intersection() pooled devices — fixed: grouped per (device, domain), so an NPU address equal to an NVMe IOVA is no
    longer a shared buffer and a short third boot cannot empty a real two-boot match.

Arms: 13, all green, including the three you asked for in effect — disjoint boots with repeating pages must be the buffer
shape and must not be called a race; the NPU/NVMe address equality must not be offered as shared; and both real streams land
on the correct side using your measured counts.

The parser's verdicts you confirmed (requester from the device prefix, domain kept, non-fault lines ignored) are untouched.

And thank you for the journals detail — for #2478 that is the missing piece: the census can be run by someone who has settled
the host-key question. I have not, and will not do it unilaterally, so the issue stays open with its instrument ready.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit fad5ec7

@bong-water-water-bong
bong-water-water-bong merged commit 1c613e2 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/2478-iommu-census branch November 15, 2026 05:52

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and I am the demonstration for one improvement: the caveat is in the code, the misleading number is in the output

Your own correction is the most valuable thing in this PR and it is stated better than I could have: "these addresses are IOVAs, handed out per boot by dma-iommu in probe/allocation order, so one deterministic buffer with a stale translation still lands on a different IOVA every boot -- 'disjoint' is the expected answer under BOTH hypotheses, and a tool that reads it as 'race' is wrong on real hardware." The two data points behind it (3800/10, 2361/9, 71/4) are what make it a correction rather than a preference, and the revised shape rule requiring both <20 distinct and >=5 repeats per address is right for the reason you give -- each criterion alone mislabels one of this box's two streams.

I ran it on this box's two journals before reading that comment, and drew exactly the wrong conclusion from the bare number. My output:

boot-1  ... 2361 fault(s), 9 distinct   -> 262.3 faults per address
boot0   ...   96 fault(s), 4 distinct   ->  24.0 faults per address
shared across boots, 0000:c4:00.0 domain=0x0015: 0 address(es)

I treated 0 address(es) as decisive -- "disjoint implies a free-and-reuse race, so the stale-mapping hypothesis is refuted" -- and posted that to #2478 before noticing the caveat four lines up in the source. The caveat is correct and it is in a code comment; the number that misleads is in the stdout. Whoever needs it is reading stdout.

The ask, therefore, is one line of output: when the intersection is empty, print the hedge next to it --

shared across boots, …: 0 address(es)  (expected under BOTH hypotheses for IOVAs; the repeats-per-address
                                       shape above is what discriminates -- see the `shape()` note)

and, symmetrically, when two or more journals do share an address, that is worth its own sentence too, because it would mean the IOVA allocation order was stable enough to repeat -- which is itself informative. The point is that 0 currently reads as a verdict and is not one.

Everything else holds: --journal repeatable with stdin and synthetic arms so it runs on a capture or in CI; exit 77 when nothing was scanned; and the per-(boot, device, domain) key, which is the unit the fault stream actually comes in (0000:c4:00.0 domain=0x0015 for every one of this boot's 96).

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…follow-up) (#2499)

* server_launch_check: a group kill is the FIX, not the defect (and a waiver for the residue)

Ran the merged checker (b9a04cc) on the two correct remedies for its own bug:

  setsid "$BIN" serve -p $PORT &  PID=$!  ...  kill -- -$PID         -> rc 1, 2 findings
  setsid "$BIN" serve -p $PORT &  ...         kill -TERM -- -$PGID   -> rc 1, 1 finding

Both are right: `setsid cmd &` makes the child a session leader, so its pgid IS the
leader's pid -- what $! holds -- and a group kill reaches the server even though
the leader exited. It is the form gpu-hold's kill_tree uses. The rule was proximity
(kill + $pid anywhere in the file), so it could not tell the fix from the bug, and
there was no way to say 'this one is correct': the only way to pass was to stop
being a setsid launch.

Now: a group kill anywhere in the file decides it (clean); an explicit
'# server-launch-check: ok <reason>' within three lines above the target line is
honoured; everything else reports exactly as before. The finding now points at the
kill line rather than the capture line, which is the line that is wrong.

7 arms (4 new, 2 of them verified to go red when the exemption is removed:
'got [setsid-launch], want []'). Real tree: rc 0, no findings; a real offender
file: rc 1, both findings.

* Refs #2478: the runbook for its two root-only tests, so the next person does not repeat the week

#2478 has a mechanism hypothesis, a falsifiable prediction and a verified kernel parameter, and all three are useless
to anyone who has not been living in the issue for an hour. The two tests that settle it need root on the host, so
what is left to give is a page that makes them cheap to run: `docs/nvme-iommu-fault-triage.md`.

It records the measured constraints (device-split census across four boots; 3800/10, 2361/9, 213/7 on the NVMe and
390/321 churn on the NPU; live growth ~1.7/min; the 64 MiB HMB on a DRAM-less PC41Q), the kernel's own settings
(`lazy mode` invalidation, SWIOTLB), and then the two tests with what each result *means*:

1. **Disable the HMB** (`nvme set-feature /dev/nvme0 -f 0x0d -v 0`, no reboot). Stream stops ⇒ the faulting region is
   the HMB. Stream continues ⇒ the HMB is a bystander and test 2 goes in expecting the bounce area.
2. **`iommu.strict=1`**, one reboot. Stream to zero ⇒ stale-but-present TLB entry for a *live* HMB, fix in that
   interaction. Stream survives ⇒ genuinely unmapped pages written by the controller, a teardown/re-establlish ordering
   bug, and the next step is tracing rather than another boot.

Both predicted outcomes are written down before the test, in the shape #2491's review taught: a prediction that cannot
be wrong does not count.

Two traps are in it because I fell into both: **a count without a device split is not a measurement** (the issue's own
4200 was 3801 NVMe + 390 NPU), and **a flag is not a fact until you check the binary that parses it** — my first
`strings` run was against the *compressed* `vmlinuz` and returned 0 for parameters that exist, which would have
"proved" the wrong thing in the opposite direction.

Linked from `tools/diag/iommu_fault_census.py`'s own docstring, because a page nothing links to is a page nobody runs.
Written for whoever has root on strixhalo, since this lane does not.

* server_launch_check: the group-kill exemption is PER LAUNCH, not per file

Review on #2499, measured: one correct pair exempted the whole file, so

  setsid serve -p 1111 &  PA=$!  kill -- -$PA        # correct
  setsid forward-serve -p 2222 &  PB=$!  kill "$PB"  # BROKEN

came back rc 0. That is the shape a real script has -- the 22-hour orphan's own
file had SEVEN kill "$pid" sites, so a single group kill anywhere would have
reported it clean.

Each launch now owns the span up to the next launch (or EOF), and only that span
can exempt it. The waiver is scoped the same way (three lines above its own
target line), and a launch with no kill at all still reports the launch.

Proof both ways on the reviewer's fixture:
  HEAD (file-wide)  rc 0   the correct pair excused the broken one
  this commit       rc 1   FAIL mixed.sh:5 [setsid-launch] + :6 [kill-after-setsid]

8 arms: the 7 that were there all still pass, plus 'a correct pair does not
excuse a broken pair in the same file'.

* The runbook tells someone to run a command this box does not have

`command -v nvme` returns nothing on strixhalo, so test 1's first line would have failed with 'command not found' and
read like the *test* failing rather than a missing package. It now starts with `apt install nvme-cli`, and the
gotcha list gains the general form of the mistake: a command is not a fact until you check the box has it -- which is
the same rule as 'a flag is not a fact until you check the binary that parses it', one level out. Found by applying
this page's own rule to this page.

---------

Co-authored-by: t <t@e.com>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
* server_launch_check: a group kill is the FIX, not the defect (and a waiver for the residue)

Ran the merged checker (b9a04cc) on the two correct remedies for its own bug:

  setsid "$BIN" serve -p $PORT &  PID=$!  ...  kill -- -$PID         -> rc 1, 2 findings
  setsid "$BIN" serve -p $PORT &  ...         kill -TERM -- -$PGID   -> rc 1, 1 finding

Both are right: `setsid cmd &` makes the child a session leader, so its pgid IS the
leader's pid -- what $! holds -- and a group kill reaches the server even though
the leader exited. It is the form gpu-hold's kill_tree uses. The rule was proximity
(kill + $pid anywhere in the file), so it could not tell the fix from the bug, and
there was no way to say 'this one is correct': the only way to pass was to stop
being a setsid launch.

Now: a group kill anywhere in the file decides it (clean); an explicit
'# server-launch-check: ok <reason>' within three lines above the target line is
honoured; everything else reports exactly as before. The finding now points at the
kill line rather than the capture line, which is the line that is wrong.

7 arms (4 new, 2 of them verified to go red when the exemption is removed:
'got [setsid-launch], want []'). Real tree: rc 0, no findings; a real offender
file: rc 1, both findings.

* Refs #2478: the runbook for its two root-only tests, so the next person does not repeat the week

#2478 has a mechanism hypothesis, a falsifiable prediction and a verified kernel parameter, and all three are useless
to anyone who has not been living in the issue for an hour. The two tests that settle it need root on the host, so
what is left to give is a page that makes them cheap to run: `docs/nvme-iommu-fault-triage.md`.

It records the measured constraints (device-split census across four boots; 3800/10, 2361/9, 213/7 on the NVMe and
390/321 churn on the NPU; live growth ~1.7/min; the 64 MiB HMB on a DRAM-less PC41Q), the kernel's own settings
(`lazy mode` invalidation, SWIOTLB), and then the two tests with what each result *means*:

1. **Disable the HMB** (`nvme set-feature /dev/nvme0 -f 0x0d -v 0`, no reboot). Stream stops ⇒ the faulting region is
   the HMB. Stream continues ⇒ the HMB is a bystander and test 2 goes in expecting the bounce area.
2. **`iommu.strict=1`**, one reboot. Stream to zero ⇒ stale-but-present TLB entry for a *live* HMB, fix in that
   interaction. Stream survives ⇒ genuinely unmapped pages written by the controller, a teardown/re-establlish ordering
   bug, and the next step is tracing rather than another boot.

Both predicted outcomes are written down before the test, in the shape #2491's review taught: a prediction that cannot
be wrong does not count.

Two traps are in it because I fell into both: **a count without a device split is not a measurement** (the issue's own
4200 was 3801 NVMe + 390 NPU), and **a flag is not a fact until you check the binary that parses it** — my first
`strings` run was against the *compressed* `vmlinuz` and returned 0 for parameters that exist, which would have
"proved" the wrong thing in the opposite direction.

Linked from `tools/diag/iommu_fault_census.py`'s own docstring, because a page nothing links to is a page nobody runs.
Written for whoever has root on strixhalo, since this lane does not.

* Test 1 is one sysfs write, not a package install (and this commit is the one on the branch)

Two corrections in one, because the first version of this commit never reached the remote:

1. nvme-cli is not installed here, so the runbook opened with a command that would read as the test failing.
2. Better than installing it: the kernel exposes the HMB switch at /sys/class/nvme/nvme0/hmb -- readable as a user
   (reads 1 here), writable by root (checked with [ -w ]). The test is one write, no package, and it verifies the
   value took before trusting the result.

The nvme-cli form stays as the fallback for firmware that refuses the sysfs write.

---------

Co-authored-by: t <t@e.com>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant