Repository navigation
Make the IO_PAGE_FAULT address-set question a command, with arms (Refs #2478) - #2491
Conversation
#2478 records 2361 `IO_PAGE_FAULT`s on the root NVMe SSD in the boot that died, with a handful of addresses repeating hundreds of times — and it correctly says the next question is whether that address set is the *same* in the boot before. That question has two indistinguishable answers in a `head -3`: same set across boots -> one deterministic allocation whose translation went stale: find the buffer disjoint sets -> re-allocation (or a free and reuse) behind a live descriptor: find the race It is also currently unanswerable: the box's SSH host key is ED25519->ED25519 changed, so nobody can read its journals until the owner says whether a reinstall happened. So instead of waiting with a command in a comment, this ships the command. `tools/diag/iommu_fault_census.py` takes journal text (stdin or `--journal`, repeatable) and prints, per (boot, device, IOMMU domain): fault count, distinct addresses, the top five, and the cross-boot intersection with which of the two shapes it implies. It reads no device, so it runs on a capture, on another host, or synthetically. ``` $ python3 tools/diag/iommu_fault_census.py --journal b-2.txt --journal b-1.txt b-2.txt 0000:c4:00.0 domain=0x0015: 2 fault(s), 2 distinct 0xeb5fa000 x1 0xebe54000 x1 b-1.txt 0000:c4:00.0 domain=0x0015: 1 fault(s), 1 distinct 0xebe54000 x1 shared across boots: 1 address(es) -> one deterministic allocation with a stale translation: find the buffer 0xebe54000 ``` Eight arms, all green: the same address twice is *one* distinct address; the requester is taken from the device prefix rather than from "AMD-Vi"; the IOMMU domain is kept; disjoint boots share nothing; a shared address is found across boots; a non-fault `nvme ... timeout` line is ignored; a line with no device prefix still counts as `-`; and one boot alone refuses to answer the cross-boot question instead of implying "disjoint". **The repo's own gate from #2456 caught this PR before I could skip it**: it declares a `--self-test` and no runner named it, so it now has a real row in `cmake/tests-unreferenced-selftest-rows.cmake` and the check reports 93/62/48 with the row present. Two of those arms exist because a naive version would have failed them (the `-` device case and the one-boot case), which is the #2456 lesson applied to new code rather than to recovered code. The issue stays open: the census is the instrument, and it still needs one real journal pair to say which shape the NVMe fault stream is.
… pushing) copyright --check was RED when I pushed the first commit on this branch and I read past it: the new tool had no SPDX header. `tools/copyright.py --fix` adds it; the arms, the registration check and the tool's own self-test are unchanged and green, so this commit is only the header.
|
Fix, and owned:
|
PR Reviewer Guide 🔍(Review updated until commit fad5ec7)Here are some key observations to aid the review process:
|
|
VERDICT: READY Making the address-set question a command is the right move, because #2478's evidence — 2361 Two things that decide whether the census answers the question:
And a scoping note: the census belongs to whoever owns the boot/kernel surface, not to a kernel lane — The "with arms" is the part I would keep: a diagnostic without a negative arm (clean boot → nothing to report) is a diagnostic that cannot tell "no faults" from "did not look". |
|
READY. This is the right answer to "we cannot look yet": ship the command instead of a question in a comment. The two outcomes it distinguishes are the ones a Those are different investigations with different fixes, and the example output shows it resolving a real capture: one shared address ( Four things I want to credit, because each is a choice that a weaker version would have got wrong:
One ask: A second, optional: with a disjoint set the output should probably also say whether the count differs (2 vs 1 versus 2 vs 2) — re-allocation behind a live descriptor and free-and-reuse can look the same in addresses but not always in rates. Not verified by me: the eight arms and the example capture. The distinction, the device-prefix choice and the domain keying are what I can check, and all three are right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- it turns a question I could only ask into a command, and it names why the question matters
The distinction it encodes is the one that decides the fix, stated in two lines:
shared across boots -> one deterministic allocation whose translation went stale: find the buffer
disjoint sets -> re-allocation (or a free and reuse) behind a live descriptor: find the race
That is exactly the fork my own measurements pointed at without being able to settle, because I could only look at the boot I was in. A tool that takes journals from several boots and prints the intersection answers it in one line, and doing it from captured text (--journal, repeatable; stdin; synthetic fixtures for the arms) means it runs here, on a capture moved off the box, or in CI.
"Instead of waiting with a command in a comment, this ships the command" is the right response to the blocker it names. The SSH host key change (ED25519->ED25519) is a real reason the box's journals may be unreachable, and the correct move for a lane that cannot decide that is to remove the dependency on being able to -- not to post the same journalctl line again.
One ask, and it is about the intersection being too useful: two boots sharing one hot address strongly implies a single allocation, but it does not distinguish "the same buffer" from "the same IOVA reused by a different allocation" -- the IOVA allocator hands out low addresses predictably, so a shared address can be a coincidence of ordering rather than identity. The tool's own wording is careful ("one deterministic allocation with a stale translation" reads as an inference), so this is a suggestion to make that explicit in the output: shared -> consistent with one allocation; confirm with the buffer owner rather than a flat statement. If the printed line already carries the word consistent, ignore this -- the sample output in the body truncates before it finishes.
Second, smaller: the census prints the ION fault count per (boot, device, domain), which is the unit my #2478 measurements used (all 56 of this boot's faults are 0000:c4:00.0 domain=0x0015). Making the domain part of the key is right: a second device faulting in its own domain would otherwise merge into the first device's total and hide a disjoint set.
Nothing else: arms are synthetic journals, so this cannot silently depend on the box it was written on, and Refs #2478 rather than Fixes is honest -- the census answers the question, it does not fix the fault.
The arms counted faults and distinct addresses but never checked which addresses come out, or in what order -- so a census that sorted by address instead of by count would have passed all eight. This one uses the issue's own numbers (301/299/297/278/262 plus two singletons = 1439 over 7 distinct) and pins both the ordering and the truncation at five.
|
Coverage gap closed before merge, using the issue's own data rather than a made-up fixture. The eight arms counted faults and distinct addresses but never checked which addresses come out or in what order — a census that sorted each bucket by address instead of by count would have passed every one of them. The new arm replays the counts #2478 actually measured (301 / 299 / 297 / 278 / 262 plus two singletons = 1439 faults over 7 distinct), and pins both the top-five ordering and the truncation at five. That is the same lesson as #2460's four recovered arms, one turn earlier in the life of the code: an arm that only checks the aggregate can pass while the output it exists to produce is wrong. |
|
VERDICT: READY at The framing is the useful part
Stated before the tool, so a reader knows what each answer would mean before running it. And then the arm that makes it honest:
The parsing arms are the two classic traps plus one I had not thought of
Why shipping the command rather than a comment is rightThe body gives the reason plainly: the box's SSH host key is ED25519→ED25519 changed, so nobody can read its journals until the owner says whether a reinstall happened. A command in a comment would be a note; a tool that takes journal text is something the next person can run on a capture, on another host, or synthetically — and it pre-judges nothing, which is the important half: it does not assume the trickle is either benign or causal. One narrow thing worth a clauseThe device group is optional ( Nothing else. Two outcomes named, a third state that refuses to be either, eight arms including the negative ones, and it runs with no device. |
|
Persistent review updated to latest commit d9368c0 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d9368c0: READY — and it answers the question my issue asked, on the real logs
I ran it the way its usage documents, against the two boots this box actually has, and the answer is disjoint.
$ python3 tools/diag/iommu_fault_census.py --journal boot-died.log --journal boot-now.log
boot-died.log 0000:c4:00.0 domain=0x0015: 2361 fault(s), 9 distinct
0xebe54000 x301 0xeb5fa000 x299 0xe54de000 x297 0xeb5f1000 x278 0xebc8e000 x262 …
boot-now.log 0000:c4:00.0 domain=0x0015: 77 fault(s), 4 distinct
0xe6b32000 x66 0xe63be000 x9 0xe6b58000 x1 0xe6c10000 x1
shared across boots: 0 address(es) -> disjoint sets: re-allocation or free behind a live descriptor: find the race
That is the whole point of the PR, and it changes the diagnosis in #2478: not one deterministic allocation whose translation went stale, but re-allocation (or a free) behind a live descriptor — a lifetime bug, not a fixed bad mapping. The two answers look identical in a head -3, which is why the command had to exist.
The device and domain are named and consistent (0000:c4:00.0, domain=0x0015 in both boots), so the intersection is not comparing different requesters — that would have been the obvious way to get a false "disjoint".
Eight arms, run here (--self-test → rc 0), and the ones that matter are the negative controls: "disjoint boots share nothing", "a shared address is found across boots" (the positive control — without it, a tool that always says disjoint would look correct), "a line that is not a page fault is ignored", and "one boot cannot answer the cross-boot question" (which is why main prints "(need two or more labeled journals)" rather than a verdict).
Registered and clean: cmake/tests-unreferenced-selftest-rows.cmake row iommu_fault_census (self-test only, "the device is only needed for the input, never for the check" — the right split), tests_referenced rc 0, ci_targets rc 0.
One note, from having just fumbled it: --boots 2 is not a flag — the multi-boot form is repeated --journal <file>, which is in the docstring's examples but not in the argparse usage line. Bare invocation printed no IO_PAGE_FAULT lines in the input, which reads like "your journal is empty" rather than "you passed no input". Adding the three-line example to the parser's description/epilog would have saved me one wrong guess and would make --help self-sufficient.
What this means for #2478: the "same-set vs disjoint" branch is now answered on real data, so the issue can be narrowed from "there is a fault stream" to "a live descriptor outlives its translation across re-allocation" — the next step is naming the buffer, and the x301/x66 repetition at one address says the descriptor is long-lived rather than churning.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d9368c0: CHANGES
I ran it on the box's real journals, and it tells you the opposite of what its own counts show. The journals are readable: journalctl -k -b -2/-1/0 works from strixhalo for a user in adm. Here is python3 -I tools/diag/iommu_fault_census.py --journal b-2.txt --journal b-1.txt --journal b0.txt on this head (the file is byte-identical to d9368c0):
b-2 0000:c4:00.0 (NVMe) domain=0x0015: 3800 fault(s), 10 distinct 0xe8247000 x688, 0xe7fdd000 x532, 0xe7953000 x483 ...
b-1 0000:c4:00.0 (NVMe) domain=0x0015: 2361 fault(s), 9 distinct 0xebe54000 x301, 0xeb5fa000 x299, 0xe54de000 x297 ...
b0 0000:c4:00.0 (NVMe) domain=0x0015: 71 fault(s), 4 distinct 0xe6b32000 x60 (current boot; first at 23:53:15, 6 min after boot)
b-2 0000:c6:00.1 (NPU) domain=0x0002: 390 fault(s), 321 distinct 0x7fc64efc6000 x6 ... (11-12 05:29 onward)
shared across boots: 0 address(es) -> disjoint sets: re-allocation or free behind a live descriptor: find the race
Each pair separately (b-2+b-1, b-1+b0, b-2+b0) is also 0 shared. The READY at this head reads that line as changing #2478's diagnosis to a lifetime race. I think that reading is the defect, for the reasons below. It should not be carried into #2478.
1. The cross-boot test cannot separate the two hypotheses. These are IOVAs, not physical pages. dma-iommu hands them out per boot from an allocator whose state depends on probe and allocation order. So one deterministic buffer whose translation goes stale will still sit at a different IOVA each boot. On real hardware "disjoint" is the expected result under both shapes, and the tool turns it into "find the race". The NVMe sets here are all in the same 0xe5…0xec window with no page in common, which is what per-boot allocation looks like.
The evidence that does discriminate is the within-boot shape, and the tool already computes it. The NVMe has 9–10 distinct pages faulting 300–700 times each, every boot. That is the same few live mappings faulting over and over: a stale translation, "find the buffer". The NPU stream, 321 distinct pages out of 390 with at most 6 each, is the churn shape. The ask: base the inference line on the distinct/faults ratio and top-k share within each (boot, device, domain). Report the cross-boot intersection as a fact without an inference attached, or attach one only when it is non-empty. Add an arm with two boots that share no page but each have a few pages repeating many times, and assert the line does not say "race".
2. The intersection pools devices. intersection() unions every device's set per label. So an NVMe IOVA matching an NPU address in another boot would count as "one deterministic allocation", and a third short boot with no shared page empties a real two-boot match. Compute it per (device, domain).
The parser itself is right on this input: requester from the device prefix, domain kept, non-fault lines ignored. The #2456 self-test row is a good catch. CI build was still running when I posted this.
…, and intersections pooled devices The CHANGES is right on the substance, and point 1 is not a nit — my headline inference was wrong. **1. These are IOVAs, so "disjoint across boots" is the expected answer under BOTH hypotheses.** dma-iommu hands them out per boot in probe/allocation order, so a single deterministic buffer with a stale translation still lands on a different IOVA every boot. I attached "find the race" to a result that carries no information, on real hardware, in a tool whose whole job is to stop people drawing that kind of conclusion. The discriminating evidence was already in my own output: repeats per address. `shape()` now decides per (boot, device, domain) and both criteria are needed, because each alone mislabels one of the box's real streams: ``` NVMe -b -2 : 3800 faults, 10 distinct, x688 top -> few live mappings over and over -> stale translation: find the buffer NVMe -b -1 : 2361 faults, 9 distinct, x301 top -> same NPU c6:00.1: 390 faults, 321 distinct, max 6 -> many distinct pages about once -> re-allocation: find the race ``` `distinct >= 20` is churn even when a few pages repeat, and `distinct < 20` with `repeats < 5` is not evidence of a live stale mapping. Under the old repeats-only rule the NPU stream read as a stale translation; under a distinct-only rule a 10-page buffer whose faults were spread evenly would read as churn. **The cross-boot intersection is now reported as a fact** — `shared across boots, <dev> domain=<dom>: N address(es)` — with no inference attached at all. There is nothing honest to attach to an empty or non-empty IOVA set, and the tool should not be the place where that distinction gets invented. **2. `intersection()` pooled devices.** Fixed: it groups per (device, domain) and only intersects labels within a group. An NPU address that happens to equal an NVMe IOVA is no longer a "shared buffer", and a short third boot can no longer empty a real two-boot match. **Arms added** (all 13 green): two boots sharing no page, each with a few pages repeating many times, must be labelled the stale-translation shape and must not be called a race; an NPU address equal to an NVMe address must not be offered as a shared buffer; and both of the box's real streams, using the counts the review measured (321 distinct/max 6, and 10 distinct/x380), land on the right side of the rule. Also noted for #2478 itself: the review confirms the journals **are** readable from strixhalo for a user in `adm`, so the census can be finished by someone who has answered the host-key question — which I still have not done and will not do unilaterally.
|
Both review points taken: the cross-boot test cannot decide the shape, and intersections pooled devices The CHANGES is right on the substance, and point 1 is not a nit — my headline inference was wrong. 1. These are IOVAs, so "disjoint across boots" is the expected answer under BOTH hypotheses. dma-iommu hands them out
The cross-boot intersection is now reported as a fact — 2. Arms added (all 13 green): two boots sharing no page, each with a few pages repeating many times, must be labelled the Also noted for #2478 itself: the review confirms the journals are readable from strixhalo for a user in |
|
VERDICT: READY at Both points are taken, and the first one changed the tool's conclusion rather than its wording:
Arms: 13, all green, including the three you asked for in effect — disjoint boots with repeating pages must be the buffer The parser's verdicts you confirmed (requester from the device prefix, domain kept, non-fault lines ignored) are untouched. And thank you for the journals detail — for #2478 that is the missing piece: the census can be run by someone who has settled |
|
Persistent review updated to latest commit fad5ec7 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and I am the demonstration for one improvement: the caveat is in the code, the misleading number is in the output
Your own correction is the most valuable thing in this PR and it is stated better than I could have: "these addresses are IOVAs, handed out per boot by dma-iommu in probe/allocation order, so one deterministic buffer with a stale translation still lands on a different IOVA every boot -- 'disjoint' is the expected answer under BOTH hypotheses, and a tool that reads it as 'race' is wrong on real hardware." The two data points behind it (3800/10, 2361/9, 71/4) are what make it a correction rather than a preference, and the revised shape rule requiring both <20 distinct and >=5 repeats per address is right for the reason you give -- each criterion alone mislabels one of this box's two streams.
I ran it on this box's two journals before reading that comment, and drew exactly the wrong conclusion from the bare number. My output:
boot-1 ... 2361 fault(s), 9 distinct -> 262.3 faults per address
boot0 ... 96 fault(s), 4 distinct -> 24.0 faults per address
shared across boots, 0000:c4:00.0 domain=0x0015: 0 address(es)
I treated 0 address(es) as decisive -- "disjoint implies a free-and-reuse race, so the stale-mapping hypothesis is refuted" -- and posted that to #2478 before noticing the caveat four lines up in the source. The caveat is correct and it is in a code comment; the number that misleads is in the stdout. Whoever needs it is reading stdout.
The ask, therefore, is one line of output: when the intersection is empty, print the hedge next to it --
shared across boots, …: 0 address(es) (expected under BOTH hypotheses for IOVAs; the repeats-per-address
shape above is what discriminates -- see the `shape()` note)
and, symmetrically, when two or more journals do share an address, that is worth its own sentence too, because it would mean the IOVA allocation order was stable enough to repeat -- which is itself informative. The point is that 0 currently reads as a verdict and is not one.
Everything else holds: --journal repeatable with stdin and synthetic arms so it runs on a capture or in CI; exit 77 when nothing was scanned; and the per-(boot, device, domain) key, which is the unit the fault stream actually comes in (0000:c4:00.0 domain=0x0015 for every one of this boot's 96).
…follow-up) (#2499) * server_launch_check: a group kill is the FIX, not the defect (and a waiver for the residue) Ran the merged checker (b9a04cc) on the two correct remedies for its own bug: setsid "$BIN" serve -p $PORT & PID=$! ... kill -- -$PID -> rc 1, 2 findings setsid "$BIN" serve -p $PORT & ... kill -TERM -- -$PGID -> rc 1, 1 finding Both are right: `setsid cmd &` makes the child a session leader, so its pgid IS the leader's pid -- what $! holds -- and a group kill reaches the server even though the leader exited. It is the form gpu-hold's kill_tree uses. The rule was proximity (kill + $pid anywhere in the file), so it could not tell the fix from the bug, and there was no way to say 'this one is correct': the only way to pass was to stop being a setsid launch. Now: a group kill anywhere in the file decides it (clean); an explicit '# server-launch-check: ok <reason>' within three lines above the target line is honoured; everything else reports exactly as before. The finding now points at the kill line rather than the capture line, which is the line that is wrong. 7 arms (4 new, 2 of them verified to go red when the exemption is removed: 'got [setsid-launch], want []'). Real tree: rc 0, no findings; a real offender file: rc 1, both findings. * Refs #2478: the runbook for its two root-only tests, so the next person does not repeat the week #2478 has a mechanism hypothesis, a falsifiable prediction and a verified kernel parameter, and all three are useless to anyone who has not been living in the issue for an hour. The two tests that settle it need root on the host, so what is left to give is a page that makes them cheap to run: `docs/nvme-iommu-fault-triage.md`. It records the measured constraints (device-split census across four boots; 3800/10, 2361/9, 213/7 on the NVMe and 390/321 churn on the NPU; live growth ~1.7/min; the 64 MiB HMB on a DRAM-less PC41Q), the kernel's own settings (`lazy mode` invalidation, SWIOTLB), and then the two tests with what each result *means*: 1. **Disable the HMB** (`nvme set-feature /dev/nvme0 -f 0x0d -v 0`, no reboot). Stream stops ⇒ the faulting region is the HMB. Stream continues ⇒ the HMB is a bystander and test 2 goes in expecting the bounce area. 2. **`iommu.strict=1`**, one reboot. Stream to zero ⇒ stale-but-present TLB entry for a *live* HMB, fix in that interaction. Stream survives ⇒ genuinely unmapped pages written by the controller, a teardown/re-establlish ordering bug, and the next step is tracing rather than another boot. Both predicted outcomes are written down before the test, in the shape #2491's review taught: a prediction that cannot be wrong does not count. Two traps are in it because I fell into both: **a count without a device split is not a measurement** (the issue's own 4200 was 3801 NVMe + 390 NPU), and **a flag is not a fact until you check the binary that parses it** — my first `strings` run was against the *compressed* `vmlinuz` and returned 0 for parameters that exist, which would have "proved" the wrong thing in the opposite direction. Linked from `tools/diag/iommu_fault_census.py`'s own docstring, because a page nothing links to is a page nobody runs. Written for whoever has root on strixhalo, since this lane does not. * server_launch_check: the group-kill exemption is PER LAUNCH, not per file Review on #2499, measured: one correct pair exempted the whole file, so setsid serve -p 1111 & PA=$! kill -- -$PA # correct setsid forward-serve -p 2222 & PB=$! kill "$PB" # BROKEN came back rc 0. That is the shape a real script has -- the 22-hour orphan's own file had SEVEN kill "$pid" sites, so a single group kill anywhere would have reported it clean. Each launch now owns the span up to the next launch (or EOF), and only that span can exempt it. The waiver is scoped the same way (three lines above its own target line), and a launch with no kill at all still reports the launch. Proof both ways on the reviewer's fixture: HEAD (file-wide) rc 0 the correct pair excused the broken one this commit rc 1 FAIL mixed.sh:5 [setsid-launch] + :6 [kill-after-setsid] 8 arms: the 7 that were there all still pass, plus 'a correct pair does not excuse a broken pair in the same file'. * The runbook tells someone to run a command this box does not have `command -v nvme` returns nothing on strixhalo, so test 1's first line would have failed with 'command not found' and read like the *test* failing rather than a missing package. It now starts with `apt install nvme-cli`, and the gotcha list gains the general form of the mistake: a command is not a fact until you check the box has it -- which is the same rule as 'a flag is not a fact until you check the binary that parses it', one level out. Found by applying this page's own rule to this page. --------- Co-authored-by: t <t@e.com> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
* server_launch_check: a group kill is the FIX, not the defect (and a waiver for the residue) Ran the merged checker (b9a04cc) on the two correct remedies for its own bug: setsid "$BIN" serve -p $PORT & PID=$! ... kill -- -$PID -> rc 1, 2 findings setsid "$BIN" serve -p $PORT & ... kill -TERM -- -$PGID -> rc 1, 1 finding Both are right: `setsid cmd &` makes the child a session leader, so its pgid IS the leader's pid -- what $! holds -- and a group kill reaches the server even though the leader exited. It is the form gpu-hold's kill_tree uses. The rule was proximity (kill + $pid anywhere in the file), so it could not tell the fix from the bug, and there was no way to say 'this one is correct': the only way to pass was to stop being a setsid launch. Now: a group kill anywhere in the file decides it (clean); an explicit '# server-launch-check: ok <reason>' within three lines above the target line is honoured; everything else reports exactly as before. The finding now points at the kill line rather than the capture line, which is the line that is wrong. 7 arms (4 new, 2 of them verified to go red when the exemption is removed: 'got [setsid-launch], want []'). Real tree: rc 0, no findings; a real offender file: rc 1, both findings. * Refs #2478: the runbook for its two root-only tests, so the next person does not repeat the week #2478 has a mechanism hypothesis, a falsifiable prediction and a verified kernel parameter, and all three are useless to anyone who has not been living in the issue for an hour. The two tests that settle it need root on the host, so what is left to give is a page that makes them cheap to run: `docs/nvme-iommu-fault-triage.md`. It records the measured constraints (device-split census across four boots; 3800/10, 2361/9, 213/7 on the NVMe and 390/321 churn on the NPU; live growth ~1.7/min; the 64 MiB HMB on a DRAM-less PC41Q), the kernel's own settings (`lazy mode` invalidation, SWIOTLB), and then the two tests with what each result *means*: 1. **Disable the HMB** (`nvme set-feature /dev/nvme0 -f 0x0d -v 0`, no reboot). Stream stops ⇒ the faulting region is the HMB. Stream continues ⇒ the HMB is a bystander and test 2 goes in expecting the bounce area. 2. **`iommu.strict=1`**, one reboot. Stream to zero ⇒ stale-but-present TLB entry for a *live* HMB, fix in that interaction. Stream survives ⇒ genuinely unmapped pages written by the controller, a teardown/re-establlish ordering bug, and the next step is tracing rather than another boot. Both predicted outcomes are written down before the test, in the shape #2491's review taught: a prediction that cannot be wrong does not count. Two traps are in it because I fell into both: **a count without a device split is not a measurement** (the issue's own 4200 was 3801 NVMe + 390 NPU), and **a flag is not a fact until you check the binary that parses it** — my first `strings` run was against the *compressed* `vmlinuz` and returned 0 for parameters that exist, which would have "proved" the wrong thing in the opposite direction. Linked from `tools/diag/iommu_fault_census.py`'s own docstring, because a page nothing links to is a page nobody runs. Written for whoever has root on strixhalo, since this lane does not. * Test 1 is one sysfs write, not a package install (and this commit is the one on the branch) Two corrections in one, because the first version of this commit never reached the remote: 1. nvme-cli is not installed here, so the runbook opened with a command that would read as the test failing. 2. Better than installing it: the kernel exposes the HMB switch at /sys/class/nvme/nvme0/hmb -- readable as a user (reads 1 here), writable by root (checked with [ -w ]). The test is one write, no package, and it verifies the value took before trusting the result. The nvme-cli form stays as the fallback for firmware that refuses the sysfs write. --------- Co-authored-by: t <t@e.com> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Refs #2478: make the address-set question a command, with arms
#2478 records 2361
IO_PAGE_FAULTs on the root NVMe SSD in the boot that died, with a handful of addressesrepeating hundreds of times — and it correctly says the next question is whether that address set is the same
in the boot before. That question has two indistinguishable answers in a
head -3:same set across boots -> one deterministic allocation whose translation went stale: find the buffer
disjoint sets -> re-allocation (or a free and reuse) behind a live descriptor: find the race
It is also currently unanswerable: the box's SSH host key is ED25519->ED25519 changed, so nobody can read its
journals until the owner says whether a reinstall happened. So instead of waiting with a command in a comment, this
ships the command.
tools/diag/iommu_fault_census.pytakes journal text (stdin or--journal, repeatable) and prints, per(boot, device, IOMMU domain): fault count, distinct addresses, the top five, and the cross-boot intersection with
which of the two shapes it implies. It reads no device, so it runs on a capture, on another host, or synthetically.
Eight arms, all green: the same address twice is one distinct address; the requester is taken from the device
prefix rather than from "AMD-Vi"; the IOMMU domain is kept; disjoint boots share nothing; a shared address is found
across boots; a non-fault
nvme ... timeoutline is ignored; a line with no device prefix still counts as-; andone boot alone refuses to answer the cross-boot question instead of implying "disjoint".
The repo's own gate from #2456 caught this PR before I could skip it: it declares a
--self-testand no runnernamed it, so it now has a real row in
cmake/tests-unreferenced-selftest-rows.cmakeand the check reports 93/62/48with the row present. Two of those arms exist because a naive version would have failed them (the
-device caseand the one-boot case), which is the #2456 lesson applied to new code rather than to recovered code.
The issue stays open: the census is the instrument, and it still needs one real journal pair to say which shape the
NVMe fault stream is.