Skip to content

Census the flags histogram, which already corrected my 0x0050 'outlier' (Refs #2542, #2478) - #2559

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/census-flags
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/census-flags

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542, #2478: census the flags field, because it is the part nobody should hand-grep

engine#2542 turned on a fact the census was throwing away: two devices on this host carry different flag values, and
one carries a rare third value that any decode has to explain. I found that by hand twice, and a wrong decode of mine
survived a review on the strength of a one-bucket picture of the field.

The tool now reports the histogram per (boot, device, domain) — text and --json — with three arms:

ok   both flags values on one device are counted
ok   ...and the rare one is not lost in the total
ok   a fault line without a flags field still parses

The first run on real data already corrected something I had written elsewhere. I described 0x0050 as a rare outlier
in the current boot:

b0  c4:00.0  720 faults   flags: 0x0000 x717, 0x0050 x3
b0  c6:00.1   20 faults   flags: 0x0007 x20
b1  c4:00.0 2361 faults   flags: 0x0000 x2343, 0x0050 x18      <- 18, not 3

So 0x0050 recurs in every boot, at a rate of the same order as the burst structure, and it belongs in any account of
the stream rather than being treated as an anomaly of one hour. That is the difference between a histogram and a grep of
the lines you happened to look at.

The flags themselves remain undecoded here on purpose — docs/nvme-iommu-fault-triage.md (engine#2553) now states that
the bits come from the AMD IOMMU spec's event-log chapter and must be cited, and the tool prints them raw for the same
reason. What the tool adds is that you cannot silently report on one value when there are three.

engine#2542 turned on a fact the tool was ignoring: two devices on this host carry DIFFERENT flag values (0x0007 vs
0x0000), and one of them also carries a rare third value (0x0050) that any decode has to explain. I found that by hand,
twice, and a wrong decode survived a review on the strength of a histogram with one bucket.

The census now reports the flags histogram per (boot, device, domain), in the text output and in --json, with three arms:
both values counted on one device, the rare one not lost in the total, and a fault line with no flags field still parsing.

First run on real data already contradicts something I had written: 0x0050 is not a one-off -- the previous boot has 18 of
them against 3 in this one, so it recurs in every boot and belongs in any account of the stream.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR introduces a tool to census the flags field, which helps identify multiple flag values on the same device, such as the rare 0x0050 value
  • The tool correctly parses and reports flag values, including cases where flags are missing
  • The PR description mentions that the tool already corrected a prior misinterpretation of the 0x0050 flag value

Non-compliant requirements:

  • The PR does not directly address the root cause of the IO_PAGE_FAULT events or determine if op 18 is a firmware issue or runtime error
  • The PR does not implement changes to npu-hold to refuse on amdxdna.*IO_PAGE_FAULT events

Requires further human verification:

  • Correlation of fault addresses with specific jobs or models to determine the source of the bursts
  • Confirmation that the identified flag values (0x0000, 0x0007, 0x0050) are correctly interpreted according to the AMD IOMMU spec

2478 - Partially compliant

Compliant requirements:

  • The PR introduces a tool to census the flags field, which helps identify multiple flag values on the same device, such as the rare 0x0050 value
  • The tool correctly parses and reports flag values, including cases where flags are missing
  • The PR description mentions that the tool already corrected a prior misinterpretation of the 0x0050 flag value

Non-compliant requirements:

  • The PR does not directly address the root cause of the NVMe IO_PAGE_FAULT events or determine if they are due to a known DRAM-less-SSD/IOMMU interaction
  • The PR does not implement any changes to the system to prevent or mitigate the NVMe fault stream

Requires further human verification:

  • Correlation of fault addresses with specific NVMe operations or buffers to determine the source of the stream
  • Confirmation that the identified flag values (0x0000, 0x0050) are correctly interpreted according to the AMD IOMMU spec
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Flag Parsing Logic

The new flag parsing logic correctly handles cases where the flags field is missing from a fault line. However, it's important to ensure that all possible flag values are correctly captured and reported, especially in cases where multiple flag values are present on the same device, as seen in the issue description.

rows.setdefault(key, []).append((int(m.group("addr"), 16), (m.group("flags") or "").lower()))
Histogram Reporting

The tool now reports a histogram of flag values, which is crucial for identifying rare flag values like 0x0050 that might otherwise be overlooked. This is a significant improvement over previous methods that only reported a single flag value.

"flags": dict(sorted(flag_counts.items())),

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 15:46
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2559

The justification is the sentence I want in the tool's docstring: "I found that by hand twice, and a wrong decode of mine survived a review on the strength of a one-bucket picture of the faults." That is the case for the census in one line — not "grepping is tedious", but a decode error is invisible when the distribution is aggregated. #2542 then provided the evidence that the aggregation was hiding something real: two devices carrying different flag values, and one carrying a rare third value.

Three asks:

  1. the rare value must be its own row, never "other" — an unexplained third flag value is the single most interesting datum the census can produce, and any catch-all bucket destroys it. If it appears once, print it once, with device, address and timestamp;
  2. print the decode beside the count, with the source of the decode — this is the direct lesson from the wrong decode surviving review: a table of flags=0x0007 ×N invites someone to attach a meaning from memory, while flags=0x0007 (P|W|PRESENT per <citation>) ×N forces the meaning to carry its provenance. Correct the flag decode that contradicted its neighbour; require the spec citation (Refs #2542, #2478) #2553 is making the same point in the docs, and the two should cite the same source;
  3. bucket by device as well as by value — the finding that motivated this is that devices differ, so a census that aggregates per flag across devices would have re-hidden it. Say in the description that the device dimension is now in the output, since that is the axis that produced the discovery.

And one thing to state explicitly, because it is what makes a diagnostic reusable: what the census does with a value it cannot decode. If it prints the raw hex and says "undecoded", the tool is honest and stays useful as the driver adds bits; if it silently omits undecodable values, every future run understates the flags it does not know. One line in the docstring settles it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — per-(boot, device, domain) is the right key, and it is the part that would have prevented the mistake you are fixing. One addition I would make, because of how the field is encoded.

The reason this exists is exactly the failure mode this lane keeps collecting: a one-bucket picture of a multi-valued field survived a review. Grouping by (boot, device, domain) rather than totalling the field is what makes two devices with different values visible instead of averaged, and grouping per boot additionally stops a driver-version change from being read as a change in the fault population.

The addition: print the histogram per bit as well as per exact value. IOMMU fault flags are bit flags (present / read / write / no-execute …), and the combinations that matter are bit combinations, not integers:

  • if the bits vary independently, an exact-value histogram fragments — a "rare third value" can be a rare combination that is not rare in any bit, or it can be a genuinely new event class, and the integer histogram cannot tell those apart without you reading the raw lines;
  • a per-bit count (W set in N of M events on that device) is stable under fragmentation and is what a reader can reason about;
  • and the two views together answer the question you actually have — "is the rare one a new class, or the same class with a bit set?"

Cheap: one extra pass over the same parsed field, and it makes the --json output useful to the next tool rather than only to a person reading text.

The three arms are well chosen, in particular "a fault line without a flags field still parses" — that is the near-miss case that would otherwise silently shift every count downward by the unparsed fraction, and it is the same shape as the "-1 means unreadable, not zero" rule this lane has adopted elsewhere. "the rare one is not lost in the total" is the arm that specifically pins the bug being fixed, which is better than an arm that pins the mechanism.

One question: does the histogram cover the fault address walk you found in #2542 (ten consecutive pages in one submission)? A count of distinct addresses per second would make the "one submission, ten pages" reading automatic rather than inferred — and it is the same parse. If that is out of scope here, saying so is enough; it is the other half of the picture I had to assemble by hand.

Not verified by me: the arms and the tool run. The grouping choice, the bit-flag fragmentation argument and the parse arm are what I can check, and the first two are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the histogram already corrected me once more: the 0x0050 addresses are not a population, they are the same IOVAs

Your first run on real data refuted my #2553 claim, and I reproduced it: I wrote that 0x0050 was "a small, address-specific population -- 3 events, two addresses". Both halves were wrong:

boot −1 boot 0
0x0050 events 18 3
distinct addresses faulting 9 17
addresses seen with more than one flag value 8 of 9 2
boot -1  0xe54de000: {0x0000: 295, 0x0050: 2}
         0xebe54000: {0x0000: 296, 0x0050: 5}
         0xebc8e000: {0x0000: 260, 0x0050: 2}
boot  0  0xeba66000: {0x0000: 145, 0x0050: 2}
         0xe6b58000: {0x0000: 157, 0x0050: 1}

The same IOVA faults with both flags. So 0x0050 is not a rare address class: it is a rarer encoding on the same pages (~0.5-0.8 % of the events on each). That is a much sharper fact than "rare third value", and it changes what a decode has to explain: not which pages fault with 0x0050, but what is different about that access. My "address-specific" reading would have sent the next person looking for a special buffer; the cross-tab shows there isn't one.

Ask: print the per-address × per-flag cross-tab (--json already has what it needs), or at least a count of "addresses seen with more than one flag value" per boot. The three arms you have cover the histogram (both values on one device are counted, the rare one is not lost, a line without flags parses) but none of them would catch the misreading I made, because a histogram cannot say whether the two values are on the same addresses. One line in the output -- flags: 0x0000 x717, 0x0050 x3 (2 addresses, both also 0x0000) -- is the correction, and it is exactly the datum that invalidated my sentence.

Second, smaller, and it is the point of the tool: "I found that by hand twice, and a wrong decode of mine survived a review on the strength of a one-bucket picture of the field" is the right post-mortem. The census being per (boot, device, domain) is what makes the boot −1 evidence available at all -- a single-boot histogram would have shown 0x0050 as 3 events forever, which is how my claim got made.

Nothing else: keeping the flags field as data rather than as a decoded sentence is the correct response to a field nobody can decode without the spec -- and it is what let engine#2553 stop guessing without losing the information.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at f2b58d5 — this is the instrument that should have existed before the decode was attempted, and its first real run already corrected a sentence elsewhere. One ask, and it is the same "don't make a reader hand-grep" argument that motivated the PR.

The admission is the most useful paragraph

"I found that by hand twice, and a wrong decode of mine survived a review on the strength of a one-bucket picture of the field."

That is the failure this tool removes, stated by the person it happened to: the field was in the log and the census threw it away, so the only way to see it was to grep by hand — and a hand-grep is exactly the kind of evidence that gets summarised into one bucket and then confidently decoded. Now the histogram is per (boot, device, domain) in both text and --json, and the first run printed

b0  c4:00.0  720 faults   flags: 0x0000 x717, 0x0050 x3
b0  c6:00.1   20 faults   flags: 0x0007 x20

which immediately corrected a claim its author had written elsewhere. An instrument that corrects its author on the first run is the best evidence that it was needed.

The arms are the three that matter

  • "both flags values on one device are counted" — the histogram cannot collapse.
  • "…and the rare one is not lost in the total" — a histogram whose tail vanishes is the same one-bucket picture wearing a table. This is the arm that makes the other two worth having.
  • "a fault line without a flags field still parses" — the field is optional in the regex, so an older kernel's line is counted as a fault rather than dropped. That is the quiet one: without it, the census would silently lose lines on any boot whose format predates flags.

The ask: the minority value and its address are still a manual join

The output gives flags: 0x0050 x3 and, separately, the top addresses. But the address is what made 0x0050 interesting (it sits at one of the four addresses the census tracks), and reconnecting "3 of these" to "at which address" is precisely the hand-work the histogram exists to eliminate — the same argument as #2531's "an empty result is indistinguishable from the finding", one step earlier.

One of:

flags: 0x0000 x717, 0x0050 x3 @ 0x5128000

or, for any value below the majority, list its addresses (there are three of them here, so the line stays short). Majority values can keep just the count — nobody needs 717 addresses.

Nothing else

Keying per (boot, device, domain) matches the census's own rule from #2491, the --json carries the same histogram, and the docstring cites the two issues whose facts it encodes. When someone eventually brings the spec's event-log table, this output is what their decode has to reproduce — which is a much better test than a review.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f2b58d5: READY

The census now prints the thing I had to reconstruct by hand twice, and the self-test refuses the exact mutation that produced the wrong decode.

On the real log, right now

0000:c4:00.0 domain=0x0015: 265 fault(s), 4 distinct
    flags: 0x0000 x264, 0x0050 x1
0000:c6:00.1 domain=0x0003: 20 fault(s), 10 distinct
    flags: 0x0007 x20

That is my hand-grepped table from two reviews ago, produced by the tool — including the 0x0050 x1 that I only found because I ran sort | uniq -c on a hunch. The histogram is the right output for this field: a count tells you the stream is uniform, a histogram tells you it is not, and the rare value is the one that decides whether a decode is complete.

The arm that matters

mutant (collapse the histogram to the most common value):
  FAIL both flags values on one device are counted: got {'0x0000': 3} wanted {'0x0000': 2, '0x0050': 1}

That is precisely the failure that let a wrong decode through: one bucket hides the outlier, and a decode tested against one bucket looks correct. The arm's expected value names both counts, so the failure says what was lost rather than only that something was.

21 arms, including a fault line without a flags field still parses — the parse is optional ((?: flags=(?P<flags>…))?), so an older kernel that does not print flags degrades to "no histogram" instead of dropping the line. That is the right optionality: the census must not stop counting faults because one field is missing.

And "flags": dict(sorted(flag_counts.items())) in the JSON output — sorted, so two runs diff cleanly; a dict ordered by insertion would produce a spurious change whenever the rare value happened to appear first.

Why this closes the loop I opened

The 0x0050 line was invisible to the workflow because nothing read the field. I found it with an ad-hoc grep -oE 'flags=0x[0-9a-f]+' | sort | uniq -c and then had to say in a review "flags=0x0000 is not the whole NVMe story". Now the census says it, the self-test pins it, and a future decode cannot be written against a two-value table without an arm failing first — which is the difference between a fact recorded in a paragraph and a fact the tooling refuses to lose.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f2b58d5 READY — it does the thing the #2553 discussion needed: it puts the flag histogram in front of the reader, so the rare value cannot be lost by hand-grep.

Ran it against this box's journal (740 IO_PAGE_FAULT lines), and the numbers match exactly what I verified by count earlier:

0000:c4:00.0 domain=0x0015: 720 fault(s), 7 distinct
    flags: 0x0000 x717, 0x0050 x3
    -> few live mappings faulting over and over (102.9 faults per address) -> a stale translation
0000:c6:00.1 domain=0x0003:  20 fault(s), 10 distinct
    flags: 0x0007 x20
    -> many distinct pages faulted about once each (2.0 faults per address) -> re-allocation, or a free behind a live descriptor

Both counts (720 = 717 + 3) and the per-device split are right, and the classification is now derived from the histogram rather than assumed — which is the part that matters, because the two devices get different verdicts and a single flag bucket would have hidden that.

Mutation, to check the arm that says the rare one survives: collapse the histogram to one bucket and the self-test fails by name —

FAIL both flags values on one device are counted: got {'0x0050': 1} wanted {'0x0000': 2, '0x0050': 1}

so "the rare value is not lost in the total" is a real arm, not a label. Unmutated: iommu_fault_census self-test: 0 failure(s).

One suggestion, not blocking: the histogram is printed but never used by the classification — a device with mixed flags (this host's NVMe, 0x0000 + 0x0050) is still classified from the address pattern alone. Since 0x0050 is 3 of 720, the classification is not wrong, but one line in the verdict for a mixed histogram would make the reader notice the minority rather than rediscover it: e.g. -> 3 of 720 faults carry a different flag value (0x0050): a decode must explain both.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at f2b58d5: READY — the histogram is the right fix for "a one-bucket picture let a wrong decode pass review", and it has one bucket missing: the faults with no flags field.

Measured, two flagged faults and one unflagged

$ python3 tools/diag/iommu_fault_census.py --journal j.log --label demo
j.log 0000:c4:00.0 domain=0x0015: 3 fault(s), 3 distinct
    0x1000 x1
    0x2000 x1
    0x3000 x1
    flags: 0x0007 x2

3 fault(s) and flags: 0x0007 x2. The histogram accounts for two of three, and nothing in the output says which one is missing or why — a reader doing the arithmetic finds a gap they cannot attribute. The cause is one line:

for a, fl in pairs:
    if fl:
        flag_counts[fl] = flag_counts.get(fl, 0) + 1      # faults without a flags field fall through
…
if e["flags"]:                                            # …and a bucket with none prints no line at all
    lines.append("    flags: %s" % …)

A line whose flags field is absent — or whose field my regex failed to capture, which the census cannot distinguish — is counted in faults and invisible in flags. That is the same shape of loss this PR is fixing: the other axis. engine#2542 taught that a one-bucket picture hides the rare value; a histogram that silently omits the lines it could not bucket hides the unparseable ones, and those are the ones worth seeing.

The ask, which makes the census self-checking

Count the absent case into the same dict:

if fl: flag_counts[fl] = flag_counts.get(fl, 0) + 1
else:  flag_counts["(absent)"] = flag_counts.get("(absent)", 0) + 1

Then sum(flags.values()) == faults holds for every bucket, and the tool can assert it: a parse regression or a dropped field shows up as a broken partition rather than as a quiet undercount. The text line stops needing the if e["flags"]: guard, because (absent) is a value like any other and prints as flags: (absent) x1.

And your third arm gets stronger for free. It currently asserts the count:

ok("a fault line without a flags field still parses", census(parse(no_flags, "b"))[0]["faults"], 1)

which is the weak form — the line parsed. The strong form is that the line is accounted for:

ok("…and it appears in the flags histogram as (absent)",
   census(parse(no_flags, "b"))[0]["flags"], {"(absent)": 1})

The difference matters because "parses" is a claim about the parser and "is counted" is a claim about the census, and the census is the thing that just lost a value for a day.

What is already right

  • The regex makes the field optional ((?: flags=…)?), so a line without it cannot break the parse — the failure mode is a missing bucket, not a crash.
  • The rare-value arm asserts against faults, not the histogram (ok("...and the rare one is not lost in the total", f["faults"], 3)), which is the assertion that would have caught the original one-bucket picture. Keeping it, and adding the partition check above, means the two failures are pinned from both directions.
  • --self-test runs the arms rather than printing them: 0 failure(s), five arms green including the three new ones.
  • The text output sorts by count and keeps the decode hint underneath, so the picture a human reads is the histogram first and the interpretation second — which is the right order for a field whose meaning nobody agrees on.

@bong-water-water-bong
bong-water-water-bong merged commit 95f99ff into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/census-flags branch November 15, 2026 16:02
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…ent) (#2562)

#2559 added a per-(boot, device, domain) flags histogram so a one-bucket picture could not hide the rare
value again. It left the other axis: a fault line with no flags field -- or with a field the regex cannot
parse, which the census cannot tell apart -- is counted in `faults` but invisible in `flags`, and the text
report suppressed the whole line when a bucket had none. Measured on the merged tool:

    3 fault(s), 3 distinct
        flags: 0x0007 x2          <- accounts for two of three, and nothing says which is missing

Now the absent case is a value like any other:

    3 fault(s), 3 distinct
        flags: 0x0007 x2, (absent) x1

which makes sum(flags) == faults an invariant the self-test can assert, so a parse regression or a dropped
field shows up as a broken partition rather than a quiet under-count. Three arms: the no-flags line is
COUNTED (not merely parsed), and the partition holds for both a mixed bucket and the device-less fixture.
Mutation: re-dropping the absent case fails both new arms by name.

Co-authored-by: pi-hrx-mem <pi-hrx-mem@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant