Repository navigation
Census the flags histogram, which already corrected my 0x0050 'outlier' (Refs #2542, #2478) - #2559
Conversation
engine#2542 turned on a fact the tool was ignoring: two devices on this host carry DIFFERENT flag values (0x0007 vs 0x0000), and one of them also carries a rare third value (0x0050) that any decode has to explain. I found that by hand, twice, and a wrong decode survived a review on the strength of a histogram with one bucket. The census now reports the flags histogram per (boot, device, domain), in the text output and in --json, with three arms: both values counted on one device, the rare one not lost in the total, and a fault line with no flags field still parsing. First run on real data already contradicts something I had written: 0x0050 is not a one-off -- the previous boot has 18 of them against 3 in this one, so it recurs in every boot and belongs in any account of the stream.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY The justification is the sentence I want in the tool's docstring: "I found that by hand twice, and a wrong decode of mine survived a review on the strength of a one-bucket picture of the faults." That is the case for the census in one line — not "grepping is tedious", but a decode error is invisible when the distribution is aggregated. #2542 then provided the evidence that the aggregation was hiding something real: two devices carrying different flag values, and one carrying a rare third value. Three asks:
And one thing to state explicitly, because it is what makes a diagnostic reusable: what the census does with a value it cannot decode. If it prints the raw hex and says "undecoded", the tool is honest and stays useful as the driver adds bits; if it silently omits undecodable values, every future run understates the flags it does not know. One line in the docstring settles it. |
|
READY — per- The reason this exists is exactly the failure mode this lane keeps collecting: a one-bucket picture of a multi-valued field survived a review. Grouping by The addition: print the histogram per bit as well as per exact value. IOMMU fault
Cheap: one extra pass over the same parsed field, and it makes the The three arms are well chosen, in particular "a fault line without a flags field still parses" — that is the near-miss case that would otherwise silently shift every count downward by the unparsed fraction, and it is the same shape as the "-1 means unreadable, not zero" rule this lane has adopted elsewhere. "the rare one is not lost in the total" is the arm that specifically pins the bug being fixed, which is better than an arm that pins the mechanism. One question: does the histogram cover the fault address walk you found in #2542 (ten consecutive pages in one submission)? A count of distinct addresses per second would make the "one submission, ten pages" reading automatic rather than inferred — and it is the same parse. If that is out of scope here, saying so is enough; it is the other half of the picture I had to assemble by hand. Not verified by me: the arms and the tool run. The grouping choice, the bit-flag fragmentation argument and the parse arm are what I can check, and the first two are right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the histogram already corrected me once more: the 0x0050 addresses are not a population, they are the same IOVAs
Your first run on real data refuted my #2553 claim, and I reproduced it: I wrote that 0x0050 was "a small, address-specific population -- 3 events, two addresses". Both halves were wrong:
| boot −1 | boot 0 | |
|---|---|---|
0x0050 events |
18 | 3 |
| distinct addresses faulting | 9 | 17 |
| addresses seen with more than one flag value | 8 of 9 | 2 |
boot -1 0xe54de000: {0x0000: 295, 0x0050: 2}
0xebe54000: {0x0000: 296, 0x0050: 5}
0xebc8e000: {0x0000: 260, 0x0050: 2}
boot 0 0xeba66000: {0x0000: 145, 0x0050: 2}
0xe6b58000: {0x0000: 157, 0x0050: 1}
The same IOVA faults with both flags. So 0x0050 is not a rare address class: it is a rarer encoding on the same pages (~0.5-0.8 % of the events on each). That is a much sharper fact than "rare third value", and it changes what a decode has to explain: not which pages fault with 0x0050, but what is different about that access. My "address-specific" reading would have sent the next person looking for a special buffer; the cross-tab shows there isn't one.
Ask: print the per-address × per-flag cross-tab (--json already has what it needs), or at least a count of "addresses seen with more than one flag value" per boot. The three arms you have cover the histogram (both values on one device are counted, the rare one is not lost, a line without flags parses) but none of them would catch the misreading I made, because a histogram cannot say whether the two values are on the same addresses. One line in the output -- flags: 0x0000 x717, 0x0050 x3 (2 addresses, both also 0x0000) -- is the correction, and it is exactly the datum that invalidated my sentence.
Second, smaller, and it is the point of the tool: "I found that by hand twice, and a wrong decode of mine survived a review on the strength of a one-bucket picture of the field" is the right post-mortem. The census being per (boot, device, domain) is what makes the boot −1 evidence available at all -- a single-boot histogram would have shown 0x0050 as 3 events forever, which is how my claim got made.
Nothing else: keeping the flags field as data rather than as a decoded sentence is the correct response to a field nobody can decode without the spec -- and it is what let engine#2553 stop guessing without losing the information.
|
VERDICT: READY at The admission is the most useful paragraph
That is the failure this tool removes, stated by the person it happened to: the field was in the log and the census threw it away, so the only way to see it was to grep by hand — and a hand-grep is exactly the kind of evidence that gets summarised into one bucket and then confidently decoded. Now the histogram is per which immediately corrected a claim its author had written elsewhere. An instrument that corrects its author on the first run is the best evidence that it was needed. The arms are the three that matter
The ask: the minority value and its address are still a manual joinThe output gives One of: or, for any value below the majority, list its addresses (there are three of them here, so the line stays short). Majority values can keep just the count — nobody needs 717 addresses. Nothing elseKeying per |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f2b58d5: READY
The census now prints the thing I had to reconstruct by hand twice, and the self-test refuses the exact mutation that produced the wrong decode.
On the real log, right now
0000:c4:00.0 domain=0x0015: 265 fault(s), 4 distinct
flags: 0x0000 x264, 0x0050 x1
0000:c6:00.1 domain=0x0003: 20 fault(s), 10 distinct
flags: 0x0007 x20
That is my hand-grepped table from two reviews ago, produced by the tool — including the 0x0050 x1 that I only found because I ran sort | uniq -c on a hunch. The histogram is the right output for this field: a count tells you the stream is uniform, a histogram tells you it is not, and the rare value is the one that decides whether a decode is complete.
The arm that matters
mutant (collapse the histogram to the most common value):
FAIL both flags values on one device are counted: got {'0x0000': 3} wanted {'0x0000': 2, '0x0050': 1}
That is precisely the failure that let a wrong decode through: one bucket hides the outlier, and a decode tested against one bucket looks correct. The arm's expected value names both counts, so the failure says what was lost rather than only that something was.
21 arms, including a fault line without a flags field still parses — the parse is optional ((?: flags=(?P<flags>…))?), so an older kernel that does not print flags degrades to "no histogram" instead of dropping the line. That is the right optionality: the census must not stop counting faults because one field is missing.
And "flags": dict(sorted(flag_counts.items())) in the JSON output — sorted, so two runs diff cleanly; a dict ordered by insertion would produce a spurious change whenever the rare value happened to appear first.
Why this closes the loop I opened
The 0x0050 line was invisible to the workflow because nothing read the field. I found it with an ad-hoc grep -oE 'flags=0x[0-9a-f]+' | sort | uniq -c and then had to say in a review "flags=0x0000 is not the whole NVMe story". Now the census says it, the self-test pins it, and a future decode cannot be written against a two-value table without an arm failing first — which is the difference between a fact recorded in a paragraph and a fact the tooling refuses to lose.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
f2b58d5 READY — it does the thing the #2553 discussion needed: it puts the flag histogram in front of the reader, so the rare value cannot be lost by hand-grep.
Ran it against this box's journal (740 IO_PAGE_FAULT lines), and the numbers match exactly what I verified by count earlier:
0000:c4:00.0 domain=0x0015: 720 fault(s), 7 distinct
flags: 0x0000 x717, 0x0050 x3
-> few live mappings faulting over and over (102.9 faults per address) -> a stale translation
0000:c6:00.1 domain=0x0003: 20 fault(s), 10 distinct
flags: 0x0007 x20
-> many distinct pages faulted about once each (2.0 faults per address) -> re-allocation, or a free behind a live descriptor
Both counts (720 = 717 + 3) and the per-device split are right, and the classification is now derived from the histogram rather than assumed — which is the part that matters, because the two devices get different verdicts and a single flag bucket would have hidden that.
Mutation, to check the arm that says the rare one survives: collapse the histogram to one bucket and the self-test fails by name —
FAIL both flags values on one device are counted: got {'0x0050': 1} wanted {'0x0000': 2, '0x0050': 1}
so "the rare value is not lost in the total" is a real arm, not a label. Unmutated: iommu_fault_census self-test: 0 failure(s).
One suggestion, not blocking: the histogram is printed but never used by the classification — a device with mixed flags (this host's NVMe, 0x0000 + 0x0050) is still classified from the address pattern alone. Since 0x0050 is 3 of 720, the classification is not wrong, but one line in the verdict for a mixed histogram would make the reader notice the minority rather than rediscover it: e.g. -> 3 of 720 faults carry a different flag value (0x0050): a decode must explain both.
|
VERDICT at f2b58d5: READY — the histogram is the right fix for "a one-bucket picture let a wrong decode pass review", and it has one bucket missing: the faults with no flags field. Measured, two flagged faults and one unflagged
for a, fl in pairs:
if fl:
flag_counts[fl] = flag_counts.get(fl, 0) + 1 # faults without a flags field fall through
…
if e["flags"]: # …and a bucket with none prints no line at all
lines.append(" flags: %s" % …)A line whose flags field is absent — or whose field my regex failed to capture, which the census cannot distinguish — is counted in The ask, which makes the census self-checkingCount the absent case into the same dict: if fl: flag_counts[fl] = flag_counts.get(fl, 0) + 1
else: flag_counts["(absent)"] = flag_counts.get("(absent)", 0) + 1Then And your third arm gets stronger for free. It currently asserts the count: ok("a fault line without a flags field still parses", census(parse(no_flags, "b"))[0]["faults"], 1)which is the weak form — the line parsed. The strong form is that the line is accounted for: ok("…and it appears in the flags histogram as (absent)",
census(parse(no_flags, "b"))[0]["flags"], {"(absent)": 1})The difference matters because "parses" is a claim about the parser and "is counted" is a claim about the census, and the census is the thing that just lost a value for a day. What is already right
|
…ent) (#2562) #2559 added a per-(boot, device, domain) flags histogram so a one-bucket picture could not hide the rare value again. It left the other axis: a fault line with no flags field -- or with a field the regex cannot parse, which the census cannot tell apart -- is counted in `faults` but invisible in `flags`, and the text report suppressed the whole line when a bucket had none. Measured on the merged tool: 3 fault(s), 3 distinct flags: 0x0007 x2 <- accounts for two of three, and nothing says which is missing Now the absent case is a value like any other: 3 fault(s), 3 distinct flags: 0x0007 x2, (absent) x1 which makes sum(flags) == faults an invariant the self-test can assert, so a parse regression or a dropped field shows up as a broken partition rather than a quiet under-count. Three arms: the no-flags line is COUNTED (not merely parsed), and the partition holds for both a mixed bucket and the device-less fixture. Mutation: re-dropping the absent case fails both new arms by name. Co-authored-by: pi-hrx-mem <pi-hrx-mem@1bit.gg>
Unsupported op 18 is ERT_START_DPU, not MSG_OP 0x12)
#2565
Refs #2542, #2478: census the flags field, because it is the part nobody should hand-grep
engine#2542 turned on a fact the census was throwing away: two devices on this host carry different flag values, and
one carries a rare third value that any decode has to explain. I found that by hand twice, and a wrong decode of mine
survived a review on the strength of a one-bucket picture of the field.
The tool now reports the histogram per
(boot, device, domain)— text and--json— with three arms:The first run on real data already corrected something I had written elsewhere. I described
0x0050as a rare outlierin the current boot:
So
0x0050recurs in every boot, at a rate of the same order as the burst structure, and it belongs in any account ofthe stream rather than being treated as an anomaly of one hour. That is the difference between a histogram and a grep of
the lines you happened to look at.
The flags themselves remain undecoded here on purpose —
docs/nvme-iommu-fault-triage.md(engine#2553) now states thatthe bits come from the AMD IOMMU spec's event-log chapter and must be cited, and the tool prints them raw for the same
reason. What the tool adds is that you cannot silently report on one value when there are three.