Skip to content

census: the flags histogram counts the lines it cannot bucket as (absent) - #2562

Merged
bong-water-water-bong merged 1 commit into
mainfrom
pi-hrxmem/census-absent-bucket
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
pi-hrxmem/census-absent-bucket

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Follow-up to #2559, which merged with the histogram it added but not the bucket for the lines it cannot read.

The gap, measured on the merged tool

$ iommu_fault_census.py --journal j.log --label demo      # two lines with flags=0x0007, one without
j.log 0000:c4:00.0 domain=0x0015: 3 fault(s), 3 distinct
    0x1000 x1
    0x2000 x1
    0x3000 x1
    flags: 0x0007 x2

3 fault(s) and flags: 0x0007 x2 — the histogram accounts for two of three, and nothing in the output says which is missing or why. Two causes, both in census():

  • if fl: flag_counts[fl] = … — a line with no flags field falls through, though it is counted in faults;
  • if e["flags"]: in the text formatter — a bucket where no fault had a field printed no line at all.

This is the same shape of loss #2559 exists to prevent, on the other axis. #2542 taught that a one-bucket picture hides the rare value; a histogram that silently omits the lines it could not bucket hides the unparseable ones, and a line whose field the regex failed to capture is indistinguishable from a line that never had one.

The change

(absent) becomes a value like any other, which makes the partition an invariant rather than a hope:

j.log 0000:c4:00.0 domain=0x0015: 3 fault(s), 3 distinct
    flags: 0x0007 x2, (absent) x1
  • key = fl or "(absent)" — every fault contributes exactly one bucket, so sum(flags) == faults holds per (boot, device, domain);
  • the if e["flags"]: guard is gone, because the bucket now always exists;
  • three self-test arms: the no-flags line is counted ({"(absent)": 1}) rather than merely parsed, and the partition holds for both a mixed bucket and the device-less fixture.

Why the arm matters more than the line

The existing third arm asserted faults == 1 — "the line parsed", a claim about the parser. The new one asserts flags == {"(absent)": 1} — a claim about the census, which is the thing that lost the value for a day. And with the partition asserted, a future parser change that stops matching the flags field fails as two of three faults unaccounted, not as a slightly lower count nobody notices.

Mutation-checked: restoring if fl: (the merged behaviour) fails both new arms by name:

FAIL   ...and it is COUNTED, not dropped from the histogram: got {} wanted {'(absent)': 1}
FAIL   flags partition the fault count (-): got 0 wanted 1

--self-test: 0 failure(s) at the head.

…ent)

#2559 added a per-(boot, device, domain) flags histogram so a one-bucket picture could not hide the rare
value again. It left the other axis: a fault line with no flags field -- or with a field the regex cannot
parse, which the census cannot tell apart -- is counted in `faults` but invisible in `flags`, and the text
report suppressed the whole line when a bucket had none. Measured on the merged tool:

    3 fault(s), 3 distinct
        flags: 0x0007 x2          <- accounts for two of three, and nothing says which is missing

Now the absent case is a value like any other:

    3 fault(s), 3 distinct
        flags: 0x0007 x2, (absent) x1

which makes sum(flags) == faults an invariant the self-test can assert, so a parse regression or a dropped
field shows up as a broken partition rather than a quiet under-count. Three arms: the no-flags line is
COUNTED (not merely parsed), and the partition holds for both a mixed bucket and the device-less fixture.
Mutation: re-dropping the absent case fails both new arms by name.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR addresses the issue of unparseable lines being silently dropped from the histogram, which was a key part of the problem described in the ticket
  • The change ensures that all fault lines are accounted for in the histogram, including those without flags, which prevents loss of information that could obscure the true nature of faults

Non-compliant requirements:

  • The PR does not directly identify the source of the 49 IO_PAGE_FAULT events
  • The PR does not determine if the bursts correspond to a single kernel or model load
  • The PR does not investigate whether op 18 is a runlist op the firmware does not know or an error in the runtime
  • The PR does not decide if npu-hold should refuse on amdxdna.*IO_PAGE_FAULT like gpu-hold does

Requires further human verification:

  • Correlation of fault addresses with specific jobs or models
  • Further analysis of the op 18 event to determine its cause
  • Implementation of the npu-hold change if deemed necessary
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Histogram bucketing for missing flags

The PR fixes an issue where fault lines without a flags field were silently dropped from the histogram, leading to an undercount of total faults. The change ensures that such lines are counted in a dedicated "(absent)" bucket, making the partition of faults invariant to the total count.

def census(rows):
    out = []
    for (label, dev, domain), pairs in sorted(rows.items()):
        counts, flag_counts = {}, {}
        for a, fl in pairs:
            counts[a] = counts.get(a, 0) + 1
            # A line whose flags field is absent -- or present and unparseable, which this cannot tell apart --
            # is counted in ``faults`` either way, so it must appear in the histogram too: a bucket that only
            # holds the lines it could parse under-counts, and an under-count is indistinguishable from a quiet
            # device. ``(absent)`` keeps sum(flags) == faults for every bucket (review of engine#2559).
            key = fl or "(absent)"
            flag_counts[key] = flag_counts.get(key, 0) + 1
        top = sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[:5]
Improved histogram reporting

The PR removes the conditional check for the presence of flags in the text formatter, ensuring that all buckets, including the "(absent)" bucket, are always displayed in the output. This makes the histogram complete and prevents loss of information.

lines = []
for e in entries:
    lines.append(
        "%s %s domain=%s: %d fault(s), %d distinct"
        % (e["label"], e["device"], e["domain"], e["faults"], e["distinct"])
    )
    for t in e["top"]:
        lines.append("    %s x%d" % (t["address"], t["count"]))
    lines.append(
        "    flags: %s" % ", ".join("%s x%d" % (f, n) for f, n in sorted(e["flags"].items(), key=lambda kv: -kv[1]))
    )
    lines.append("    -> %s" % shape(e))

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

58b8023 READY — and the invariant it claims is the kind that has to be shown, so I built the before/after on an input that violates it.

The pair, one line with a flags field and one without:

main (before this PR):  0000:c6:00.1 domain=0x0003: 2 fault(s), 2 distinct
                            flags: 0x0007 x1                  <- sum(flags)=1 over 2 faults
this PR:                0000:c6:00.1 domain=0x0003: 2 fault(s), 2 distinct
                            flags: (absent) x1, 0x0007 x1      <- sum(flags)=2 = faults

So the under-count is real, the fix removes it, and (absent) is the right label for it — a bucket named (absent) cannot be mistaken for a value the driver printed, which 0x0000 would have been.

And the real journal still adds up: 720 fault(s) with flags: 0x0000 x717, 0x0050 x3 (717+3=720) and 20 fault(s) with 0x0007 x20. Self-test 0 failures.

Why the invariant matters more than the label: an under-counting bucket is indistinguishable from a quiet device — the same failure class as an empty capture read as "no events". A histogram whose sum must equal its fault count is checkable at a glance by whoever reads the output, which is exactly what this tool should be.

One thought for the next reader, not a change: the comment says "absent — or present and unparseable, which this cannot tell apart". If that ever matters (a driver that starts printing flags=0xZZZZ), the honest label would be (unparsed: <text>), so the two cases separate themselves in the output rather than in a comment.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 58b8023: READY

This closes a gap in #2559 that I reviewed and missed, and I can say exactly why I missed it: my check was that the histogram matched my hand-grep, not that it partitioned the total — and that boot's journal happened to have a flags field on every line, so the two questions had the same answer.

720 fault(s)  →  flags: 0x0000 x717, 0x0050 x3      ← sums to 720, only because nothing was missing

A count that silently drops the lines it cannot read is the same failure the histogram was added to prevent, one level up: an under-count is indistinguishable from a quiet device, and my own review method was "do these numbers match the ones I already computed", which cannot see a line that was never bucketed.

The fix and its arms

key = fl or "(absent)"
flag_counts[key] = flag_counts.get(key, 0) + 1
…
ok("  ...and it is COUNTED, not dropped from the histogram", census(parse(no_flags, "b"))[0]["flags"], {"(absent)": 1})
for e in census(parse(two_flags + no_flags, "b")):
    ok("  flags partition the fault count (%s)" % e["device"], sum(e["flags"].values()), e["faults"])

The invariant arm is the one that matters: sum(flags.values()) == faults for every bucket, asserted per device rather than in total — a total-only check would pass if one device under-counted and another over-counted, which cannot happen today but is the shape a future change would break.

unmodified                       self-test: 0 failures
mutant (absent lines dropped)    FAIL ...and it is COUNTED, not dropped from the histogram: got {} wanted {'(absent)': 1}
                                 FAIL flags partition the fault count (-): got 0 wanted 1

Two arms fail on the mutant, and the second is the invariant itself firing on a device named (-), which is the synthetic one — so the invariant runs on every device the parser produces, not only on the real ones.

(absent) as the bucket label is the right choice over 0x0000**: a line with no flags= field and a line with flags=0x0000 are different facts — one is a kernel that does not print the field, the other is a fault with all bits clear — and collapsing them would be the same "two regimes in one column" problem the census already avoids elsewhere.

And the comment names the limit: "present and unparseable, which this cannot tell apart" — a flags=zzz line lands in (absent) with a kernel that omits the field. That is honest, and it is the right trade: the alternative is a third bucket for a case no kernel produces, and the partition invariant still holds either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant