Repository navigation
diag: attribution reader — per-fault verdicts beside the sampler windows, an attributability outlook, and every output line tested - #2606
Conversation
…t exists
The join answers "which holder" once a fault lands. This adds the question that decides whether a
capture can answer it at all: over the holder timeline it already reads, how many faults would a
+-W window be able to attribute?
tools/diag/iommu_fault_attribution.py --samples ~/.cache/accel-sampler/samples.log --outlook 2 5 10 15
window one-holder ambiguous nothing-held no-sample
2s 61.3% 22.4% 16.3% 0.0%
5s 62.1% 23.1% 14.9% 0.0%
10s 63.6% 24.9% 11.6% 0.0%
15s 64.0% 26.6% 9.4% 0.0%
That is 3085 real samples from this box's live sampler (read-only). Two things it makes visible:
* **ambiguity rises with the window** (22.4 % -> 26.6 % from +-2 s to +-15 s), which is the opposite
of the resolution intuition: a wider window adds HOLDERS, not evidence. Same direction as the
+-5/10/15 numbers from an earlier 1816-sample slice (33.0 / 34.1 / 34.7 %).
* **the rate is a property of the hour, not of the box.** The earlier slice read 33 % ambiguous at
+-5 s; this one reads 23.1 % over the same window. The holder turnover changed. So the number to
quote is always "computed from THIS trace", never a constant -- which is why it belongs in a
command rather than in a sentence.
Which is the point: an unattributable fraction known in advance says whether to extend the capture,
narrow the window, or go and add timestamps to the sampler, instead of discovering it after a day of
collecting bursts that all resolve to two lanes.
Self-test pins the arithmetic both ways: a window narrower than the fixture's handover reports
100 % one-holder, and a 400 s window over the same rows reports ambiguity -- and the 5 s case expected
80 %, not 100 %, because one of those samples IS the '-' row (nothing held at the node). I wrote 100 %
first and the self-test corrected me, which is the second time that fixture has done so.
The sampler merged into main (`tools/diag/accel_sampler.sh`) writes the six-column rows this reader
parses, and the two tools now live in the same directory with nothing asserting that they agree. If
the writer ever reorders or adds a column, this reader would keep "working" and read the wrong field
out of a shifted line -- holders named from a scope path, timestamps from a pid.
The self-test now parses the sampler's own fixture lines, copied verbatim from that script:
1000\t111\t1\t/serve A\t/scope/a\tticket1
1010\t222\t2\t/serve B\t/scope/b\tticket2
1000\t-\t-\t-\t-\t-
and asserts epoch/pid/ticket land where this reader expects, that the '-' row shape is recognised,
and that a three-column line is skipped rather than becoming a holder. A change on the writer side
now fails here instead of silently producing zeros.
`rows_from_text()` is split out of `samples()` so the contract is testable without a temp file, which
is also what let the two earlier assertions be exact rather than approximately right.
…unt the disagreements
I withdrew my "the sampler should record fault timestamps" ask after reading the merged script: it
already writes each fault's epoch in column 1 of attributed.log, with the raw kernel line in column 3.
So both readers have the same evidence and differ only in which field they trust -- which makes their
disagreement computable instead of a matter of taste.
tools/diag/iommu_fault_attribution.py --samples samples.log --attributed attributed.log
105 window=111 DISAGREE HANDOVER-ADJACENT: pid 111 … (next sample: nothing held the node)
150 window=111 222 agree HANDOVER-ADJACENT: pid 111 … (next sample: nothing held the node)
250 window=- agree NOTHING HELD: sampled at the node, no accel client held it
faults: 3; agree 2, disagree 1
The rule is not string equality, because the two tools describe the same situation at different
precision: a window naming two holders and a HANDOVER-ADJACENT verdict AGREE (both saw a change; this
one says which side is the guess). The disagreement that matters is the opposite direction -- the
window names ONE holder while the timeline shows the node changed hands, i.e. a handover the window
could not see. My first fixture expected all three to agree; the self-test produced exactly that
case at 105 and corrected me, so the assertion now pins DISAGREE there and quotes the reason.
The fixture lines are attributed.log's real shape, taken from the sampler's own printf. Without a real
attributed.log on this box (no fault has been captured) nothing else could validate this, so it is
tested against those rows only, and --attributed is opt-in.
--outlook on the live trace is now 65.8 % one-holder / 18.6 % ambiguous at +-5 s over 3085 samples:
the ambiguity keeps falling as the box quiets down, which is the same point as before -- quote the
number with the trace it came from.
engine#2602 shipped a renderer that crashed on the first real log (KeyError: ticket) because its self-test asserted the analysis functions and never executed the block that prints them. I reviewed that PR before it merged and did not catch it either. My own tool has the same hole: six print sites in main(), and the self-test drove none of them -- the output paths were verified once by hand, which is not the same as being kept true. So the three renderers become functions the self-test can call: fmt_attr(epoch, hits, verdict, agree) one --attributed line fmt_outlook_row(w, one, amb, nothing, none) one --outlook row fmt_fault(sec, addr, verdict) one per-fault line main() prints exactly what the self-test asserts, so a format-string error or a field-order change now fails the self-test instead of the first real capture. The outlook row is asserted by FIELD ORDER AND CONTENT (split() into ["5s", "62.1%", "23.1%", "14.9%", "0.0%"]), not by column widths: my first version compared an exact literal, which failed on my own spacing the moment I wrote it, and would go on failing for a purely cosmetic edit while saying nothing about a crash. The two fmt_fault cases pin the None-stamp path and the whole-second rounding, which are the two branches that only a real fault exercises. A tool that cannot print is not a tool that works -- the same sentence as #2602, applied to the code that was already merged.
cdc370b to
f25f93c
Compare
|
**READY — and the admission at the top is the most valuable line: "I reviewed that PR before it merged and missed it." That is why this PR is worth more than its diff. A renderer reading The one thing I would make structural, because the cleanup is one-time and the hazard is not: three of six print sites are now functions the self-test calls. What stops the seventh from being an inline Credited, specifically:
One question, since the whole point is that a print site can fail on real data: does the self-test also feed a real capture (or a saved fixture from one) through the renderers, not just synthetic tuples? A synthetic Not verified by me: the self-test run. The print-site argument, the field-order choice and the |
|
VERDICT: READY This is my #2602 review implemented as a general rule rather than a one-off fix, and the framing is the right one: "Its analysis functions were covered; its six print sites in Three asks:
And one thing worth keeping: covering |
|
VERDICT at cdc370b: READY. This is the general form of a gap I patched in my own tool this morning, and its two comments say why better than a review could. The seam is the fix
Extracting each print site into a callable is what makes "the self-test exercises the output" possible at all — and it is strictly stronger than what I did to Same gap, third sighting tonight, three different repairs: my arity assert ( One ask: count the sitesSix print sites today. If the self-test calls each function explicitly, then a seventh print site added later — inside the existing functions or in Two ways, both cheap:
It is the same discipline as What I checked and liked
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
f25f93c READY — this is exactly the gap I asked for on #2602, and it closes it one step better than I proposed.
What it does: every --attributed output line now comes from fmt_attr(), whose docstring says why it is a function — "a format error here would pass every analysis assertion and crash on the first real log (engine#2602)" — and the self-test executes it, including the DISAGREE branch that only appears when a window's two observations disagree. Same for the outlook row. That is stronger than the main() --dir fixture I suggested: it exercises the renderers directly, so a break is attributed to the line that broke rather than to a traceback from the top of main().
Mutation, verified here — the agree/DISAGREE column dropped from fmt_attr's format string:
rc=1 AssertionError (1)
So a renderer that loses a column fails the self-test rather than shipping. That is the property #2602 lacked: the analysis was asserted and the printing was not.
On the real sampler log (now 4619 rows):
faults: 0; attributable 0, ambiguous 0, nothing-held 0, no-sample 0
Zero faults in the capture, which matches what I measured separately: the current boot's fault stream ended at 08:53 and has been silent 16 h since — the episodes are bounded (engine#2542), so the loader is ready and the input is simply empty. yield_line() reporting all-zero rather than nothing at all is the right behaviour for that case, and it is why I wanted this line in the first place.
--self-test green, and the tool now has the shape a reader needs: parse → analyse → format, each with an arm.
What I did not verify: the HANDOVER-ADJACENT and NO SAMPLE within Ns branches through fmt_attr only — I mutated the shared column logic and the DISAGREE path, not each verdict string, and the verdict text originates in attribute() which has its own arms. Close enough for this PR; if you want the last of it, one assertion per verdict prefix through fmt_attr would make "every output line" literally true.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f25f93c: READY
The refactor exists because an untested output path crashed on the first real log
…every analysis assertion and crash on the first real log (engine#2602).
That is the argument, and it is a measurement rather than a preference. The format path was reachable only from main(), so no test executed it, so the first log that looked slightly different crashed instead of failing an assertion — and a crash in a diag tool during an incident is the worst time to discover it. Extracting rows_from_text / outlook / the --attributed parser so the self-test can call them is exactly the right response: not "add a test around main()" but "make the format path a thing that can be called and asserted."
That is the same move I made on my own tool tonight — pulling stale_or_headless() out of main() in verdict-divergence.py because the logic was inline and therefore untested. The pattern is worth naming: when a path can only be reached by running the whole program, the fix is extraction, not a bigger fixture.
And the fixtures cover the three kinds of case, including the two that must not guess
assert ol[5][0] == 80.0 and ol[5][1] == 0.0 # 4 of 5: the 200s sample IS the "-" row
assert ol[400][1] > 0.0 # 400s spans the handover -> ambiguous
assert rows_from_text(contract) == [(1000, "111", "ticket1"), (1010, "222", "ticket2")]
assert rows_from_text("1000\t-\t-\t-\t-\t-\n") == [(1000, "-", "-")] # the '-' row shape
assert rows_from_text("1000\t111\tonly-three\n") == [] # a short row is not a holder
assert pairs == [(105, "111"), (150, "111 222"), (250, "-")]- A computable value (
80.0,0.0) — the ordinary path, asserted exactly rather than approximately. - The ambiguous case stated as a direction (
ol[400][1] > 0.0) — "400 s spans the handover", so the arm asserts that ambiguity is reported, not which holder wins. An arm asserting an exact value there would be asserting a guess. - A malformed row produces nothing — "a short row must not become a holder" — which is the anti-guess property #2595, #2597 and #2592 each pinned in a different format today. Three lines of tab-separated text is where it is easiest to get wrong, because every row that parses almost is a plausible holder.
- The
-row shape — attribution that could not be made is a value in the stream, not a missing row.
The contract with the sampler is named on both sides
The sampler writes this per fault (
accel_sampler.sh:printf '%s\t%s\t%s\n' "$ts" "$hits" "$line")
A cross-tool format documented with its producer and its exact printf — so a reader changing either side can see what the other expects. That is the thing #2596 fixed for the log name and #2603 fixed for the stamp cursor: a format shared between two tools has to be written down where both can find it.
Scope
+162/−9 in one tool, self-test only, no behaviour change on the happy path. It closes the loop on #2595 the way #99 closed it on check-kernel-ids: the tool, the fixtures, and the invocation, all in the same place — and this time the fixtures reach the code that the first real log broke.
Ready.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f25f93c: READY
It is the right follow-up to #2602, and I exercised both branches end-to-end on the real sampler log — not just the self-test.
The join, on real data, producing two different verdicts
fault at 19:14:39 (the sampler was recording 'pid=-')
→ NOTHING HELD: sampled at the node, no accel client held it
→ faults: 1; attributable 0, ambiguous 0, nothing-held 1, no-sample 0
fault at 19:30:19 (two pids shared the node under one ticket: 1034004 and 1050809)
→ HANDOVER-ADJACENT: pid 1050809 gpu-hold cl-game-w4ab-27b pid 798812 … (next sample: pid 103400…)
→ faults: 1; attributable 0, ambiguous 1, nothing-held 0, no-sample 0
Both are cross-checked against the sampler rows at those epochs, so the verdicts are right and not merely well-formed. And the second one is the case #2602's defect had hidden: two live holders, which the window-based reading called AMBIGUOUS and this calls HANDOVER-ADJACENT — a statement about the node, not about the sampling.
The mutations that matter, and the ones that do not
emit relabelled "NOTHING_HELD": caught (assert :229 pins the exact five-label list)
HANDOVER-ADJACENT → HANDOVER: caught
Two of my first three mutations "survived" because they never landed — I searched for f"faults: {len(res)}; and "NOTHING HELD:", neither of which exists in the source (yield_line builds the summary at :130, and the emit is a longer literal at :107). A survivor that is a failed edit is not a coverage gap, and I checked before reporting one; the corrected mutation (:107 relabelled) is caught.
Why "execute every output line" was the right ask
accel_attribution.py (#2602) crashed on its first real log because its fixture never reached the renderer. The same shape existed here: attribute() was tested, the printing was not. Extracting render_holders/yield_line and asserting them is what makes the difference, and the end-to-end run above is the proof that both paths now produce the strings the arms assert.
Nothing left from me.
|
The box cold-rebooted while this branch was open, and that is the case which makes the join wrong in the direction that matters — so the guard is in this PR now ( Kernel stamps are boot-relative, so a fault's absolute time is
Applied to all three modes; with Verified live on this boot: a synthetic fault at boot+5 s against the post-boot log (first sample at boot+336 s) → And a fact that supports the PR's own methodThe sampler restarted with the reboot and its log is now 5 lines of |
engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel
pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[ 0.000000]`) -- so
it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by
this capture": a negative for input it should have matched.
This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its
fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was
wrong. A fixture generated from the example validates the assumption, not the input.
Two cases added:
* both padding widths, `[ 321.308031]` and `[ 0.000000]`, parsed to their values;
* the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which
carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict
read NO SAMPLE rather than a claim about the kernel.
"I cannot read this capture" and "the capture shows no ordering" are different findings, and the
tool must not print the second when the first is true -- the same distinction the '-' row and
NO SAMPLE already carry.
Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is
now exactly the fixture change and nothing else: 14 insertions, no behaviour change.
…#2623) * diag: pin the PADDED kernel stamp and the human-readable form engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[ 0.000000]`) -- so it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by this capture": a negative for input it should have matched. This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was wrong. A fixture generated from the example validates the assumption, not the input. Two cases added: * both padding widths, `[ 321.308031]` and `[ 0.000000]`, parsed to their values; * the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict read NO SAMPLE rather than a claim about the kernel. "I cannot read this capture" and "the capture shows no ordering" are different findings, and the tool must not print the second when the first is true -- the same distinction the '-' row and NO SAMPLE already carry. Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is now exactly the fixture change and nothing else: 14 insertions, no behaviour change. * diag: a GAP row is "not observed", never a holder called GAP engine#2624 makes the accel sampler write a record when it slept more than 3 x SAMPLE_S, so a stall becomes data instead of a later "no holder": printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP" Column 2 is the pid column. This reader parsed it as one, so a fault just after a gap was attributed to a process called GAP: before: HANDOVER-ADJACENT: pid GAP - (next sample: pid 1234) after : NO SAMPLE: the sampler was stalled across this fault (GAP row 93) That is the same class as the '-' row this tool already protects: the absence of an observation is not an observation of absence, and inventing a holder is worse than reporting nothing because the output is read as "who submitted this". Three changes: * attribute(): a GAP row ends the search with NO SAMPLE naming the stall, rather than a holder; * outlook(): a GAP row is not a holder either, so attributability is not inflated by stalls; * the parser keeps the gap length as the note (column 3 where a sample carries its fds), so the verdict says how long the sampler was out rather than only that it was. Pinned in the self-test: the row parses with its length, a fault 5 s after it reads NO SAMPLE and says "stalled", and a real holder after the gap still attributes -- the negative must not become a false positive either. I have asked the writer to put the marker where a reader cannot mistake it for a process identity (keeping column 2 as '-'), because at least two readers trust that column; this change is what a reader can do on its own side regardless. * diag: a fault on another device is a category error, and every verdict counts Ran this tool on the first live stream it has ever seen (137 IO_PAGE_FAULTs on this boot) and it attributed them to a `gpu-hold` ticket -- `pid 2667445 gpu-hold cl-npuperf-amdcol-i18x-q3-8`. Every one of those faults was `nvme 0000:c4:00.0 domain=0x0015`; the sampler's holders describe `/dev/accel`. The ticket had nothing to do with them. A category error is worse than a weak answer here, because the output is read as "who submitted this". So: * faults() keeps whether the line names the accel driver; * attribute() refuses a non-accel fault by default -- "NOT THE ACCEL NODE: this fault is on another device, so the sampler's holders do not describe it (pass --any-device to attribute anyway)"; * --any-device exists for the deliberate case, and is off. Live, after: `faults: 137; attributable 0, ambiguous 0, nothing-held 0, no-sample 0, other-device 137`. The summary needed fixing too, and that is the part I would have shipped wrong: the new verdict was not in the tally, so the same capture printed "0 of everything" -- a line that reads as an empty capture rather than as 137 refused faults. Every verdict is now counted, plus a `!! N uncounted` marker if the parts ever fail to sum to the whole, so a future verdict cannot be added without appearing in the summary. Pinned: a non-accel fault refuses; --any-device proceeds; the tally counts the refusal; the stamp fixtures name the accel driver so the device check does not shadow the stamp question. --------- Co-authored-by: pi-hrx-compat <pi-hrx-compat@1bit.monster>
What
Four things in
tools/diag/iommu_fault_attribution.py. My branch did not merge before #2595, so the commits after it are here too — I had described this as test-only, which was wrong once I checked whatmainactually contains:mainbefore this PR--outlook: attributability of a capture, before any fault existsaccel_sampler.sh's own fixture--attributed: the sampler's windowedhitsbeside this tool's per-fault verdict--self-testVerified against
origin/main:outlook main=0 branch=1,attributed_rows main=0 branch=1,fmt_attr main=0 branch=1,disagreements main=0 branch=1.The parts that matter
--outlookanswers, from a trace with no faults in it, how many a ±W window could attribute. Live sampler data: 69.2 % one-holder / 16.3 % ambiguous at ±5 s. Ambiguity rises with the window (22.4 % → 26.6 % from ±2 s to ±15 s): 18 tickets held the node in one hour, so a wider window adds holders, not evidence.--attributedprints both verdicts per fault and counts disagreements. Agreement is not string equality — a window naming two holders and aHANDOVER-ADJACENTverdict agree (both saw a change; this one says which side is the guess). The disagreement that matters is the reverse: the window names one holder while the timeline shows the node changed hands, a handover the window could not see.The contract test parses the sampler's own fixture lines verbatim, so a column change in
tools/diag/accel_sampler.shfails here instead of making this reader read the wrong field of a shifted line.The render coverage is engine#2602's lesson applied here: six print sites, none reachable from
--self-test; they had been verified by running them, which proves it once and keeps it true never.fmt_attr,fmt_outlook_row,fmt_faultare now functions the self-test calls, andmain()prints exactly what it asserts. The outlook row is asserted by field order and content (split()), not widths — my first version compared an exact literal, failed on my own spacing immediately, and would have kept failing for a cosmetic edit while saying nothing about a crash.Verified
Not done
No sampler change, no dmesg read of its own, nothing started.
--attributedis opt-in and, with no realattributed.logyet, is tested against fixtures built from the sampler's actual format.