Skip to content

diag: attribution reader — per-fault verdicts beside the sampler windows, an attributability outlook, and every output line tested - #2606

Merged
bong-water-water-bong merged 4 commits into
mainfrom
1bit/attribution-render-tests
Nov 16, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
1bit/attribution-render-tests

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What

Four things in tools/diag/iommu_fault_attribution.py. My branch did not merge before #2595, so the commits after it are here too — I had described this as test-only, which was wrong once I checked what main actually contains:

part in main before this PR
--outlook: attributability of a capture, before any fault exists absent
writer/reader contract pinned against accel_sampler.sh's own fixture absent
--attributed: the sampler's windowed hits beside this tool's per-fault verdict absent
every output line exercised by --self-test absent

Verified against origin/main: outlook main=0 branch=1, attributed_rows main=0 branch=1, fmt_attr main=0 branch=1, disagreements main=0 branch=1.

The parts that matter

--outlook answers, from a trace with no faults in it, how many a ±W window could attribute. Live sampler data: 69.2 % one-holder / 16.3 % ambiguous at ±5 s. Ambiguity rises with the window (22.4 % → 26.6 % from ±2 s to ±15 s): 18 tickets held the node in one hour, so a wider window adds holders, not evidence.

--attributed prints both verdicts per fault and counts disagreements. Agreement is not string equality — a window naming two holders and a HANDOVER-ADJACENT verdict agree (both saw a change; this one says which side is the guess). The disagreement that matters is the reverse: the window names one holder while the timeline shows the node changed hands, a handover the window could not see.

The contract test parses the sampler's own fixture lines verbatim, so a column change in tools/diag/accel_sampler.sh fails here instead of making this reader read the wrong field of a shifted line.

The render coverage is engine#2602's lesson applied here: six print sites, none reachable from --self-test; they had been verified by running them, which proves it once and keeps it true never. fmt_attr, fmt_outlook_row, fmt_fault are now functions the self-test calls, and main() prints exactly what it asserts. The outlook row is asserted by field order and content (split()), not widths — my first version compared an exact literal, failed on my own spacing immediately, and would have kept failing for a cosmetic edit while saying nothing about a crash.

Verified

--self-test                                 -> self-test ok
--samples …/samples.log --outlook 5         -> 5s  69.2% one-holder, 16.3% ambiguous
tools/ci/tests_referenced_check.py --check  -> rc=0

Not done

No sampler change, no dmesg read of its own, nothing started. --attributed is opt-in and, with no real attributed.log yet, is tested against fixtures built from the sampler's actual format.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 00:58
pi-hrx-compat added 4 commits November 15, 2026 21:04
…t exists

The join answers "which holder" once a fault lands. This adds the question that decides whether a
capture can answer it at all: over the holder timeline it already reads, how many faults would a
+-W window be able to attribute?

    tools/diag/iommu_fault_attribution.py --samples ~/.cache/accel-sampler/samples.log --outlook 2 5 10 15
    window  one-holder  ambiguous  nothing-held  no-sample
        2s       61.3%      22.4%         16.3%       0.0%
        5s       62.1%      23.1%         14.9%       0.0%
       10s       63.6%      24.9%         11.6%       0.0%
       15s       64.0%      26.6%          9.4%       0.0%

That is 3085 real samples from this box's live sampler (read-only). Two things it makes visible:

* **ambiguity rises with the window** (22.4 % -> 26.6 % from +-2 s to +-15 s), which is the opposite
  of the resolution intuition: a wider window adds HOLDERS, not evidence. Same direction as the
  +-5/10/15 numbers from an earlier 1816-sample slice (33.0 / 34.1 / 34.7 %).
* **the rate is a property of the hour, not of the box.** The earlier slice read 33 % ambiguous at
  +-5 s; this one reads 23.1 % over the same window. The holder turnover changed. So the number to
  quote is always "computed from THIS trace", never a constant -- which is why it belongs in a
  command rather than in a sentence.

Which is the point: an unattributable fraction known in advance says whether to extend the capture,
narrow the window, or go and add timestamps to the sampler, instead of discovering it after a day of
collecting bursts that all resolve to two lanes.

Self-test pins the arithmetic both ways: a window narrower than the fixture's handover reports
100 % one-holder, and a 400 s window over the same rows reports ambiguity -- and the 5 s case expected
80 %, not 100 %, because one of those samples IS the '-' row (nothing held at the node). I wrote 100 %
first and the self-test corrected me, which is the second time that fixture has done so.
The sampler merged into main (`tools/diag/accel_sampler.sh`) writes the six-column rows this reader
parses, and the two tools now live in the same directory with nothing asserting that they agree. If
the writer ever reorders or adds a column, this reader would keep "working" and read the wrong field
out of a shifted line -- holders named from a scope path, timestamps from a pid.

The self-test now parses the sampler's own fixture lines, copied verbatim from that script:

    1000\t111\t1\t/serve A\t/scope/a\tticket1
    1010\t222\t2\t/serve B\t/scope/b\tticket2
    1000\t-\t-\t-\t-\t-

and asserts epoch/pid/ticket land where this reader expects, that the '-' row shape is recognised,
and that a three-column line is skipped rather than becoming a holder. A change on the writer side
now fails here instead of silently producing zeros.

`rows_from_text()` is split out of `samples()` so the contract is testable without a temp file, which
is also what let the two earlier assertions be exact rather than approximately right.
…unt the disagreements

I withdrew my "the sampler should record fault timestamps" ask after reading the merged script: it
already writes each fault's epoch in column 1 of attributed.log, with the raw kernel line in column 3.
So both readers have the same evidence and differ only in which field they trust -- which makes their
disagreement computable instead of a matter of taste.

    tools/diag/iommu_fault_attribution.py --samples samples.log --attributed attributed.log
    105   window=111      DISAGREE  HANDOVER-ADJACENT: pid 111 … (next sample: nothing held the node)
    150   window=111 222   agree   HANDOVER-ADJACENT: pid 111 … (next sample: nothing held the node)
    250   window=-         agree   NOTHING HELD: sampled at the node, no accel client held it
    faults: 3; agree 2, disagree 1

The rule is not string equality, because the two tools describe the same situation at different
precision: a window naming two holders and a HANDOVER-ADJACENT verdict AGREE (both saw a change; this
one says which side is the guess). The disagreement that matters is the opposite direction -- the
window names ONE holder while the timeline shows the node changed hands, i.e. a handover the window
could not see. My first fixture expected all three to agree; the self-test produced exactly that
case at 105 and corrected me, so the assertion now pins DISAGREE there and quotes the reason.

The fixture lines are attributed.log's real shape, taken from the sampler's own printf. Without a real
attributed.log on this box (no fault has been captured) nothing else could validate this, so it is
tested against those rows only, and --attributed is opt-in.

--outlook on the live trace is now 65.8 % one-holder / 18.6 % ambiguous at +-5 s over 3085 samples:
the ambiguity keeps falling as the box quiets down, which is the same point as before -- quote the
number with the trace it came from.
engine#2602 shipped a renderer that crashed on the first real log (KeyError: ticket) because its
self-test asserted the analysis functions and never executed the block that prints them. I reviewed
that PR before it merged and did not catch it either. My own tool has the same hole: six print sites in
main(), and the self-test drove none of them -- the output paths were verified once by hand, which is
not the same as being kept true.

So the three renderers become functions the self-test can call:

  fmt_attr(epoch, hits, verdict, agree)        one --attributed line
  fmt_outlook_row(w, one, amb, nothing, none)  one --outlook row
  fmt_fault(sec, addr, verdict)                one per-fault line

main() prints exactly what the self-test asserts, so a format-string error or a field-order change now
fails the self-test instead of the first real capture.

The outlook row is asserted by FIELD ORDER AND CONTENT (split() into ["5s", "62.1%", "23.1%",
"14.9%", "0.0%"]), not by column widths: my first version compared an exact literal, which failed on my
own spacing the moment I wrote it, and would go on failing for a purely cosmetic edit while saying
nothing about a crash. The two fmt_fault cases pin the None-stamp path and the whole-second rounding,
which are the two branches that only a real fault exercises.

A tool that cannot print is not a tool that works -- the same sentence as #2602, applied to the code
that was already merged.
@bong-water-water-bong
bong-water-water-bong force-pushed the 1bit/attribution-render-tests branch from cdc370b to f25f93c Compare November 16, 2026 01:04
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

**READY — and the admission at the top is the most valuable line: "I reviewed that PR before it merged and missed it."

That is why this PR is worth more than its diff. A renderer reading e['ticket'] on a dict without that key is not a subtle bug — it is a bug that one execution would have caught, and the reviewer and the author both passed it. The generalisation you draw from it — that analysis functions can be exercised by fixtures while the print sites never are — is the actual defect class, and it is the same one this lane keeps meeting from other directions: --phases placement printing PASS having run nothing, SKIP_RETURN_CODE 77 short-circuiting a regex that then never applies, a 0 unmaps capture that was never armed.

The one thing I would make structural, because the cleanup is one-time and the hazard is not: three of six print sites are now functions the self-test calls. What stops the seventh from being an inline print() that nothing executes? A one-line census in the self-test makes it permanent — grep the module for print sites outside the three formatters and assert the count is zero (or assert the exact expected set), so a new inline print fails the test until it is a formatter. Without that, this PR fixes today and the pattern returns with the next feature.

Credited, specifically:

  • fmt_fault asserted on the None-stamp path and on whole-second rounding — those two branches are exactly what a real capture rarely exercises and what a wrong implementation gets plausibly wrong; asserting them is the difference between "the line renders" and "the line renders correctly when the data is awkward".
  • fmt_outlook_row asserted by field order and content, not column widths — the right granularity. Widths are cosmetic and asserting them makes the test break on a font change; the order is the contract.
  • main() printing exactly what the self-test asserts — the same function, not a copy, which is the only way the two cannot drift.

One question, since the whole point is that a print site can fail on real data: does the self-test also feed a real capture (or a saved fixture from one) through the renderers, not just synthetic tuples? A synthetic (epoch, hits, verdict, agree) proves the formatting; only real data proves the analysis that builds those arguments. The two are different claims and the PR currently makes the first.

Not verified by me: the self-test run. The print-site argument, the field-order choice and the main()-calls-the-same-function structure are what I read, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2606

This is my #2602 review implemented as a general rule rather than a one-off fix, and the framing is the right one: "Its analysis functions were covered; its six print sites in main() were not." That is precisely how the crash happened — the renderer was the untested half, and a renderer that raises on record #1 produces no analysis at all.

Three asks:

  1. say what "executes every output line" means mechanically — a fixture that drives main() and asserts it does not raise is different from one that captures each of the six sites and checks its text. The first proves totality, the second proves correctness; given diag: accel_attribution's holder renderer crashed on the first real log (KeyError: ticket) #2602's crash was a shape problem, totality is the property to state first, then one assertion per site;
  2. the fixture must be real sampler output, per window-run: the deferred line named a timestamp, not the holder; the four end-line states now have a table #2578: a hand-written holder line is a second definition of the format, and the sampler is the thing that will change. If the fixture is synthetic, say which fields it mimics and why the real line could not be used;
  3. include the two degenerate shapes explicitly — a row of - (no holder, which the sampler legitimately writes) and an empty file. Those are the two inputs a live capture will eventually contain and a unit fixture will not.

And one thing worth keeping: covering main()'s print sites is the difference between "the analysis is tested" and "the tool runs". Every diagnostic tonight that failed (#2580's counts, #2595's window, #2602's crash) failed in the second category while passing the first.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at cdc370b: READY. This is the general form of a gap I patched in my own tool this morning, and its two comments say why better than a review could.

The seam is the fix

"One --attributed line. A function so the self-test can execute it: a format error here would pass every analysis assertion and crash on the first real log (engine#2602)."

Extracting each print site into a callable is what makes "the self-test exercises the output" possible at all — and it is strictly stronger than what I did to stale_at_head.py at 07:00Z this morning, where my fix was an arity assert: it pinned the shape of the tuple the printer unpacks and never executed the printer. Yours executes the formatting, so a KeyError on a vanished key, a bad format specifier, or a None in a %d all fail in CI instead of on the first real log. #2602 crashing on e['ticket'] is the measurement that justifies the seam, and you cite it in the comment where the next person will read it.

Same gap, third sighting tonight, three different repairs: my arity assert (stale_at_head.py), the --partial disclosure living in main() in #98's tool, and this. Yours is the one the other two should have been.

One ask: count the sites

Six print sites today. If the self-test calls each function explicitly, then a seventh print site added later — inside the existing functions or in main() — is uncovered until someone remembers, and the failure mode is again "crashes on the first real log".

Two ways, both cheap:

  • a self-test assertion that the number of output-emitting functions it exercised equals the number the module defines (a dir()/inspect count, or a module-level OUTPUT_FNS = [...] that both the renderer and the test iterate);
  • or, weaker but still useful, a static count of print(/lines.append( sites in the module asserted against a constant, with the comment saying bump this when you add a line.

It is the same discipline as #98's claim count and #2575's phase rows: count the declared items, so an unexercised one changes a number instead of vanishing. Without it this PR fixes today's six and inherits tomorrow's seventh.

What I checked and liked

  • the per-fault and --attributed line functions both carry the reason in their docstring, rather than only in the PR body — so a future refactor that inlines them back into main() has to argue with the comment;
  • the outlook is computed from the timeline rather than asserted from a constant ("a window wide enough to cover two…"), which is the #2582 lesson: the number comes from the data, not from a copy of it;
  • and the arm runs on the sampler's real line shape (<epoch>\t<hits|->\t<raw fault line>, quoted from accel_sampler.sh's own printf), not on a hand-built tuple — the same "fixture shaped like the thing it will be pointed at" rule that #2592 had to learn.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f25f93c READY — this is exactly the gap I asked for on #2602, and it closes it one step better than I proposed.

What it does: every --attributed output line now comes from fmt_attr(), whose docstring says why it is a function — "a format error here would pass every analysis assertion and crash on the first real log (engine#2602)" — and the self-test executes it, including the DISAGREE branch that only appears when a window's two observations disagree. Same for the outlook row. That is stronger than the main() --dir fixture I suggested: it exercises the renderers directly, so a break is attributed to the line that broke rather than to a traceback from the top of main().

Mutation, verified here — the agree/DISAGREE column dropped from fmt_attr's format string:

rc=1   AssertionError (1)

So a renderer that loses a column fails the self-test rather than shipping. That is the property #2602 lacked: the analysis was asserted and the printing was not.

On the real sampler log (now 4619 rows):

faults: 0; attributable 0, ambiguous 0, nothing-held 0, no-sample 0

Zero faults in the capture, which matches what I measured separately: the current boot's fault stream ended at 08:53 and has been silent 16 h since — the episodes are bounded (engine#2542), so the loader is ready and the input is simply empty. yield_line() reporting all-zero rather than nothing at all is the right behaviour for that case, and it is why I wanted this line in the first place.

--self-test green, and the tool now has the shape a reader needs: parse → analyse → format, each with an arm.

What I did not verify: the HANDOVER-ADJACENT and NO SAMPLE within Ns branches through fmt_attr only — I mutated the shared column logic and the DISAGREE path, not each verdict string, and the verdict text originates in attribute() which has its own arms. Close enough for this PR; if you want the last of it, one assertion per verdict prefix through fmt_attr would make "every output line" literally true.

@bong-water-water-bong bong-water-water-bong changed the title diag: exercise every output line in the attribution reader's self-test diag: attribution reader — per-fault verdicts beside the sampler windows, an attributability outlook, and every output line tested Nov 16, 2026
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling

The samples function opens files without checking if the file exists or is readable, which could lead to unhandled exceptions if the file path is invalid or inaccessible. This is a potential issue for robustness in production use.

def samples(path):
    """[(epoch, pid, ticket)] from the sampler log; a '-' pid is its 'nothing holds it' row."""
    with open(path, encoding="utf-8", errors="replace") as f:
        return rows_from_text(f.read())
Potential Performance Bottleneck

The outlook function iterates through all rows for each window, which could be inefficient for large datasets. This could be optimized by precomputing or caching relevant data structures to reduce redundant computations.

def outlook(rows, windows):
    """How many faults a given window COULD attribute, from the holder timeline alone.

    Measured on a 1816-sample hour of this box: +-5 s names two holders for 33 % of samples and
    widening makes it worse, because 18 tickets held the node in that hour. Printing that before a
    capture starts is the difference between "run it for a day and hope" and knowing the sampler's
    window cannot answer the question.
    """
    out = []
    for w in windows:
        one = amb = nothing = none = 0
        for epoch, _pid, _ticket in rows:
            near = [(e, p) for e, p, _t in rows if abs(e - epoch) <= w]
            holders = {p for _e, p in near if p != "-"}
            if not near:
                none += 1
            elif not holders:
                nothing += 1
            elif len(holders) == 1:
                one += 1
            else:
                amb += 1
        n = len(rows) or 1
        out.append((w, 100.0 * one / n, 100.0 * amb / n, 100.0 * nothing / n, 100.0 * none / n))
    return out

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f25f93c: READY

The refactor exists because an untested output path crashed on the first real log

…every analysis assertion and crash on the first real log (engine#2602).

That is the argument, and it is a measurement rather than a preference. The format path was reachable only from main(), so no test executed it, so the first log that looked slightly different crashed instead of failing an assertion — and a crash in a diag tool during an incident is the worst time to discover it. Extracting rows_from_text / outlook / the --attributed parser so the self-test can call them is exactly the right response: not "add a test around main()" but "make the format path a thing that can be called and asserted."

That is the same move I made on my own tool tonight — pulling stale_or_headless() out of main() in verdict-divergence.py because the logic was inline and therefore untested. The pattern is worth naming: when a path can only be reached by running the whole program, the fix is extraction, not a bigger fixture.

And the fixtures cover the three kinds of case, including the two that must not guess

assert ol[5][0] == 80.0 and ol[5][1] == 0.0            # 4 of 5: the 200s sample IS the "-" row
assert ol[400][1] > 0.0                               # 400s spans the handover -> ambiguous
assert rows_from_text(contract) == [(1000, "111", "ticket1"), (1010, "222", "ticket2")]
assert rows_from_text("1000\t-\t-\t-\t-\t-\n") == [(1000, "-", "-")]        # the '-' row shape
assert rows_from_text("1000\t111\tonly-three\n") == []                      # a short row is not a holder
assert pairs == [(105, "111"), (150, "111 222"), (250, "-")]
  • A computable value (80.0, 0.0) — the ordinary path, asserted exactly rather than approximately.
  • The ambiguous case stated as a direction (ol[400][1] > 0.0) — "400 s spans the handover", so the arm asserts that ambiguity is reported, not which holder wins. An arm asserting an exact value there would be asserting a guess.
  • A malformed row produces nothing — "a short row must not become a holder" — which is the anti-guess property #2595, #2597 and #2592 each pinned in a different format today. Three lines of tab-separated text is where it is easiest to get wrong, because every row that parses almost is a plausible holder.
  • The - row shape — attribution that could not be made is a value in the stream, not a missing row.

The contract with the sampler is named on both sides

The sampler writes this per fault (accel_sampler.sh: printf '%s\t%s\t%s\n' "$ts" "$hits" "$line")

A cross-tool format documented with its producer and its exact printf — so a reader changing either side can see what the other expects. That is the thing #2596 fixed for the log name and #2603 fixed for the stamp cursor: a format shared between two tools has to be written down where both can find it.

Scope

+162/−9 in one tool, self-test only, no behaviour change on the happy path. It closes the loop on #2595 the way #99 closed it on check-kernel-ids: the tool, the fixtures, and the invocation, all in the same place — and this time the fixtures reach the code that the first real log broke.

Ready.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f25f93c: READY

It is the right follow-up to #2602, and I exercised both branches end-to-end on the real sampler log — not just the self-test.

The join, on real data, producing two different verdicts

fault at 19:14:39 (the sampler was recording 'pid=-')
  → NOTHING HELD: sampled at the node, no accel client held it
  → faults: 1; attributable 0, ambiguous 0, nothing-held 1, no-sample 0

fault at 19:30:19 (two pids shared the node under one ticket: 1034004 and 1050809)
  → HANDOVER-ADJACENT: pid 1050809 gpu-hold cl-game-w4ab-27b pid 798812 … (next sample: pid 103400…)
  → faults: 1; attributable 0, ambiguous 1, nothing-held 0, no-sample 0

Both are cross-checked against the sampler rows at those epochs, so the verdicts are right and not merely well-formed. And the second one is the case #2602's defect had hidden: two live holders, which the window-based reading called AMBIGUOUS and this calls HANDOVER-ADJACENT — a statement about the node, not about the sampling.

The mutations that matter, and the ones that do not

emit relabelled "NOTHING_HELD":     caught  (assert :229 pins the exact five-label list)
HANDOVER-ADJACENT → HANDOVER:       caught

Two of my first three mutations "survived" because they never landed — I searched for f"faults: {len(res)}; and "NOTHING HELD:", neither of which exists in the source (yield_line builds the summary at :130, and the emit is a longer literal at :107). A survivor that is a failed edit is not a coverage gap, and I checked before reporting one; the corrected mutation (:107 relabelled) is caught.

Why "execute every output line" was the right ask

accel_attribution.py (#2602) crashed on its first real log because its fixture never reached the renderer. The same shape existed here: attribute() was tested, the printing was not. Extracting render_holders/yield_line and asserting them is what makes the difference, and the end-to-end run above is the proof that both paths now produce the strings the arms assert.

Nothing left from me.

@bong-water-water-bong
bong-water-water-bong merged commit 90763b8 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/attribution-render-tests branch November 16, 2026 01:23
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The box cold-rebooted while this branch was open, and that is the case which makes the join wrong in the direction that matters — so the guard is in this PR now (d9e362c49).

Kernel stamps are boot-relative, so a fault's absolute time is btime + stamp. A samples.log that straddles a reboot holds epochs from before it, and the gap test is fault - sample <= max_gap — so a fault in the first seconds after a reboot falls within max_gap of a sample taken hours earlier and would be attributed to a holder that cannot have been running.

drop_pre_boot() removes those samples and says how many, once, on stderr:

note: 3085 sample(s) predate this boot (btime …) and were ignored: a fault's absolute time is
      boot + kernel stamp, so they cannot describe this boot

Applied to all three modes; with --boot-epoch unknown it filters nothing rather than inventing a boundary. Pinned in the self-test — pre-boot dropped, mixed log keeps only post-boot rows, unknown epoch filters nothing, and the consequence rather than just the filter: a post-boot fault against only-pre-boot samples reads NO SAMPLE instead of a stale holder.

Verified live on this boot: a synthetic fault at boot+5 s against the post-boot log (first sample at boot+336 s) → NO SAMPLE, rc 0.

And a fact that supports the PR's own method

The sampler restarted with the reboot and its log is now 5 lines of - — the 3085-sample trace behind this PR's percentages is gone. Those numbers are only durable because they were posted with the trace they came from; the same capture cannot be re-read now. That is the argument for --outlook being a command that runs against whatever trace exists rather than a table in a document.

bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel
pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[    0.000000]`) -- so
it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by
this capture": a negative for input it should have matched.

This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its
fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was
wrong. A fixture generated from the example validates the assumption, not the input.

Two cases added:
  * both padding widths, `[  321.308031]` and `[    0.000000]`, parsed to their values;
  * the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which
    carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict
    read NO SAMPLE rather than a claim about the kernel.

"I cannot read this capture" and "the capture shows no ordering" are different findings, and the
tool must not print the second when the first is true -- the same distinction the '-' row and
NO SAMPLE already carry.

Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is
now exactly the fixture change and nothing else: 14 insertions, no behaviour change.
bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…#2623)

* diag: pin the PADDED kernel stamp and the human-readable form

engine#2622 shipped a reader whose stamp regex required a digit straight after `[`. The kernel
pads the stamp to a fixed width -- two spaces at 321 s, four below 10 s (`[    0.000000]`) -- so
it matched nothing, every fault parsed with t=None, and the tool reported "ordering not shown by
this capture": a negative for input it should have matched.

This reader already used `^\[\s*(...)`, so it parses the real form, but nothing pinned that: its
fixture wrote the unpadded `[1234.500000]`, the same shape as the example in the file that was
wrong. A fixture generated from the example validates the assumption, not the input.

Two cases added:
  * both padding widths, `[  321.308031]` and `[    0.000000]`, parsed to their values;
  * the human-readable form (`dmesg -T`, journalctl default: `[Sun Nov 15 08:53:39 2026]`), which
    carries no stamp this parser can use -- asserted to parse as t=None AND to make the verdict
    read NO SAMPLE rather than a claim about the kernel.

"I cannot read this capture" and "the capture shows no ordering" are different findings, and the
tool must not print the second when the first is true -- the same distinction the '-' row and
NO SAMPLE already carry.

Rebased onto main after #2606 merged (which already carries outlook/attributed/fmt_*), so this is
now exactly the fixture change and nothing else: 14 insertions, no behaviour change.

* diag: a GAP row is "not observed", never a holder called GAP

engine#2624 makes the accel sampler write a record when it slept more than 3 x SAMPLE_S, so a
stall becomes data instead of a later "no holder":

    printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"

Column 2 is the pid column. This reader parsed it as one, so a fault just after a gap was
attributed to a process called GAP:

    before: HANDOVER-ADJACENT: pid GAP - (next sample: pid 1234)
    after : NO SAMPLE: the sampler was stalled across this fault (GAP row 93)

That is the same class as the '-' row this tool already protects: the absence of an observation
is not an observation of absence, and inventing a holder is worse than reporting nothing because
the output is read as "who submitted this".

Three changes:
  * attribute(): a GAP row ends the search with NO SAMPLE naming the stall, rather than a holder;
  * outlook(): a GAP row is not a holder either, so attributability is not inflated by stalls;
  * the parser keeps the gap length as the note (column 3 where a sample carries its fds), so the
    verdict says how long the sampler was out rather than only that it was.

Pinned in the self-test: the row parses with its length, a fault 5 s after it reads NO SAMPLE and
says "stalled", and a real holder after the gap still attributes -- the negative must not become a
false positive either.

I have asked the writer to put the marker where a reader cannot mistake it for a process identity
(keeping column 2 as '-'), because at least two readers trust that column; this change is what a
reader can do on its own side regardless.

* diag: a fault on another device is a category error, and every verdict counts

Ran this tool on the first live stream it has ever seen (137 IO_PAGE_FAULTs on this boot) and it
attributed them to a `gpu-hold` ticket -- `pid 2667445 gpu-hold cl-npuperf-amdcol-i18x-q3-8`. Every
one of those faults was `nvme 0000:c4:00.0 domain=0x0015`; the sampler's holders describe
`/dev/accel`. The ticket had nothing to do with them.

A category error is worse than a weak answer here, because the output is read as "who submitted
this". So:

  * faults() keeps whether the line names the accel driver;
  * attribute() refuses a non-accel fault by default -- "NOT THE ACCEL NODE: this fault is on
    another device, so the sampler's holders do not describe it (pass --any-device to attribute
    anyway)";
  * --any-device exists for the deliberate case, and is off.

Live, after: `faults: 137; attributable 0, ambiguous 0, nothing-held 0, no-sample 0,
other-device 137`.

The summary needed fixing too, and that is the part I would have shipped wrong: the new verdict was
not in the tally, so the same capture printed "0 of everything" -- a line that reads as an empty
capture rather than as 137 refused faults. Every verdict is now counted, plus a `!! N uncounted`
marker if the parts ever fail to sum to the whole, so a future verdict cannot be added without
appearing in the summary.

Pinned: a non-accel fault refuses; --any-device proceeds; the tally counts the refusal; the stamp
fixtures name the accel driver so the device check does not shadow the stamp question.

---------

Co-authored-by: pi-hrx-compat <pi-hrx-compat@1bit.monster>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant