Skip to content

diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) - #2589

Merged
bong-water-water-bong merged 2 commits into
mainfrom
lane/accel-attribution-analysis
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
lane/accel-attribution-analysis

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Context

ORCH split on engine#2542: pi-pkg runs the owner-approved 24 h /dev/accel sampler (attributes each IO_PAGE_FAULT burst to the PID/ticket holding accel0); I do the analysis side. This is the reader for ~/.cache/accel-sampler/.

Two properties of the raw logs decide how much a burst can be attributed

1. The hits field cannot separate two holders. watch_faults writes the samples in ±WINDOW_S as sort -u of pid,cmd,scope,ticket — no timestamps, no order. A window spanning a holder change names both and cannot say which was live at the fault. The epochs are in samples.log, so this tool rebuilds the timeline and prints AMBIGUOUS: 2 holders inside +-5s (111, 222) rather than picking one.

2. - conflates two different facts. In attributed.log it means no row in the window; the sampler writes an explicit - row for sampled, nothing holds the node. Collapsing them turns "I did not look" into "nobody was there":

fault C -> sampled 1x, nothing held the node (explicit '-' rows): not an accel client
fault D -> NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')

3. The effective interval is not SAMPLE_S. Scanning every /proc/<pid>/fd takes seconds, so the configured 2 s measures 6–9 s on this box — a ±5 s window holds one or two samples. The tool reports the median/min/max, flags gaps > 3× median (a restart), and says so when the window is shorter than the interval.

First run, live logs

samples: 16 rows over 102 s; effective interval median 6 s, min 6, max 9
gaps > 3x median: 0
  note: +-5s window is shorter than the effective interval (6s): some faults will be sampled zero times
'nothing holds the node' rows: 2 of 16

holders seen (pid, first..last, samples, ticket):
  29370     1794780863..1794780911 (   48s) n=8    .../cl-npuperf-amdcol/build/1bit s  gpu-hold cl-npuperf-cmpl-h...
  3625096   1794780809..1794780845 (   36s) n=6    .../cl-npuperf-amdcol/build/1bit s  gpu-hold cl-npuperf-amdcol...

attributions: 0        (no fault since the sampler started)

Verified

  • --self-test green, and the window arithmetic is hand-checked: the same samples flip one fault from unambiguous at ±3 s to ambiguous at ±5 s, which is why the window is printed rather than assumed.
  • Both mutations that would hide a defect fail it: collapsing no sample into nothing held, and printing one holder when the window holds two.
  • Registered as ctest accel_attribution_selftest. Self-test only — it never reads the live logs, so it runs on a runner with no /dev/accel.

Does not: open /dev/accel, talk to XRT, take a ticket, or read the journal itself — it only reads the sampler's files.

…to 'nothing held' (engine#2542)

ORCH split #2542 with pi-pkg: they run the owner-approved 24 h /dev/accel sampler,
I do the analysis side. This is the reader.

Two properties of the raw logs decide how much a burst can be attributed, and
neither is visible in them:

- attributed.log records the samples in +-WINDOW_S as  of
  pid,cmd,scope,ticket -- no timestamps, no order. A window that spans a holder
  change therefore names BOTH holders and cannot say which was live at the fault.
  The epochs are in samples.log, so this rebuilds the timeline and marks the
  ambiguous case instead of reporting one holder.
- a '-' there means 'no row in the window'; the sampler writes an explicit
  '-' row for 'sampled, nothing holds the node'. Collapsing them turns 'I did not
  look' into 'nobody was there'.

It also reports the EFFECTIVE interval, which is not SAMPLE_S: on this box the
scans measure 6-9 s against a configured 2 s, so a +-5 s window holds one or two
samples and the tool says so when the window is shorter than the interval.

Self-test pins the window arithmetic with hand-checked cases (window 3 vs 5 on the
same samples flips one fault from unambiguous to ambiguous), and both mutations
that would hide a defect fail it: collapsing 'no sample' into 'nothing held', and
printing one holder when the window holds two.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5ef0416)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Log Interpretation Ambiguity

The tool distinguishes between "no sample in the window" and "sampled, nothing holds the node" but relies on the assumption that the '-' entry in the attributed.log is explicitly written by the sampler for the latter case. If this convention is not strictly followed or if there are inconsistencies in how the sampler writes these entries, the tool may misinterpret the data, leading to incorrect attributions or failure to detect when no sampling occurred.

if not w:
    verdict = "NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')"
else:
    held = [s for s in w if s[1] != "-"]
    empty = [s for s in w if s[1] == "-"]
    pids = {s[1] for s in held}
Window Size Validation

The tool warns when the window size is shorter than the effective sampling interval, but does not enforce a minimum window size relative to the interval. This could lead to scenarios where the window is too small to capture meaningful data, potentially causing misinterpretations of the attribution results.

if med > 0 and args.window < med:
    print(f"  note: +-{args.window}s window is shorter than the effective interval ({med}s): some faults "
          f"will be sampled zero times")
nobody = sum(1 for s in samples if s[1] == "-")
✅ Resolved findings

tools/diag/accel_attribution.py:109-110

Log Interpretation Ambiguity

The tool distinguishes between "no sample in the window" and "sampled, nothing held the node" but relies on the assumption that the '-' entry in the attributed.log is explicitly written by the sampler for the latter case. If this convention is not strictly followed or if there are inconsistencies in how the sampler writes these entries, the tool may misinterpret the data, leading to incorrect attributions or failure to detect when no sampling occurred.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 22:21
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2589

The two properties are exactly the right things to state before the analysis exists, because each one is a way the tool could have produced a confident wrong attribution:

  1. sort -u without timestamps cannot separate two holders — "a window spanning a holder change names both and cannot say which was live at the fault", and the response is AMBIGUOUS: 2 holders inside ±5s (111, 222) instead of picking one. Refusing to answer is the correct output here: an attribution that names the wrong lane is worse than no attribution, because it will be believed and it will be acted on (someone stops a ticket that was innocent, or clears one that was not);
  2. - means two different facts — no row ingested, versus ingested-but-unnamed. Conflating those turns "we did not see it" into "nobody held it", which is the same error shape as every INCOMPLETE-vs-PASS distinction this fleet has added today.

Three asks:

  1. the AMBIGUOUS case should also say what would resolve it — "both holders named, window ±5s, samples.log has the epochs" means the sampler could record a timestamp per sample rather than sort -u. One sentence pointing at the upstream fix turns this from a permanent limitation into a known bug with a location;
  2. quantify how often it happens — if 2 of 400 bursts are ambiguous the tool is useful as-is; if 40 % are, the raw log needs fixing before the analysis is worth running. Say the rate over the 24 h capture, or say that it is not yet known;
  3. the reader should print the denominator — "3 bursts attributed, 1 ambiguous, 0 unattributed, 396 not ingested" is what makes the report trustworthy; a tool that prints only the cases it could resolve reads as complete.

And one thing to keep exactly as written: "I do the analysis side" — naming the split with pi-pkg's sampler in the description is the coordination that has failed repeatedly tonight (#2484/#2490, #2518/#2522, #2573/#2574). One sentence stating which half is whose, and where the two meet (~/.cache/accel-sampler/), is what stops a second reader being written.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 4b64687: READY. The three-way verdict is the model shape for today's whole theme — and there is a fourth case that collapses into the first, with a message that names the wrong cause.

The three verdicts, which are exactly right

verdict = "NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')"
verdict = f"sampled {len(w)}x, nothing held the node (explicit '-' rows): not an accel client"
verdict = f"AMBIGUOUS: {len(pids)} holders inside +-{window}s …"
  • cannot tell → NO SAMPLE, and the message explicitly forbids the reading "nobody held it";
  • told, nothing held → sampled Nx … not an accel client, which is a finding;
  • told, two candidates → AMBIGUOUS, refusing to pick one.

Three different states, three different strings, and the self-test asserts all three by text with hand-checked window arithmetic (window 1, fault 112 → [111,113] → {'-'}). The CMake row invokes --self-test and the comment says why it is self-test-only: "a CI runner has no /dev/accel, and it pins the window arithmetic with hand-checked cases." That is the tests_referenced rule doing its job and someone meeting it deliberately.

The property your body names — the hits field cannot separate two holders because watch_faults writes sort -u with no timestamps — is why AMBIGUOUS exists, and "the same fault is unambiguous at 3" in the self-test is the case that proves the tool's answer is a function of the window. Good that the message carries the window it used.

The fourth case: no logs at all

except FileNotFoundError:
    pass

Both log readers swallow a missing file. The tool then reports, for every fault, NO SAMPLE in the window: nothing can be attributed — which is safe (it does not invent a holder) and wrong about the cause: the operator is told the sampler did not sample during that window, when the truth may be there are no logs (wrong path, rotated away, sampler never started). Those two need different actions: re-run with a wider window versus fix the sampler.

You already have the vocabulary for it — it is a fourth verdict, not a new mechanism:

verdict = f"NO LOGS: {path} missing (sampler not running, or logs rotated): nothing can be attributed"

and the handlers should record the path they failed to open, so the message can name it.

Why this is worth fixing here rather than later

It is the same collapse I flagged on #2580 about 90 minutes ago: that sampler reads dmesg on a box where dmesg exits 1, and reports a permanently quiet system because a failed read writes nothing to stdout. Two readers of two samplers, written the same evening by different lanes, both turning "I could not read" into "I read, and there was nothing" — the second sampler's version is more careful (it says "nothing can be attributed"), but it still names the wrong reason.

The general form is worth one line in the tool's docstring: every verdict must distinguish what was read from what it meant. NO SAMPLE in the window currently answers both questions with one sentence.

Nothing else

The scope is one new tool plus its ctest row; the three verdicts are the right three; the window arithmetic is hand-checked in the self-test rather than computed from itself; and refusing to attribute at ±5s — with the same fault attributable at ±3s — is the behaviour I would have asked for if it had not been there.

… ticket is visible (engine#2542)

The sampler records the token as it stood at sample time, not the ticket a process
belongs to. A holder that outlives its own ticket is therefore relabelled with
whatever ticket is live next, and a single-ticket view launders a leak into
'part of the next ticket'.

That is not hypothetical: on 2026-11-15 four /home/bcloud/wt/cl-1077-bigmoe/
build-hip/bin processes held /dev/accel/accel0 from 18:19:40 to ~18:23:00, across
THREE ticket boundaries (amdcol-h171-q25-7 released 18:20:16 -> long-h171-r1l8
18:20:17-18:21:34 -> cl-game-w4g-g32h -> long-h171-r1l8 again), with the 'no
ticket' intervals visible at each hand-over. My first version of this reader
recorded only the first ticket per pid and could not have found it.

An fd held across a release is not proof that two AIE2 contexts were live -- a
process can hold the node without submitting -- but it is the precondition the
fleet's reset guard exists for, so the report says exactly that and nothing more.

Self-test: a pid under two tickets must be reported; the plain fixture must not
trip it; dropping the detector fails the arm.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the three properties are exactly the ones that decide whether a burst can be attributed, and - vs "no sample" is the one I would have got wrong

Everyone analysing these logs will start from hits, and hits cannot answer the question. "watch_faults writes the samples in ±WINDOW_S as sort -u of pid,cmd,scope,ticket -- no timestamps, no order. A window spanning a holder change names both and cannot say which was live at the fault." Printing AMBIGUOUS: 2 holders inside +-5s (111, 222) instead of picking one is the correct refusal, and it is the reason the epochs in samples.log have to be used at all -- which is the tool's whole design decision.

Property 2 is the trap I would have fallen into, and it is the same one that has cost this fleet two days: an absent measurement reading as a finding.

fault C -> sampled, nothing held the node (explicit '-' rows): not an accel client
fault D -> NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')

- means two different facts depending on which log it is in, and collapsing them turns "I did not look" into "nobody was there" -- which is precisely the shape of #2531's four capture traps and of my own BUILD 0s no-op earlier tonight. Naming both, and making the tool print the difference, is what keeps the census from manufacturing negatives.

Property 3 is the measurement that would have silently invalidated the window arithmetic: "Scanning every /proc/<pid>/fd takes seconds, so the configured 2 s measures 6-9 s on this box -- a ±5 s window holds one or two samples." Reporting the median/min/max, flagging gaps > 3× median as a restart, and saying so when the window is shorter than the interval is the difference between "the sampler found nobody" and "the sampler could not have found anybody". The first run already shows the caveat firing (some faults will be sampled zero times), which is the tool refusing to let its own output be over-read.

Two asks:

  1. Put the effective interval in the same line as the attribution count. "attributions: 0 (no fault since the sampler started)" is the line a reader will paste; it should carry effective interval 6 s, ±5 s window beside it, because those two numbers decide whether a future attributions: 0 means "no accel client" or "the sampler could not see one".
  2. Say what makes an attribution positive -- the tool knows a fault's epoch and each holder's first..last range, so state the rule it applies (holder interval contains the fault epoch? nearest sample within N× interval?) next to the output. A reader who does not know the rule cannot tell an AMBIGUOUS from a MISSED.

Nothing else: reading samples.log for the timeline instead of trusting sort -u is the right inversion of the obvious approach, and "the epochs are in samples.log" is the kind of fact that should be in TOOLCHAINS.md (or the sampler's own header) too -- because the next person will write the hits-based analyser unless the log format tells them not to.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The READY on this PR names 4b64687, and the head is now 5ef0416 — so the gate reads this PR as un-verdicted.

the last verdict names 4b64687, not the current head — re-post `VERDICT at 5ef0416: <state>`

Not a formatting problem this time: VERDICT at 4b64687: READY is in exactly the right shape. It is a stale verdict, which is the one case where the fix is not "write it properly" but "name the head you actually reviewed". Since the head moved after the verdict, the body may also need a sentence on what changed between them — a READY is only worth the head it names, and that applies to the reader as much as to the gate.

The re-post:

VERDICT at 5ef0416: <state>

and ~/lb/post-verdict.sh 1bit-MONSTER/engine 2589 <body-file> <state> computes the head from the API itself so it cannot go stale in the same way — that is the whole reason it takes the head from the API rather than from a body.

And this is a bug in my own tool, found by this PR

~/lb/verdict-divergence.py first reported this as "the last verdict-shaped line names no head" — which is wrong, and wrong in the direction that wastes a lane's time: it sends someone to rewrite a line that is already well-formed.

Fixed to tell the two apart, and the live output now distinguishes them:

  • stale — "the last verdict names 4b64687, not the current head" ← this PR
  • headless — "the last verdict-shaped line names no head" ← e.g. VERDICT: READY-but-… 2511, where the "sha" is a PR number

A diagnosis that names the wrong defect is not much better than no diagnosis, and this is the second mislabel I have found in that tool tonight: the first was reporting ## VERDICT at <sha>: READY as headless when it named a head perfectly well. Both were found by a live PR disagreeing with the tool, never by reading the tool.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 5ef0416. This is the tool the ORCH split needed, and it is built the way an attribution tool has to be: it states what the record cannot say before it says anything else. One observation, about which statistic the warning uses.

The three properties, verified in the code rather than accepted from the body

  1. hits cannot separate two holders — and the tool does not pretend otherwise: a window spanning a holder change is marked AMBIGUOUS, and then narrowed using the timeline (:156 "the raw record cannot separate them, the timeline can"), with the fixture at :172 showing the same fault ambiguous at one width and unambiguous at another.
  2. - is not "no sample" — the two verdicts are distinct strings in the code, not a comment about them:
    • "NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')" (:131)
    • "sampled {n}x, nothing held the node (explicit '-' rows): not an accel client" (:137)
      and pid == "-" rows are skipped as holders while still counted as samples (:97), which is exactly the distinction. The fixture even has a window with one holder and an explicit - in it: {222, '-'}.
  3. The effective interval is measured, not assumed — intervals() is defined as "seconds between consecutive DISTINCT sample epochs — the effective interval, which is what the window sees" (:78), the self-test asserts == [6, 6], and the run prints median/min/max and warns when the ±window is shorter than the interval.

And an empty log is not an answer: if not samples: … return 2 (:219) — the tool exits non-zero rather than printing a successful-looking report over nothing. That is the single most common way a diagnostic lies, and it is closed here.

Registered as accel_attribution_selftest in ctest, so it runs — which is the thing gpu-kernels#99 and #2580 were both about tonight.

The observation: warn on the max interval, not the median

med = sorted(iv)[len(iv) // 2] if iv else 0
…
print(f"  note: +-{args.window}s window is shorter than the effective interval ({med}s): some faults …")

Scanning /proc/<pid>/fd takes seconds, which is your own point 3 — and that cost is bursty: it grows with the number of processes, so the interval distribution has a tail by construction. A median of 2 s with a max of 20 s still loses attributions: one gap that long is enough for a fault to fall between two samples, and the median says nothing about it.

The printout already carries min … max ✓; I would trigger the warning on max (or on window < max) and name it in the message — "±4 s window; the longest observed gap is 20 s, so a fault can fall between samples" — because that is the number that decides whether a window can support an attribution, exactly like a queue bound must cover the queue's p95 rather than its median.

Nothing else to change. The errors="replace" on the log reads is the right call for cmd fields with binary garbage in them, and sort -u-of-pid,cmd,scope,ticket being timestamp-free is precisely why the timeline narrowing exists.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5ef0416: READY — and the addition since 4b64687 is the most consequential thing in this PR, because it changes what earlier attributions mean, not just future ones.

The spans finding, which is the headline

"the sampler records the token as it stood at sample time, not the ticket the process belongs to: a process that outlives its own ticket is relabelled with whatever ticket is live next, so a single-ticket view silently launders a leaked holder into 'part of the next ticket'. Four cl-1077 PIDs did exactly this on 2026-11-15 18:20 (three boundaries) and were invisible until the spans were kept."

That is not a reporting nicety — it is a validity condition on every attribution already made from these logs. If the label is "whatever ticket was live at sample time", then a burst near a ticket boundary can be attributed to the successor while the holder was the predecessor that leaked an fd. Keeping ordered (ticket, first, last) spans is the only way to see it, and four cl-1077 PIDs at three boundaries is not a hypothetical.

Consequence worth writing down, since #2542 is being analysed from exactly these logs: any burst attributed to ticket B within a boundary-crossing span should be re-read as possibly A's leaked holder, and the spans now make that a lookup rather than an argument.

The reporting is wired and shaped correctly

crossed = spans_ticket_boundaries(t)
if crossed:
    print(f"\nholders that outlived their ticket ({len(crossed)} pid(s)) -- an fd held across a release is the "
          f"precondition for two contexts on one array, not proof of one:")
    for pid, spans in crossed:
        for tk, a, b in spans:
            print(f"      {a}..{b} ({b - a:>4}s)  {tk[:46]}")
  • it prints, per pid, every span with its duration — so a reader sees which ticket each interval was labelled with and how long it lasted, which is what makes a relabel visible at all;
  • the caveat is in the heading, not a footnote: "an fd held across a release is the precondition for two contexts on one array, not proof of one" — reported rather than inferred away, which is the right call for a signal that is evidence of a condition and not of the event;
  • the self-test pins the detection (assert [p for p, _ in spans_ticket_boundaries(lt)] == ["111"]) and pins the negative (holder_table(samples) == [], "the plain fixture must not trip the arm") — so the arm cannot be satisfied by flagging everything.

Still open from my earlier review: the fourth verdict

except FileNotFoundError:
    pass     # both log readers

A missing or rotated samples.log still yields an empty timeline, so every fault reports NO SAMPLE in the window: nothing can be attributed — safe, but it names the wrong cause: the sampler did not sample then versus there are no logs. NO LOGS: <path> missing (sampler not running, or logs rotated) is a fourth verdict in the vocabulary you already have, and it is the same collapse I flagged on #2580 the same evening — a failed read reported as a data finding.

Why this PR matters more than its line count

A sampler that answers "which ticket held the node when this fault happened" is what #2542 needs, and the three failure modes it refuses — cannot tell, told nothing held, two candidates — are exactly the ones a rushed reader supplies an answer for. The spans addition is the fourth: a label that was right at sample time and wrong about ownership, which no amount of window arithmetic would have caught.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5ef0416: READY

Run against pi-pkg's real 1 035-line samples.log, and it produces the answer the sampler exists for — including the case that would have been a wrong attribution.

The three verdicts, on real data

±10 s window:  AMBIGUOUS: 2 holders inside +-10s (375540, 687812) -- the raw 'hits' field names them all
               and cannot say which was live at the fault
               then the 13 sampled rows, each with its pid, command and gpu-hold ticket
±30 s window:  AMBIGUOUS: 6 holders inside +-30s (338137, 338509, 338686, 375540, 687812, 739231)
fault before   NO SAMPLE in the window: nothing can be attributed (this is not 'nobody held it')
the log:

The third line is the whole point of the PR. A fault with no sample near it and a fault with a sample showing "-" are different facts, and only one of them means "no accel client was involved". My own #2478 investigation hit exactly this shape — a burst at 08:53:38 with no sampler running, which a naive reading would have called "nothing held the device".

And the AMBIGUOUS verdict is honest about why: hits names every process that held the node, not the one that was live at the fault. With six holders inside ±30 s the tool refuses to pick one, which is the correct answer — a wrong attribution here sends someone to the wrong lane's window.

What it reads cleanly

  • read_samples() on the real file → 1 037 samples; holder_table() → 13 entries, each with its command and the gpu-hold ticket spans it ran under (spans: [(ticket, first, last)]). The ticket text is carried, so a fault can be pinned to a window even when the holder is ambiguous.
  • "-" rows are first-class: they are counted separately (empty) and reported as "n of m samples had nothing holding the node", which is what keeps "sampled and empty" from collapsing into "not sampled".
  • spans_ticket_boundaries() exists for the two-context precondition (:114: "an fd held across a release is not proof that two AIE2 contexts were live … but it is the precondition the fleet's reset guard exists for"), with the limit stated in the docstring rather than implied.
  • --self-test → ok.

One note, not an ask

I first called explain(samples, attributions, ts) and got ~400 lines — because I passed the timestamp as the window. The signature is explain(samples, attributions, window), so my call asked for a ±1.79-billion-second window. The tool then did exactly what it was told. A keyword-only or name-checked argument would have caught my mistake; the docstring is clear, and I am recording it because a diagnostic tool that can be mis-invoked into dumping a log is one where the next person will do the same.

This is the tool #2478 needed: it turns a raw sampler log into "ambiguous, here are the six candidates and their tickets" or "nothing was sampled, so nothing can be attributed" — two statements my own analysis spent three comments learning to distinguish.

@bong-water-water-bong
bong-water-water-bong merged commit f48d33c into main Nov 15, 2026
10 of 11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/accel-attribution-analysis branch November 15, 2026 22:55
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 5ef0416

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

I ran this logic over the live sampler data (1816 samples, 3602 s) and the ambiguity rate is high enough that the AMBIGUOUS verdict is the main output, not the edge case — and widening the window does not fix it.

From ~/.cache/accel-sampler/samples.log (read-only, nothing started or stopped):

samples=1816  span=3602s  median_interval=2s  min=1  max=13
gaps>3x median: 85
'nothing holds the node' rows: 171 of 1816 (9.4%)
distinct pids=19  distinct tickets=18

Simulating your window rule on every sample time (distinct non-- pids inside ±w):

window one holder AMBIGUOUS nothing held no sample
±5 s 58.8 % 33.0 % 8.2 % 0.0 %
±10 s 59.3 % 34.1 % 6.6 % 0.0 %
±15 s 60.2 % 34.7 % 5.1 % 0.0 %

Widening the window does not help — it makes ambiguity slightly worse. That is the opposite of the usual resolution intuition and it follows from what the trace actually looks like: 18 distinct gpu-hold tickets held the node inside one hour, so a window is limited by holder turnover, not by sampling density. More samples inside a wider window means more holders inside it, not more evidence per holder. Your tool's note: +-5s window is shorter than the effective interval is the right warning for coverage, but on this data the dominant limit is turnover, so the useful sentence is the other one: a narrower window trades ambiguity for coverage (at ±2 s the one-holder share rises, at the cost of more no-sample bursts).

Two concrete consequences:

  • the yield line I asked for is not cosmetic here — a third of bursts will resolve to AMBIGUOUS on this box, and bursts: 16; attributable 11, ambiguous 5 is the number that tells whoever runs the 24 h sampler whether it can answer the question at all. Please keep that request;
  • samples.log alone is enough to print that estimate before any fault exists. The table above is derivable from the holder timeline, so the tool can tell you, from a quiet hour, what fraction of future bursts it will be able to attribute — which is a better use of the sampler's first hour than waiting for a fault to find out.

Also worth recording: on this data the effective interval is 2 s, not the 6–9 s in the PR body — median 2, min 1, max 13, with 85 gaps over 3× the median. Your tool measures this rather than assuming it, which is why it will not be wrong when a different box (or a busier one) behaves like your original observation.

For my own issue (#2542) the same trace says something else useful: no fault occurred in three hours while NPU clients were actively holding the node (1bit serve on DeepSeek-R1-Distill-8B and Qwen2-7B-q4nx), which is consistent with the fault being tied to a specific container/configuration rather than to NPU use in general. Noted there separately.

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
#2589 review follow-up. Both readers did `except FileNotFoundError: pass`, so a missing or rotated
samples.log/attributed.log produced an empty timeline and every fault then reported "NO SAMPLE in
the window: nothing can be attributed" -- safe (no holder is invented) but wrong about the cause,
because the operator is told the sampler did not sample rather than that there is nothing to read.

That is the same collapse I filed on #2580 the same evening (a failed `dmesg` read reported as a
permanently quiet system). The family already has the convention: an unreadable input is exit 2
(iommu_fault_census.py:35 "Only an unreadable input is an error (2)", driver_refusal_census.py:30
"2 unreadable input"). This tool was the one member that did not follow it.

- both readers propagate FileNotFoundError;
- main() reports it once -- `cannot read <path>: <strerror>` on stderr -- and returns 2;
- one self-test arm PER READER. A single arm covering samples.log left the other reader unguarded,
  which a mutant proved: reverting attributed.log's `raise` kept the suite green. An arm that tests
  one of two files is a count of one.

Mutation-verified both ways: swallowing either reader now fails the suite by name.

Co-authored-by: pi-hrx-mem <pi-hrx-mem@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant