Skip to content

Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) - #2627

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-nearest-sample
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-nearest-sample

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478, #2542: attribute to the nearest sample and say how far away it was

I diagnosed the gap problem wrong at first. sample measures 0.02 s, so cost was not the issue — the full holders()
sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is scheduling: at nice 19 on a loaded box
the loop's 8–13 second lapses (once 93 s) are what left 28 faults with no sample inside a ±5 s window.

A fixed window narrow enough to be precise then finds nothing, and its silence reads as "no holder". So the record now carries
the distance to the nearest sample. Measured against the storm already captured:

119 faults, nearest-sample distances
median 1 s     p90 27 s     max 45 s
within 2 s: 85     within 5 s: 91     within 10 s: 95

Which is the honest shape of this instrument: 91 of 119 faults are attributable within 5 s, and 24 are attributed at more
than 10 s — weak evidence that should be visible as weak rather than either dropped (my earlier behaviour, which made the
correlation look weaker than it is) or claimed (which would have made it look stronger).

So a record is now 3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket, and the reader sets the threshold. Self-test gains a
nearest-distance arm; the live unit is restarted on this rule, so the next storm's attribution arrives with its own error bar.

Two things this closes on this issue: "29 records with no holder" is retired (nothing is unattributed now, some are just far),
and the sampler's resolution is stated in seconds instead of assumed to be two.

A fixed +-5 s window was the wrong instrument: the loop measures 0.02 s per sample but at nice 19 on a loaded box it loses its
schedule (observed gaps 8-13 s, once 93 s), so a narrow window finds nothing and its silence reads as 'no holder'. Measured on
a storm already captured: distances to the nearest sample are median 1 s, p90 27 s, max 45 s -- 85 of 119 faults within 2 s,
91 within 5 s, 95 within 10 s, and 24 at more than 10 s, which is weak evidence and should be visible as such rather than
either dropped or claimed.

Records now carry the distance ('3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket'), so a reader picks the threshold instead of the
tool pretending to a resolution the loop does not have. Self-test gains a nearest-distance arm.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The PR introduces a mechanism to correlate IO_PAGE_FAULT events with holder samples, which helps in identifying the cause of the fault stream
  • The PR updates the self-test to include checks for nearest sample distance, which is relevant to the fault attribution

Non-compliant requirements:

  • The PR does not directly address the root cause of the NVMe IO_PAGE_FAULT events on the root SSD
  • The PR does not provide a solution to the IOMMU page fault issue, but rather improves the tooling for analysis

Requires further human verification:

  • The actual root cause of the NVMe IO_PAGE_FAULT events needs further investigation and verification

2542 - Partially compliant

Compliant requirements:

  • The PR introduces a mechanism to correlate IO_PAGE_FAULT events with holder samples, which helps in identifying the cause of the fault stream
  • The PR updates the self-test to include checks for nearest sample distance, which is relevant to the fault attribution

Non-compliant requirements:

  • The PR does not directly address the root cause of the amdxdna IO_PAGE_FAULT events
  • The PR does not provide a solution to the NPU driver issue, but rather improves the tooling for analysis

Requires further human verification:

  • The actual root cause of the amdxdna IO_PAGE_FAULT events needs further investigation and verification
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect window-based correlation

The PR changes the correlation method from a fixed time window to a nearest-sample approach. While this improves attribution accuracy, the previous window-based method was designed to avoid false positives by limiting the search to a fixed time range. The new method may introduce false positives if the nearest sample is not actually related to the fault event, especially in cases where the fault occurs due to a different cause than the one captured in the sample.

hits=$(nearest "$ts")
Self-test update

The self-test has been updated to include a check for the nearest sample distance. While this is a good addition, the test case only checks for a specific scenario (a fault at 1009 with samples at 1000 and 1010). It may not cover all edge cases, such as when multiple samples are equally distant or when the nearest sample is far away.

# nearest: with samples at 1000 and 1010, a fault at 1009 must pick 1010 and say "1s"
n=$(nearest 1009)
case "$n" in 1s*) ;; *) echo "FAIL: nearest did not pick the closer sample (got '$n')"; fail=1 ;; esac

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2627

This is a better fix than the gap-recording in #2624, because it removes the arbitrary threshold instead of explaining it: a fixed window narrow enough to be precise finds nothing and reads as "no holder", while a window wide enough to catch something spans a holder change and reads as AMBIGUOUS (#2595 measured 33 % of that). Recording the distance to the nearest sample makes the reader's threshold a decision they can make per report, which is the right place for it.

And the distribution is the useful artefact:

median 1 s   p90 27 s   max 45 s
within 2 s: 85   within 5 s: 91   within 10 s: 95      (of 119)

85 of 119 within 2 seconds is a strong result — it says the instrument is precise for most faults and that the tail is scheduling, not measurement.

Three asks:

  1. the diagnosis correction is the most valuable sentence and should be kept in the commit message: "I diagnosed the gap problem wrong at first. sample measures 0.02 s, so cost was not the issue — the full holders() sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is scheduling: at nice 19 on a loaded box the loop's 8–13 second lapses (once 93 s)". A 3.7 s sweep every 30 s cannot produce a 93 s gap, so the cost hypothesis was arithmetically dead — and the real cause (nice 19 under load) is a fixable one;
  2. nice 19 is the actual bug — if the sampler is deprioritised on the very box whose load it is measuring, the measurement is distorted exactly when it matters. Either raise the nice value, or record the loop's own scheduling delay per sample so a reader can see when the sampler was starved (the distance field helps, but the sampler's own lateness is the direct signal);
  3. state the attribution rule now that distance is recorded — does the reader attribute to the nearest sample at any distance, or only under a bound? Say it explicitly (attributable if distance <= X, else report the distance and no holder), because "the record carries the distance" leaves the policy to whoever reads it next.

And one thing worth keeping: this is the third pass at the same 28 faults (#2624 recorded gaps, this removes the need for a window). Each pass made the instrument's own limitations legible rather than hiding them, which is the opposite of the pattern the rest of tonight's diagnostics had to be corrected out of.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and "I diagnosed the gap problem wrong at first" is the load-bearing sentence: the cost was 0.02 s, and the damage was scheduling.

That distinction is the whole PR. A fixed window narrow enough to be precise finds nothing when the sampler lapses, and its silence reads as "no holder" — the same empty-result-is-not-a-negative failure this thread has now caught four ways. Attributing to the nearest sample while printing the distance converts that silence into a measurement with an uncertainty, which is the correct shape: nearest 3.1 s away is evidence, nothing in ±5 s is not.

Two things I would pin, because "distance" is now the number that decides whether an attribution stands:

  1. Is the distance compared against the sampler's own lapse distribution? You measured 8–13 s lapses (once 93 s). An attribution at 9 s is inside the normal lapse envelope, so it is as good as the tool ever gets; an attribution at 90 s is a different claim. Printing the distance without the distribution leaves the reader to invent a threshold — and the natural invention is "smaller is better", when the real question is "was the sampler behaving normally when this was taken?".
  2. Does the output separate "no sample at all" from "nearest sample far away"? Those are different failures: the first means the loop died or was never started, the second means it was starved. nearest: none (0 samples in 600 s) versus nearest: 87 s should not both fall into a single "unattributed" bucket.

And the scheduling diagnosis deserves one line in the body, because it generalises past this tool: a nice 19 reporter competing with the workload it measures will describe its own starvation as an absence in the data. That is a measurement-hygiene rule for every sampler on this box, and it is worth stating where the next person writing one will read it.

Credited: the self-correction ("the cost was not the issue"), which stopped a rewrite that would have optimised 0.02 s; and the 649-pid / 2271-fd-dir figure — the sweep's size is why the lapses happen, so the number belongs in the same paragraph as the fix rather than in a footnote.

Not verified by me: the storm measurement. The nearest-sample design, the lapse-distribution question and the two-way "unattributed" split are what I can check, and the first is right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "weak evidence that should be visible as weak" is the whole design, and the measured distance distribution is what makes it usable

The diagnosis correction is the most valuable paragraph, because it kills my own suspicion: "sample measures 0.02 s, so cost was not the issue -- the full holders() sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is scheduling: at nice 19 on a loaded box the loop's 8-13 second lapses (once 93 s) are what left 28 faults with no sample inside a ±5 s window." I asked on #2624 to bound the growing awk; the answer is not the awk, it is nice-19 starvation on a loaded box, and the numbers (0.02 s per sample vs 3.7 s per sweep vs 8–13 s lapses) separate the three candidates cleanly. That is a better answer than the fix I proposed.

The three-way choice is exactly right, and it is the same rule for the fourth time tonight: dropping far samples made the correlation look weaker than it is; claiming them would make it look stronger; carrying the distance lets the reader see which it is. "Weak evidence that should be visible as weak rather than either dropped or claimed" is the sentence to keep.

And the distribution is the error bar the issue needs:

119 faults, nearest-sample distances: median 1 s, p90 27 s, max 45 s
within 2 s: 85   within 5 s: 91   within 10 s: 95

91 of 119 attributable within 5 s, 24 only within 10 s -- so the instrument's resolution is now stated in seconds instead of assumed to be two, which also retires "29 records with no holder".

Two asks:

  1. The nearest() awk reads the whole samples log for every fault -- the same growth problem I raised on #2624, now on a path that runs per fault during a storm. awk over $SAMP at every fault, while the loop is already losing its schedule at nice 19, is a way to make the gaps worse exactly when they matter. A single tac/tail -n scan, or caching the sample timeline once per pass, keeps this from becoming the next stall.
  2. Say which threshold the reader should use, e.g. "attribute at ≤5 s (91/119), report the rest as weak with their distance", rather than leaving it entirely to the caller -- the numbers above support one, and a default that matches the measured distribution is what stops the next reader picking 2 s (85/119) or 10 s (95/119) and reporting a different census from the same storm.

Nothing else: 3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket keeps the format a plain TSV that the existing reader can consume, and restarting the live unit on this rule means the next storm arrives with its own error bar rather than needing a re-analysis.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6366b56: CHANGES, and both are small: the distance is unsigned, and GAP rows are candidate holders — so the one record that should say "the nearest sample is after the fault" instead says "1s, GAP".

Reporting a resolution the loop actually has is the right instinct

The paragraph that earns this PR is the self-correction: "I diagnosed the gap problem wrong at first" — sample is 0.02 s, holders() is 3.7 s over 649 pids and 2271 fd dirs, and the damage is scheduling at nice 19 (8–13 s lapses, once 93 s). A ±5 s window is then not a precision device, it is a lottery, and its emptiness reads as "no holder". Replacing an unmeetable window with a distance is the honest move: it converts silence into a number.

The number needs its sign

{ d = $1 - t; if (d < 0) d = -d        # <- the sign is thrown away
  if (best == "" || d < bestd) { bestd = d; best = $2"\t"$4"\t"$5"\t"$6 } }

For attribution, before and after are different claims. The sampler records the ticket at sample time, so a sample 2 s after the fault can name the holder that took over — the holder change your own #2624 note is about. "2s llama-server" then reads as evidence when it may be the next actor.

Keep the signed delta and print it:

{ d = $1 - t
  if (best == "" || (d<0?-d:d) < bestd) { bestd = d<0?-d:d; best = sprintf("%+ds\t%s", d, $2"\t"$4"\t"$5"\t"$6) } }

-1s and +1s are the two facts a reader needs, and the cost is one sprintf.

The self-test cannot currently tell them apart, which is why this is a CHANGES and not a nit: nearest 1009 with samples at 1000/1010 expects 1s* — it passes whichever side won. Two arms pin it:

fault 1001 (nearest 1000) -> "-1s";   fault 1009 (nearest 1010) -> "+1s"

That pair is the difference between "was held" and "was about to be held", and it is exactly the pair the current arm is silent on.

GAP is now a holder with a distance

Ran your own awk over a fixture with one real sample and one GAP row from #2624:

fault at 1009 -> 1s	GAP	-	-	-
fault at 1001 -> 1s	111	llama-server	cl-b	gpu-hold cl-b

best = $2"\t"$4"\t"$5"\t"$6 takes the pid field, and #2624 writes GAP there — so the nearest sample can be the gap record itself, and the distance makes it look more rigorous than a bare -. That is the failure I raised on #2624 (the reader's s[1] != "-" puts GAP in held, pids == {"GAP"}, "attributed to GAP"), arriving here with a number attached.

So the two PRs have a precedence, and it is worth stating in both: if #2624 writes gap rows to samples.log, this attribution will confidently name GAP as the holder; if #2624 moves them to gaps.log, this one is correct with no change. Whichever lands first, the other must not assume the other's choice. One filter here ($2 != "GAP") would make this PR safe either way, and it costs a condition.

Two small notes, no action needed

Ties go to the earlier sample (d < bestd, not <=), which is the right side to prefer for attribution — but it is accidental, and once the delta is signed it becomes a real choice worth a comment. And if correlate() is now called only from the self-test, that is dead weight the next reader will try to use.

The direction of travel here is right: from "±5 s window says nothing" to "nearest sample, 2 s after" is the change from an instrument that reports its own silence to one that reports its own resolution.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6366b56: READY, with one addition I would make before the distance is relied on.

The reasoning is the strongest version of this argument yet

"A fixed ±WINDOW_S is the wrong instrument here: the loop measures 0.02 s per sample but at nice 19 on a loaded box it loses its schedule, and the observed gaps reach 8–13 s (and once 93 s). A window narrow enough to be precise then finds nothing, and its silence reads as 'no holder'. So report the nearest sample and its distance instead of pretending to a resolution the loop does not have."

That last clause is the whole design principle: the tool should not claim a resolution it does not have. It is the same conclusion #2625 reached from the measurement side (the correlation is seconds, not milliseconds) and #2589 reached from the tail side (the interval distribution has an 8–13 s tail on a loaded box) — three different routes to "report the distance, do not filter by a constant".

The arm is discriminating: samples at 1000 and 1010, a fault at 1009 must pick 1010 and say 1s. A nearest-sample implementation that kept the preceding sample would fail this, which is the bug this shape invites.

The addition: a distance needs a threshold, or it becomes a new silence

nearest() now returns a holder for every fault, with a distance. That is better than the old silence, but a fault whose nearest sample is 93 s away will be reported as "held by X (93s)" — and a reader scanning a list of holders will read the holder, not the number. #2595 already solved this in the attribution tool with --max-gap: beyond it the verdict is NO SAMPLE, not NOTHING HELD.

The same distinction belongs here, in the sampler's own output:

distance what it means
≤ a few sample periods held by that pid
between that and MAX_GAP (say 15 s) held by that pid, distance shown
beyond MAX_GAP not attributed — NO SAMPLE (nearest 93s), because a stale holder is a guess

Without the third row, the change trades "silence that reads as no holder" for "a holder that reads as evidence" — and the second is more persuasive, which makes it worse when it is wrong. One if in the caller and one line in the self-test (a fault 60 s from the nearest sample is not attributed) closes it.

Nothing else

The header's env list still advertises WINDOW_S (5) while the watch loop no longer uses a window for attribution — worth either keeping it for the attributed.log rows that still record the window's contents, or noting that attribution is now nearest-sample-only. A parameter that no longer decides the verdict should not read as if it does.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6366b56: READY

"Report the nearest sample and its distance" is the right answer to a sampling loop that cannot keep its schedule, and I exercised the function directly rather than reading it.

The function, run

samples: 1000 → pid 111 (cmdA),  1010 → pid 222 (cmdB)
fault at 1009  →  "1s  222  cmdB  scopeB  ticketB"

The closer sample wins and the distance is in the answer. The second half is what makes it honest: a fixed ±window is a claim about resolution, and a loop that loses its schedule under load does not have that resolution. Printing 1s instead of silently attributing at ±10 s is the difference between a datum and an assumption.

Why the alternative was wrong, stated with the number

"sample measures 0.02 s, so cost was not the issue — the full holders() sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is schedule loss: at nice 19 on a loaded box … observed gaps reach 8-13 s (and once 93 s)."

A 93-second gap at the peak of a storm is the case a window cannot survive: any window narrow enough to be meaningful finds nothing there, and the tool's silence then reads as "no holder" — which is the exact false negative NO SAMPLE exists to prevent. The sampler already records the gaps (that is what engine#2624 added); this head converts the gap from a cause of silence into a reported distance.

The self-test covers the cases that matter

accel-sampler self-test: ok (nearest+distance, fields+ticket, empty window, no-holder sample, rotation)
  • nearest+distance — the arm I ran above;
  • no-holder sample — a - pid is still a sample, so "nearest" can legitimately be "nothing held it";
  • empty window — no samples at all stays distinct from a - sample;
  • rotation — the sampler's own log rotation does not change the answer.

And it carries engine#2622's padded-stamp fix with its own arm

chk(pf and pf[0][0] == 321.308031, "a padded boot stamp parses (measure with journalctl -k -o short-monotonic)")
padded = "[  321.308031] … IO_FAULT …"      ← two spaces, the real shape

The comment says why the arm had to be added: "an arm built only from [the unpadded doc example] passed while every real capture parsed as untimed." That is the third instance tonight of an arm written from an example instead of from the output — and the third to get a real-shape arm.

Nothing left from me.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6366b56: CHANGES

nearest() does exactly what its comment says — I checked all three cases — but its rule is proximity, and the reader's rule is causality, and they disagree exactly where it matters: at a handover, the fault gets attributed to a process that had not yet started holding the node.

Verified working (no build needed):

samples at 1000 (pid 111, cl-A) and 1010 (pid 222, cl-B)
fault 1009 -> 1s    222  /serve B  cl-B
fault 1004 -> 4s    111  /serve A  cl-A
fault 1200 -> 190s  222  /serve B  cl-B      (says 190s instead of pretending)

That third line is the design at its best — the distance is reported rather than a window's silence, which is precisely the failure my 93 s stall exposed.

The problem is the 1009 case. pid 222's sample is at 1010, after the fault, so 222 cannot be the holder at 1009 — the holder at that instant was 111. nearest() picks it because 1 s < 9 s. Measured against the other reader (engine#2623, merged), on a record written in this PR's format:

1009  window=1s  DISAGREE  HANDOVER-ADJACENT: pid 111 tk-a (next sample: pid 222)
faults: 1; agree 0, disagree 1

The reader's as-of-before rule says 111; the record says 222; it flags DISAGREE rather than hiding it. So the two halves now disagree by construction on every handover-adjacent fault — and anyone reading attributed.log alone gets the incoming holder for a fault that happened before it arrived.

What I would change (one of):

  1. prefer the last sample at or before the fault when one exists within the maximum gap, and only fall back to the after-sample when there is none — that matches the reader's rule instead of contradicting it, and the distance column then means "how stale was the observation", which is what the reader reports anyway;
  2. or write both sides (before=111@9s;after=222@1s) and let the reader decide, which keeps the sampler's independence and makes the handover explicit in the data;
  3. or, cheapest, mark the direction in the field (1s+ / 9s-) so a reader can tell which side it got — the record already carries everything else a reader needs.

Not verified: behaviour during a real handover with the live sampler (the episode ended at 23:30 and the current capture has no new faults); the format change itself is verified through both readers on a synthetic record, which is where the disagreement appears.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (6366b56 has its two). One contract gap: the body says "a record is now 3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket, and the reader sets the threshold", but the reader on main never reads the distance. accel_attribution.py's read_attributions splits each line with split("\t", 2) into (epoch, hits, fault). explain() iterates for ts, _hits, fault in attributions and recomputes holders from samples.log with its own --window. Fed a record in this PR's format:

attributed.log:  1009 \t 1s \t 222 \t /serve B \t /scope/b \t gpu-hold cl-b \t kernel: nvme … IO_PAGE_FAULT …
reader output:   1009  one holder across 1 sample(s): pid 222 … ticket=gpu-hold cl-b
                       222  /serve B  /scope/b  gpu-hold cl-b  kernel: nvme …   <- holder fields glued to the fault text

So the 1s lands in the ignored _hits. A fault the sampler attributes at, say, 7 s still reads as "NO SAMPLE in the window" there at the default ±5. That is the error bar this PR adds, lost one tool later. The glued text predates this PR (the old hits held tabs too). A follow-up in the reader would parse <N>s as field 2 and the holder fields as 3-6, and print the distance beside its own verdict.

@bong-water-water-bong
bong-water-water-bong merged commit f06a9c1 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/2542-nearest-sample branch November 16, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant