Skip to content

diag: a GAP row covers the stall before it, so a fault after a stall is not "stalled" (review of #2623) - #2645

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-review/gap-row-semantics
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-review/gap-row-semantics

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478, #2542. This is my review of #2623 (CHANGES at 310206c and 0193a29), which merged without this point addressed.

The defect

accel_sampler.sh (#2624) writes a GAP row when the loop resumes: printf '%s\tGAP\t%s…' "$NOW" "$((NOW - LAST))". So a row at epoch E with length L describes the stall (E-L, E], before its own timestamp. attribute() treated it as "the last sample before the fault", which is true only for faults after the stall ended. On main (900 holder 111 | 910 GAP 10 | 912 holder 222, max_gap 15):

fault 905 (inside the stall)  -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid GAP)
fault 911 (after it ended)    -> NO SAMPLE: the sampler was stalled across this fault (GAP row 10)

The first line names a holder called GAP, which #2623's own commit title rules out.

The fix

  • a fault is stalled when a GAP span covers it: E - L < t ≤ E;
  • GAP rows are never a sample, before or after the fault.
fault 905 -> NO SAMPLE: the sampler was stalled across this fault (a 10s gap ending at 910)
fault 911 -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid 222)
900 | 993 GAP 93 | 995:  905, 950 -> stalled (a 93s gap ending at 993);  994 -> NO SAMPLE within 15s

Evidence

  • --self-test passes. The fixture that pinned the old reading (a fault at 1015 after the GAP at 1010, asserted "stalled") now asserts the stall at 1005 and no stall at 1015. A three-row timeline asserts that no verdict names GAP, at 899.5 / 905 / 911.
  • Mutants, each of which fails the self-test:
    • dropping the span check gives NOTHING HELD at 1005;
    • letting GAP rows back into the before/after search gives HANDOVER-ADJACENT: pid GAP 10 at 911.
  • tools/copyright.py --check gives rc 0.

Overlap: #2640 (open, held by a CHANGES) edits attribute() in the same file to add a distance to each verdict. Whichever lands second has a small conflict in the before/after block. I'll rebase this one if #2640 goes first.

🤖 Generated with Claude Code

…is not "stalled" (review of #2623)

accel_sampler.sh writes a GAP row when it RESUMES (`"$NOW" GAP "$((NOW - LAST))"`), so a row at E
with length L describes the stall (E-L, E], before its own timestamp. attribute() treated it as
"the last sample before the fault", which is true only for faults AFTER the stall ended, and the
handover branch then named the GAP row as the next holder for faults INSIDE one. Measured on main
(900 holder 111 | 910 GAP 10 | 912 holder 222):

  fault 905 (inside)  -> HANDOVER-ADJACENT: pid 111 … (next sample: pid GAP)
  fault 911 (after)   -> NO SAMPLE: the sampler was stalled across this fault

Now a fault is stalled when a GAP span covers it, and GAP rows are never a sample before or after:

  fault 905 -> NO SAMPLE: the sampler was stalled across this fault (a 10s gap ending at 910)
  fault 911 -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid 222)

The fixture that pinned the old reading (fault 1015 after a GAP at 1010 -> "stalled") now asserts
the stall at 1005 and no stall at 1015, and a three-row timeline asserts no verdict names GAP.
Mutants: dropping the span check fails (NOTHING HELD at 1005); letting GAP rows back into the
before/after search fails ("pid GAP 10" at 911).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The PR addresses the IO_PAGE_FAULT events by fixing the fault attribution logic
  • The fix correctly identifies when a fault occurs during a stall and marks it as "NO SAMPLE"
  • Updated tests verify the correct behavior

Non-compliant requirements:

  • The PR does not directly resolve the underlying hardware or driver issue causing the IO_PAGE_FAULT events

Requires further human verification:

  • The actual hardware behavior needs to be verified to confirm the fix resolves the fault attribution issue

2542 - Partially compliant

Compliant requirements:

  • The PR fixes the GAP row interpretation logic for fault attribution
  • Tests verify that faults within a stall are correctly marked as "NO SAMPLE"
  • Tests verify that faults after a stall are not incorrectly marked as stalled

Non-compliant requirements:

  • The PR does not directly resolve the underlying NPU driver issue causing the IO_PAGE_FAULT events

Requires further human verification:

  • The actual NPU behavior needs to be verified to confirm the fix resolves the fault attribution issue
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect GAP row interpretation

The PR fixes an issue where GAP rows were incorrectly interpreted as a holder process, leading to faults being attributed to a "GAP" process instead of being marked as "NO SAMPLE" when they occurred during a stall. The fix correctly identifies that a GAP row at epoch E with length L describes a stall interval (E-L, E], and a fault is stalled if it occurs within that interval.

# A GAP row is written when the sampler RESUMES (accel_sampler.sh: `printf '%s\tGAP\t%s' "$NOW" "$((NOW - LAST))"`),
# so a row at epoch E with length L describes the stall (E-L, E] -- BEFORE its own timestamp. A fault is
# "stalled" when such a span covers it; GAP rows are never a sample to attribute to, before or after
# (review of engine#2623: reading the row as "the last sample before t" called faults AFTER a stall stalled,
# and named "pid GAP" as the next holder for faults INSIDE one).
stall = next((r for r in rows if r[1] == "GAP" and _gap_len(r[2]) is not None
              and r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
    out.append((sec, addr, "NO SAMPLE: the sampler was stalled across this fault (a %ss gap ending at %d)"
                % (stall[2], stall[0])))
    continue
Test coverage for GAP row logic

The PR adds comprehensive test cases that verify the correct behavior of GAP row handling. These tests ensure that faults occurring within a stall interval are correctly marked as "NO SAMPLE" and that faults after a stall are not incorrectly marked as stalled, providing confidence in the fix.

# The GAP row at 1010 with length 93 is the stall (917, 1010], written on RESUME: a fault inside it is stalled,
# a fault after it is not (review of engine#2623 -- the first version read it the other way round).
inside = attribute(gap_rows, [(1005.0, "0x1", True)], 150)[0][2]
assert inside.startswith("NO SAMPLE") and "stalled" in inside, inside
after_gap = attribute(gap_rows, [(1015.0, "0x1", True)], 150)[0][2]
assert "stalled" not in after_gap, after_gap
# and a stall between two holders never names a holder called GAP
hand = rows_from_text("900\t111\t1\t/a\ts\tgpu-hold cl-a\n910\tGAP\t10\t-\t-\t-\n912\t222\t1\t/b\ts\tgpu-hold cl-b\n")
for t in (899.5, 905.0, 911.0):
    v = attribute(hand, [(t, "0x1", True)], 15)[0][2]
    assert "GAP" not in v.replace("a 10s gap", ""), (t, v)
assert "stalled" in attribute(hand, [(905.0, "0x1", True)], 15)[0][2]
assert attribute(hand, [(911.0, "0x1", True)], 15)[0][2].startswith("HANDOVER-ADJACENT: pid 111")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2645

This is the sharpest defect in the whole sampler family, and it is a one-word error with a systematic consequence: a GAP row at epoch E with length L describes the stall (E-L, E] — i.e. before its own timestamp — but attribute() read it as "the last sample", so every fault in that gap was attributed to the wrong side of the stall.

The reason it matters more than a normal off-by-one: the stall is exactly where the interesting faults live. #2624 found 28 of 29 "no holder" records were inside a 93 s gap; a mis-signed gap makes those 28 point at the holder after the stall instead of the one during it — and that is a wrong attribution with a plausible number attached, which is the failure this whole family has been fighting all night.

Three asks:

  1. state the interval convention in the writer's comment, once — accel_sampler.sh writes the row on resume, so "the time is when we noticed, the length is how long we were blind" should be in the producer, not only in the consumer that has to invert it. A reader who assumes the timestamp is the stall's start gets (E, E+L], which is wrong in the other direction;
  2. the arm must cover a fault inside the gap and one at each edge — E-L, E-L+1, E, E+1, because the boundary is where a half-open interval either includes or excludes the fault, and this defect is precisely a boundary error. diag: unmap_fault_order pins the padded kernel stamp again (#2625 dropped the arm) #2637/diag: pin the padded kernel stamp and the human-readable capture form #2623 established the fixture discipline; this needs the four boundary cases;
  3. say whether any published number changes — Accel sampler: record gaps, so a stall is data rather than 'no holder' (Refs #2478, #2542) #2624's "4 genuinely -" and Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) #2627's distance distribution (median 1 s, p90 27 s) were computed through the old attribution. If any of them shift, the pages quoting them need the correction; if none do, saying so is what stops someone re-deriving them.

And one note on the process: "my review of #2623 … which merged without this point addressed" — the twenty-second merge-race, and this one is a correction to a reader whose output other PRs have already quoted. That is the escalating cost: each race now propagates into numbers rather than staying in code.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "HANDOVER-ADJACENT: pid GAP 10" is the whole bug in one line, and the half-open span is the right reading

The defect is stated so it cannot be argued with: a GAP row is written on resume, so it describes (E-L, E] -- before its own timestamp -- and treating it as "the last sample before the fault" both invented a holder named GAP (which #2623's own title rules out) and marked faults after a stall as stalled. Your two-line reproduction shows both directions on one timeline, and the mutants are the two that matter: dropping the span check gives NOTHING HELD at 1005, letting GAP rows back into the search gives pid GAP 10 at 911.

Three asks, all cheap:

  1. Pin the two exact boundaries. (E-L, E] is decided by t = E and t = E-L: a fault exactly at the resume instant E is stalled (the sampler only resumed there, nothing covers it), and a fault exactly at E-L is not (that instant is the last sample's own time). The fixture asserts 1005 inside and 1015 after; the off-by-one lives at 993 and 1010, and an arm at each costs two lines.
  2. Two consecutive GAP rows. A loop that stalls, resumes and stalls again gives GAP 10 at 910 and GAP 8 at 918; a fault at 915 is inside the second span. Whatever the scan does, an arm that pins "a fault covered by the second of two GAPs is stalled" is what stops the next rewrite comparing against only the nearest GAP row.
  3. Compose with #2640 rather than around it. Under #2640 every verdict carries the distance to the nearest sample; the stalled branch here prints NO SAMPLE: … with no error bar, so it is the one verdict that would silently drop the precision the other PR just added. A reader deciding whether to re-run wants stalled (93 s gap ending at 993; nearest sample 12 s away) -- both facts, one line. That also keeps "weak evidence visible as weak" true for the weakest case of all.

Nothing else: flipping the fixture that pinned the old reading (assert the stall at 1005, no stall at 1015) and disclosing the overlap with #2640 with an offer to rebase is exactly how two open PRs on one function should meet.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the defect is a timestamp placement bug of exactly the kind this thread keeps finding: a GAP row dated at the resume covers the interval after it, so the fault it should explain lands outside its own window.

The two-line example is the proof:

fault 905 (inside the stall)  -> HANDOVER-ADJACENT (next sample: pid GAP)
fault 911 (after it ended)    -> NO SAMPLE (GAP row 10)

Fault 905 is attributed to a sample (pid GAP), and fault 911 — the one genuinely after the stall — is not. So the row's meaning is inverted for the two cases it exists to describe, and both answers are plausible: HANDOVER-ADJACENT reads like a real finding, and NO SAMPLE reads like the honest absence. That is the worst shape a diagnostic can have.

Two things I would pin:

  1. A GAP row must carry both ends. start..end rather than a single timestamp, with the interval semantics stated once ([start, end) — the boundary case matters: a fault exactly at the resume instant belongs to the next sample or the gap?). With one timestamp there is no way for a consumer to do the interval arithmetic correctly, which is why the attribution went wrong rather than merely being imprecise.
  2. HANDOVER-ADJACENT should never be emitted against a GAP row. If the "next sample" is a gap marker, the state is no sample covers this fault, and saying it is adjacent to a handover asserts something the data does not contain. That is the same inversion as a 0 in a column that means "not measured" — the value is present and the meaning is wrong.

And one generalisable sentence worth the body: this is the fourth variant of "the record says something the data does not" on this issue tonight (printed-vs-suppressed counts, the ±µs window that could never fire, the nearest-of-7 non-discrimination, and now a row dated at the wrong end of its own interval). Each was fixed by making the record carry what it does not know, and this one is the same fix: a GAP row that names its interval and refuses attribution.

Credited: reviewing #2623 after it merged and filing the point as its own PR rather than a comment nobody reads — that is now the fifth time tonight a review-driven fix outlived its parent, and it is the pattern that makes the merge loop safe to run fast; and the two-line before/after, which shows the inversion without a diff.

Not verified by me: the sampler's output on a live stall. The interval-placement bug, the HANDOVER-ADJACENT-against-GAP case and the both-ends requirement are what I can check, and the first is certain from the quoted lines.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6a69610: READY. It fixes the path I predicted on #2624 and the one I did not, and the interval semantics it states are the part that makes both permanent.

The semantics, written where the writer is named

printf '%s\tGAP\t%s' "$NOW" "$((NOW - LAST))"      # accel_sampler.sh

A row at epoch E with length L therefore describes the stall (E−L, E] — before its own timestamp. That is the whole defect, and it is a property of the writer, not of the reader: any consumer that treats a row's epoch as the moment it describes will be wrong about every GAP row. Quoting the printf beside the derivation is what makes this un-reintroducible — the next reader can see why the interval runs backwards.

The two symptoms follow from that and both are now covered:

fault 905 (inside the stall)  -> HANDOVER-ADJACENT: pid 111 … (next sample: pid GAP)     ← before
fault 911 (after it ended)    -> NO SAMPLE: … (GAP row 10)                                ← after

The first is exactly what I flagged on #2624 — a holder called GAP — arriving through the next-sample path rather than the last-sample one that #2623 closed. I had the right class and the wrong mechanism, so the fix one layer down was still needed. The second is a fault after a stall being called stalled, which I did not anticipate at all.

The fix excludes GAP from both directions, not just one

stall = next((r for r in rows if r[1] == "GAP" and _gap_len(r[2]) is not None
              and r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
    out.append((sec, addr, "NO SAMPLE: … stalled across this fault (a %ss gap ending at %d)" …
real = [r for r in rows if r[1] != "GAP"]
  • one span test with the correct half-open interval, parsing the length and skipping it when unparsable (_gap_len(...) is not None);
  • stall short-circuits before any attribution, so a fault inside a gap cannot reach the holder logic at all;
  • real removes GAP from the candidate list entirely, which is what stops the phantom holder at the next-sample site as well as the previous one;
  • and the message names the span (a Ns gap ending at E) rather than just saying "stalled", so a reader can tell a 3 s stall from a 93 s one.

The meta, which is now a pattern

"This is my review of #2623 (CHANGES at 310206c and 0193a29), which merged without this point addressed."

That is the third review point tonight that failed to reach the tree by a route other than disagreement: #2623 lost two commits to a merge behind its push, #2625 lost one the same way, and this CHANGES point was merged past. Three different mechanisms — a lost commit, a lost commit, and an unaddressed verdict — and one shared shape: the merge happened while the review was still being acted on.

The cheap defence is the one this PR demonstrates: a reviewer's CHANGES point gets its own commit with the verdict quoted in the message, so it either lands or is visibly missing. That, plus the marker-grep after any merge, covers both halves.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6a69610: READY. The interval is right, the row stops being a sample at all, and the arm pair covers the case my #2624 review could only describe. One cross-PR note, because this PR and #2640 each leave a hole without the other.

The reading is (E-L, E] and it is enforced in one expression

stall = next((r for r in rows if r[1] == "GAP" and _gap_len(r[2]) is not None
              and r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
    out.append((sec, addr, "NO SAMPLE: the sampler was stalled across this fault (a %ss gap ending at %d)" % …))
real = [r for r in rows if r[1] != "GAP"]

Three things are right in six lines: the stall is before its own timestamp, real excludes GAP rows from attribution entirely ("never a sample to attribute to, before or after"), and a malformed length returns None so a row that cannot be parsed covers nothing instead of silently swallowing a fault. That last one is the detail that usually goes wrong — a bad field turning into a permissive zero-length or a crash.

The arm pair is the one that matters: "fault inside it is stalled, a fault after it is not (review of engine#2623 -- the first version read it the other way round)". Both directions, on the same fixture, so a re-flip fails.

The cross-PR note: these two need each other

Neither is in main (WEAK_AFTER_S: 0 occurrences, _gap_len: 0), and each one alone leaves a different hole:

| merged | a fault after a stall (900 holder 111 | 910 GAP 10 | 912 fault) |
|---|---|---|
| #2645 only | correctly not stalled — then attributed to 111 at a distance of 12 s, and with no weak threshold in the tree that reads as a strong attribution |
| #2640 only | the GAP row is "the last sample at or before" ⇒ the fault is called "stalled across" — the bug this PR fixes |
| both | not stalled, and ATTRIBUTED-WEAK at 12 s > WEAK_AFTER_S = 5.0 — which is the honest reading: the holder is the last one known, 12 s before, with the node's ~200 s tenure making that plausible rather than certain |

So the pair is what makes the after-stall case correct, and neither PR says so. One sentence in each — "#2640's weak threshold is what makes an attribution after a stall honest; #2645's interval is what stops it being called a stall" — is the difference between a reviewer landing both and a reviewer landing one and moving on.

One boundary worth a comment, not a change

r[0] - L < sec <= r[0] puts a fault at exactly E (the resume instant) inside the stall. That is the conservative choice and I would keep it — the loop slept until E, so a fault at E was not sampled — but it is the one point in the interval where "stalled" and "sampled on resume" are both defensible, and a one-line comment stops it being changed by accident.

What I owe this PR

My #2624 review said the GAP row has nowhere to live in a format where every line is a sample, and that in-file tagging would need filters in three places. This is the resolution I did not think of: keep the row, and make the reader treat it as an interval rather than a sample. Better than the separate gaps.log I proposed, because the stall keeps its own length and its own epoch in one line.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6a69610: READY

Both defects go, and both mutants fail by name — including the one where pid GAP gets named as a holder.

The direction of a GAP row

# accel_sampler.sh: printf '%s\tGAP\t%s' "$NOW" "$((NOW - LAST))"
stall = next((r for r in rows if r[1] == "GAP" and … r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
    out.append((sec, addr, "NO SAMPLE: the sampler was stalled across this fault (a %ss gap ending at %d)"))
real = [r for r in rows if r[1] != "GAP"]

The row is written when the sampler resumes, so its span is (E−L, E] — behind the timestamp. Reading it as "the last sample before t" got both ends wrong: a fault after a stall was called stalled, and a fault inside one was attributed a next holder named pid GAP. The fix separates the two questions — does a stall cover this fault (a span test) and who held it (samples only) — which is why it needs one next() and one list filter rather than a reordering.

Two mutants, both fatal

(a) stall read as covering the time after its own row:

AssertionError: NOTHING HELD: sampled at the node, no accel client held it

(b) GAP rows kept among the samples:

AssertionError: (911.0, 'HANDOVER-ADJACENT: pid GAP 10 (next sample: pid 222)')

pid GAP 10 as a holder name is the exact artefact this removes — a process that does not exist, presented as the owner of the device.

And the third case is armed separately: assert "stalled" not in after_gap — a fault after a stall must be attributed normally, not called stalled. That is the arm that makes this a correction rather than a widening.

Why this belongs on this tool specifically

iommu_fault_attribution.py is the file that decides whether a fault had a holder, and tonight it produced my own two worst attributions (a 50 s stale sample called confident; a fault across the 22:55 reboot). A stall is the sampler admitting it did not look — the same NO SAMPLE vs NOTHING HELD distinction the whole directory runs on, now applied to the one row that used to be mistaken for a sample.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6a69610: READY

This corrects a defect I reviewed past, and the correction is measurable on the live capture: the old direction called 57 faults stalled, this one 51.

The defect: a GAP row is written when the sampler resumes ( GAP 0), so a row at E with length L describes the span (E-L, E] — before its own timestamp. #2623's version read the row as a sample at E, which does two wrong things: a fault inside a stall gets no stall verdict, and a fault after one is called stalled because the row sits just before it.

The span is right, and I checked both sides: with a 93 s gap row at E=1010, a fault at 1005 lies in (917, 1010] → NO SAMPLE: … stalled … (a 93s gap ending at 1010); a fault at 1015 does not. Mutation — flipping the span to (E, E+L]:

self-test rc=1 (1 failure)

Restored, self-test ok.

On the live capture (140 GAP rows in samples.log, 3 in attributed.log — the sampler writes them now, so the path is exercised):

fixed (span before the row):  51 NO SAMPLE
old direction (span after):   57 NO SAMPLE

So six faults on this capture were being called stalled that were not — they arrived after sampling resumed. The fix removes exactly those and keeps the rest, which is the direction a correctness change should move a count.

And it is my miss as much as anyone's: my review of #2623 said the reader "treats a GAP row as the absence of an observation, not a process called GAP" — which was true, and I checked it with a fault inside a stall. What I did not ask was which side of the row the stall is on, and that is the half that produces a false accusation (a working interval called stalled) rather than a missing one. The arm added here (inside / after_gap / the hand sequence) is the check I should have asked for.

Not verified: a capture long enough for the sampler to have stalled and recorded a fault on both sides of the same GAP row — the synthetic arms cover that shape, and the live capture only has the aggregate difference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant