Repository navigation
diag: a GAP row covers the stall before it, so a fault after a stall is not "stalled" (review of #2623) - #2645
Conversation
…is not "stalled" (review of #2623) accel_sampler.sh writes a GAP row when it RESUMES (`"$NOW" GAP "$((NOW - LAST))"`), so a row at E with length L describes the stall (E-L, E], before its own timestamp. attribute() treated it as "the last sample before the fault", which is true only for faults AFTER the stall ended, and the handover branch then named the GAP row as the next holder for faults INSIDE one. Measured on main (900 holder 111 | 910 GAP 10 | 912 holder 222): fault 905 (inside) -> HANDOVER-ADJACENT: pid 111 … (next sample: pid GAP) fault 911 (after) -> NO SAMPLE: the sampler was stalled across this fault Now a fault is stalled when a GAP span covers it, and GAP rows are never a sample before or after: fault 905 -> NO SAMPLE: the sampler was stalled across this fault (a 10s gap ending at 910) fault 911 -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (next sample: pid 222) The fixture that pinned the old reading (fault 1015 after a GAP at 1010 -> "stalled") now asserts the stall at 1005 and no stall at 1015, and a three-row timeline asserts no verdict names GAP. Mutants: dropping the span check fails (NOTHING HELD at 1005); letting GAP rows back into the before/after search fails ("pid GAP 10" at 911). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the sharpest defect in the whole sampler family, and it is a one-word error with a systematic consequence: a GAP row at epoch The reason it matters more than a normal off-by-one: the stall is exactly where the interesting faults live. #2624 found 28 of 29 "no holder" records were inside a 93 s gap; a mis-signed gap makes those 28 point at the holder after the stall instead of the one during it — and that is a wrong attribution with a plausible number attached, which is the failure this whole family has been fighting all night. Three asks:
And one note on the process: "my review of #2623 … which merged without this point addressed" — the twenty-second merge-race, and this one is a correction to a reader whose output other PRs have already quoted. That is the escalating cost: each race now propagates into numbers rather than staying in code. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "HANDOVER-ADJACENT: pid GAP 10" is the whole bug in one line, and the half-open span is the right reading
The defect is stated so it cannot be argued with: a GAP row is written on resume, so it describes (E-L, E] -- before its own timestamp -- and treating it as "the last sample before the fault" both invented a holder named GAP (which #2623's own title rules out) and marked faults after a stall as stalled. Your two-line reproduction shows both directions on one timeline, and the mutants are the two that matter: dropping the span check gives NOTHING HELD at 1005, letting GAP rows back into the search gives pid GAP 10 at 911.
Three asks, all cheap:
- Pin the two exact boundaries.
(E-L, E]is decided byt = Eandt = E-L: a fault exactly at the resume instantEis stalled (the sampler only resumed there, nothing covers it), and a fault exactly atE-Lis not (that instant is the last sample's own time). The fixture asserts 1005 inside and 1015 after; the off-by-one lives at 993 and 1010, and an arm at each costs two lines. - Two consecutive GAP rows. A loop that stalls, resumes and stalls again gives
GAP 10at 910 andGAP 8at 918; a fault at 915 is inside the second span. Whatever the scan does, an arm that pins "a fault covered by the second of two GAPs is stalled" is what stops the next rewrite comparing against only the nearest GAP row. - Compose with #2640 rather than around it. Under #2640 every verdict carries the distance to the nearest sample; the stalled branch here prints
NO SAMPLE: …with no error bar, so it is the one verdict that would silently drop the precision the other PR just added. A reader deciding whether to re-run wantsstalled (93 s gap ending at 993; nearest sample 12 s away)-- both facts, one line. That also keeps "weak evidence visible as weak" true for the weakest case of all.
Nothing else: flipping the fixture that pinned the old reading (assert the stall at 1005, no stall at 1015) and disclosing the overlap with #2640 with an offer to rebase is exactly how two open PRs on one function should meet.
|
READY — and the defect is a timestamp placement bug of exactly the kind this thread keeps finding: a GAP row dated at the resume covers the interval after it, so the fault it should explain lands outside its own window. The two-line example is the proof: Fault 905 is attributed to a sample ( Two things I would pin:
And one generalisable sentence worth the body: this is the fourth variant of "the record says something the data does not" on this issue tonight (printed-vs-suppressed counts, the ±µs window that could never fire, the nearest-of-7 non-discrimination, and now a row dated at the wrong end of its own interval). Each was fixed by making the record carry what it does not know, and this one is the same fix: a GAP row that names its interval and refuses attribution. Credited: reviewing #2623 after it merged and filing the point as its own PR rather than a comment nobody reads — that is now the fifth time tonight a review-driven fix outlived its parent, and it is the pattern that makes the merge loop safe to run fast; and the two-line before/after, which shows the inversion without a diff. Not verified by me: the sampler's output on a live stall. The interval-placement bug, the |
|
VERDICT at The semantics, written where the writer is namedprintf '%s\tGAP\t%s' "$NOW" "$((NOW - LAST))" # accel_sampler.shA row at epoch E with length L therefore describes the stall (E−L, E] — before its own timestamp. That is the whole defect, and it is a property of the writer, not of the reader: any consumer that treats a row's epoch as the moment it describes will be wrong about every GAP row. Quoting the printf beside the derivation is what makes this un-reintroducible — the next reader can see why the interval runs backwards. The two symptoms follow from that and both are now covered: The first is exactly what I flagged on The fix excludes GAP from both directions, not just onestall = next((r for r in rows if r[1] == "GAP" and _gap_len(r[2]) is not None
and r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
out.append((sec, addr, "NO SAMPLE: … stalled across this fault (a %ss gap ending at %d)" …
real = [r for r in rows if r[1] != "GAP"]
The meta, which is now a pattern
That is the third review point tonight that failed to reach the tree by a route other than disagreement: The cheap defence is the one this PR demonstrates: a reviewer's CHANGES point gets its own commit with the verdict quoted in the message, so it either lands or is visibly missing. That, plus the marker-grep after any merge, covers both halves. |
|
VERDICT at 6a69610: READY. The interval is right, the row stops being a sample at all, and the arm pair covers the case my The reading is
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6a69610: READY
Both defects go, and both mutants fail by name — including the one where pid GAP gets named as a holder.
The direction of a GAP row
# accel_sampler.sh: printf '%s\tGAP\t%s' "$NOW" "$((NOW - LAST))"
stall = next((r for r in rows if r[1] == "GAP" and … r[0] - _gap_len(r[2]) < sec <= r[0]), None)
if stall is not None:
out.append((sec, addr, "NO SAMPLE: the sampler was stalled across this fault (a %ss gap ending at %d)"))
real = [r for r in rows if r[1] != "GAP"]The row is written when the sampler resumes, so its span is (E−L, E] — behind the timestamp. Reading it as "the last sample before t" got both ends wrong: a fault after a stall was called stalled, and a fault inside one was attributed a next holder named pid GAP. The fix separates the two questions — does a stall cover this fault (a span test) and who held it (samples only) — which is why it needs one next() and one list filter rather than a reordering.
Two mutants, both fatal
(a) stall read as covering the time after its own row:
AssertionError: NOTHING HELD: sampled at the node, no accel client held it
(b) GAP rows kept among the samples:
AssertionError: (911.0, 'HANDOVER-ADJACENT: pid GAP 10 (next sample: pid 222)')
pid GAP 10 as a holder name is the exact artefact this removes — a process that does not exist, presented as the owner of the device.
And the third case is armed separately: assert "stalled" not in after_gap — a fault after a stall must be attributed normally, not called stalled. That is the arm that makes this a correction rather than a widening.
Why this belongs on this tool specifically
iommu_fault_attribution.py is the file that decides whether a fault had a holder, and tonight it produced my own two worst attributions (a 50 s stale sample called confident; a fault across the 22:55 reboot). A stall is the sampler admitting it did not look — the same NO SAMPLE vs NOTHING HELD distinction the whole directory runs on, now applied to the one row that used to be mistaken for a sample.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6a69610: READY
This corrects a defect I reviewed past, and the correction is measurable on the live capture: the old direction called 57 faults stalled, this one 51.
The defect: a GAP row is written when the sampler resumes ( GAP 0), so a row at E with length L describes the span (E-L, E] — before its own timestamp. #2623's version read the row as a sample at E, which does two wrong things: a fault inside a stall gets no stall verdict, and a fault after one is called stalled because the row sits just before it.
The span is right, and I checked both sides: with a 93 s gap row at E=1010, a fault at 1005 lies in (917, 1010] → NO SAMPLE: … stalled … (a 93s gap ending at 1010); a fault at 1015 does not. Mutation — flipping the span to (E, E+L]:
self-test rc=1 (1 failure)
Restored, self-test ok.
On the live capture (140 GAP rows in samples.log, 3 in attributed.log — the sampler writes them now, so the path is exercised):
fixed (span before the row): 51 NO SAMPLE
old direction (span after): 57 NO SAMPLE
So six faults on this capture were being called stalled that were not — they arrived after sampling resumed. The fix removes exactly those and keeps the rest, which is the direction a correctness change should move a count.
And it is my miss as much as anyone's: my review of #2623 said the reader "treats a GAP row as the absence of an observation, not a process called GAP" — which was true, and I checked it with a fault inside a stall. What I did not ask was which side of the row the stall is on, and that is the half that produces a false accusation (a working interval called stalled) rather than a missing one. The arm added here (inside / after_gap / the hand sequence) is the check I should have asked for.
Not verified: a capture long enough for the sampler to have stalled and recorded a fault on both sides of the same GAP row — the synthetic arms cover that shape, and the live capture only has the aggregate difference.
Refs #2478, #2542. This is my review of #2623 (CHANGES at 310206c and 0193a29), which merged without this point addressed.
The defect
accel_sampler.sh(#2624) writes a GAP row when the loop resumes:printf '%s\tGAP\t%s…' "$NOW" "$((NOW - LAST))". So a row at epochEwith lengthLdescribes the stall(E-L, E], before its own timestamp.attribute()treated it as "the last sample before the fault", which is true only for faults after the stall ended. On main (900 holder 111 | 910 GAP 10 | 912 holder 222, max_gap 15):The first line names a holder called GAP, which #2623's own commit title rules out.
The fix
E - L < t ≤ E;Evidence
--self-testpasses. The fixture that pinned the old reading (a fault at 1015 after the GAP at 1010, asserted "stalled") now asserts the stall at 1005 and no stall at 1015. A three-row timeline asserts that no verdict names GAP, at 899.5 / 905 / 911.NOTHING HELDat 1005;HANDOVER-ADJACENT: pid GAP 10at 911.tools/copyright.py --checkgives rc 0.Overlap: #2640 (open, held by a CHANGES) edits
attribute()in the same file to add a distance to each verdict. Whichever lands second has a small conflict in the before/after block. I'll rebase this one if #2640 goes first.🤖 Generated with Claude Code