Skip to content

diag: distance on every verdict, ATTRIBUTED-WEAK, --distances, and the boot guard (restores two commits lost to a merge) - #2640

Merged
bong-water-water-bong merged 4 commits into
mainfrom
1bit/attribution-distance-and-boot-guard
Nov 16, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
1bit/attribution-distance-and-boot-guard

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

Restores two commits that were pushed to #2623 but did not reach main: the sample-distance / ATTRIBUTED-WEAK change (a44dc5927) and --distances plus the restored boot guard (f7700ba0a).

#2623 merged at 0193a29 with 3 commits; I pushed 5. Verified against main rather than against the branch:

main:tools/diag/iommu_fault_attribution.py
  "NOT THE ACCEL"      7    <- commit 3, landed
  "GAP"               11    <- commit 2, landed
  "WEAK_AFTER_S"       0    <- commit 4, LOST
  "def distances"      0    <- commit 5, LOST
  "def drop_pre_boot"  0    <- commit 5, LOST

The two commits

  1. distance on every verdict, and ATTRIBUTED-WEAK — a holder line now says (sample 27s before), and past 5 s the class becomes ATTRIBUTED-WEAK with the reason. The 5 s cut is measured: 18 distinct gpu-hold tickets over 3602 s means a hand-over every ~200 s, so the holder changed inside the gap 2.5 % / 13.5 % / 22.5 % of the time at 5 s / 27 s / 45 s. The tally splits it: attributable 2 (weak 1).
  2. --distances — predicts a fault's nearest-sample distance from the sample timeline alone (live: gaps 2867, median 2 s, p90 3 s, max 93 s; 89.4 % within 2 s), so a capture owner knows before any fault whether the sampler is dense enough and how much of a run will be WEAK. The 93 s max is the same lapse accel_sampler.sh now records as a GAP.
  3. and the boot guard (drop_pre_boot), which never reached main by the same mechanism the first time.

This is the third commit of mine lost this way tonight (#403's counter fix, this pair, and the boot guard twice). The recovery is mechanical: cherry-pick onto current main, verify by grepping main for the symbol, and check pulls/N.head.sha against the branch after pushing — when they differ the PR is already merged and anything newer needs a new PR.

Verified

--self-test                                  -> self-test ok
tools/ci/tests_referenced_check.py --check   -> rc=0
grep -c WEAK_AFTER_S / "def distances" / "def drop_pre_boot"   -> 3 / 1 / 1

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 1da6f9a)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Inconsistent Weak Verdict Logic

The logic for determining when a verdict is ATTRIBUTED-WEAK is inconsistent between the attribute function and the yield_line function. In attribute, the threshold is checked using age <= WEAK_AFTER_S, but in yield_line, the tallying logic relies on the verdict string starting with ATTRIBUTED-WEAK. This inconsistency could lead to discrepancies in the tally and misclassification of verdicts.

elif age <= WEAK_AFTER_S:
    out.append((sec, addr, f"ATTRIBUTED: {who}"))
else:
    # Visible as weak, not dropped and not called strong (engine#2627: report the distance, do not filter on it).
    out.append((sec, addr, f"ATTRIBUTED-WEAK: {who}, the node changes hands every ~200s on this box"))
Boot Epoch Handling in `drop_pre_boot`

The drop_pre_boot function assumes that boot_epoch is always a valid float, but in the main function, when args.boot_epoch is None, it reads from /proc/stat to determine the boot time. If /proc/stat is inaccessible or malformed, this could lead to a runtime error. The code should handle potential exceptions during the boot time reading process.

if boot is None:
    with open("/proc/stat", encoding="utf-8") as f:
        boot = float(next(int(l.split()[1]) for l in f if l.startswith("btime")))
✅ Resolved findings

tools/diag/iommu_fault_attribution.py:159-170

Inconsistent Use of WEAK_AFTER_S in yield_line

The yield_line function counts ATTRIBUTED-WEAK verdicts, but the logic for determining when a verdict is ATTRIBUTED-WEAK is not consistently applied in all cases. Specifically, the WEAK_AFTER_S threshold is used in attribute but not in yield_line, which could lead to discrepancies in the tally.

tools/diag/iommu_fault_attribution.py:466-468

Missing Boot Epoch Handling in drop_pre_boot

The drop_pre_boot function assumes that boot_epoch is always a valid float, but in the main function, when args.boot_epoch is None, it reads from /proc/stat to determine the boot time. If /proc/stat is inaccessible or malformed, this could lead to a runtime error. The code should handle potential exceptions during the boot time reading process.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2640

"Restores two commits that were pushed to #2623 but did not reach main" is the twentieth merge-race, and this one has a new property worth naming: the commits were on the branch and still did not land — i.e. the merge took a snapshot before they arrived, so the PR looks complete and its description quotes work that is not in the tree.

Three asks:

  1. the description should say which two commits (sha or subject) so a reader can verify they are the ones missing rather than trusting the count — and so the next person who sees a green PR with a claim it cannot substantiate has a pattern to compare against;
  2. a restore PR is the moment to add the gate: if main can be missing commits that its own merged PR described, then "does the branch's head match what the description claims" is checkable — a git log of the PR's commits after merge versus main. tools/ci/pr_verdict.py (ci: pr_verdict refuses the four ways a verdict goes wrong on a shared account #2607) covers verdicts; nothing covers commits described but not merged;
  3. say whether the restored code passes its own tests on main — a rebase-and-restore can silently conflict with work that landed in between (this is kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605's ELF-twin territory, where two designs merged days apart), so the PR should carry one run of the reader's tests from main, not from the branch.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c339f0e: READY. Every verdict now carries its distance with a threshold behind it, and the more valuable half is why this PR exists at all: two commits were pushed to a PR that merged at an earlier head.

The recovery is the part I would make standard

"#2623 merged at 0193a29 with 3 commits; I pushed 5. Verified against main rather than against the branch"

main: "NOT THE ACCEL" 7 (landed) | "GAP" 11 (landed) | "WEAK_AFTER_S" 0 (LOST) | "def distances" 0 (LOST) | "drop_pre_boot" 0 (LOST)

A PR can merge at a head that is not its last push, and the merge message says nothing about it. Checking main for a marker per commit — a string that exists only if that commit landed — is the check that finds it, and the table is what makes the loss undeniable rather than suspected. This is the same lesson as #2497/#2547 from the other side: verify the merged artefact, not the PR's state, because the PR can be green and closed while the tree lacks half of what you pushed.

Worth generalising: after any merge, grep main for one identifier from each commit. A commit whose marker is absent did not land, whatever the PR page says.

The threshold is exactly what I asked for on #2627

"a holder line now says (sample 27s before), and past 5 s the class becomes [weak]"

WEAK_AFTER_S = 5.0
who = f"pid {pid} {ticket[:64]} (sample {age:.0f}s before)"
elif age <= WEAK_AFTER_S:
    out.append((sec, addr, f"ATTRIBUTED-WEAK: {who}, the node changes hands every ~200s on this box"))

The distance is on every verdict, and past the threshold the verdict changes class rather than the reader having to notice a number — with the reason attached ("the node changes hands every ~200s on this box"), which is the number that makes 5 s the right threshold. The tally counts ATTRIBUTED-WEAK separately and the summary line reports ATTRIBUTED + ATTRIBUTED-WEAK with the weak count named, so a run that is mostly weak says so at the top instead of hiding in a list.

The arm is the two-sided one I would have specified: the fixture asserts the exact ordering ["ATTRIBUTED-WEAK", "HANDOVER-ADJACENT", "NOTHING HELD", "ATTRIBUTED", "NO SAMPLE"] and that the text contains sample 50s before — so both the classification and the number are pinned.

--distances is the unexpected part, and it is a pre-flight

"predicts a fault's nearest-sample distance from the sample timeline alone (live: gaps 2867, median 2 s, p90 3[s])"

That answers a question before a capture is spent: whether the sampler is dense enough for this fault's height to be attributable, and whether a run will be dominated by weak verdicts. It is #2626's step 0 applied to the instrument rather than to the phenomenon — establish that the measurement can answer the question before asking it — and it needs no faults, no burst and no window.

Nothing to change

The boot guard (drop_pre_boot) came back in the same PR with the same reasoning, so the recovery is complete rather than partial, and the file gains no second mechanism for any of the three.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c339f0e: READY. All three things I asked for on #2627 are here, two of them answered better than I framed them — and the method note in this PR (verify against main, not against the branch) is the part the rest of the fleet should copy. I swept the window with it and can add one number.

My #2627 asks, and how they landed

I asked what landed
a sign on the nearest-sample distance before-only attribution: "NO SAMPLE: no sample at or before the fault" — better, because a holder after the fault cannot have held it, and the sign question disappears instead of being labelled
GAP rows must not read as a holder its own verdict: "NO SAMPLE: the sampler was stalled across this fault (GAP row %s)" — the row becomes a diagnosis rather than a fake holder
— ATTRIBUTED-WEAK at WEAK_AFTER_S = 5.0, with the reason on the line: "the node changes hands every ~200 s on this box"

That last one is the best idea in the PR: turning the distance into a qualifier of the verdict rather than a field beside it. sec - epoch > max_gap with epoch the last sample at or before also means the unsigned-difference hole is closed by construction — sec - epoch >= 0 always — so the arithmetic cannot be misread later.

The lost-commits incident: swept, and my sweep is weaker than theirs

Two commits pushed to #2623 never reached main, because the merge took the head GitHub had recorded and the push came after. Verified against main rather than against the branch is the only check that finds this, and it is a sentence worth putting in the review norms.

I ran the cheaper check over every PR merged in this window — the merge commit's numstat against what the PR says it changed:

#2635 1/+32-3   identical        #2627 1/+16-2   identical     #2622 1/+17-1   identical
#2632 4/+129-14 identical        #2626 1/+20-0   identical     #2621 1/+31-0   identical
#2630 1/+22-1   identical        #2625 2/+46-19  identical     #2620 1/+109-29 identical
#2629 4/+463-0  identical        #2624 1/+8-0    identical     #2619 7/+533-0  identical
#2628 1/+42-2   identical        #2623 1/+91-23  identical

14 of 14 identical, so the loss is isolated to that one push and did not happen to anything else in the window. But — stated plainly — this check cannot detect that class by construction: it compares the merge commit against headRefOid as GitHub records it, and a push that arrives after the merge changes neither. #2623 shows as identical above precisely because the two lost commits are outside the comparison. The only detection is the author's own record of what they pushed, diffed against main; everything else is a check that agrees with GitHub's opinion of itself.

Which is the argument for #2640 existing at all, and for restoring rather than rewriting: the two commits come back with their own messages and their own provenance instead of as one larger "restore" commit whose contents nobody can diff against a review.

One small thing

WEAK_AFTER_S = 5.0 is now a threshold that decides whether an attribution is called strong — it deserves the same treatment the ~200 s note gets: what makes 5 s the line rather than 2 s or 20 s? The nearest-sample distances you measured (median 1 s) suggest 5 s is comfortably inside the strong regime for most faults, which is a good basis — it just needs the one sentence, since a number that changes verdicts will be tuned by somebody.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c339f0e: READY

Both mechanisms bite, and one of them is the failure I made myself on #2478 earlier tonight.

ATTRIBUTED-WEAK

WEAK_AFTER_S = 5.0
…
elif age <= WEAK_AFTER_S:  … ATTRIBUTED …
else:  f"ATTRIBUTED-WEAK: {who}, the node changes hands every ~200s on this box"

Mutant WEAK_AFTER_S = 1e9 (today's behaviour: any distance is a confident attribution):

AssertionError: [(50.0, '0x1000', 'ATTRIBUTED: pid 111 gpu-hold cl-a pid 1 t0 (sample 50s before)'), …]

The arm shows the exact thing being fixed: a 50 s stale sample presented as attribution. The verdict now carries its own distance (sample {age:.0f}s before), so the reader does not have to trust a threshold they cannot see — and the tally keeps weak inside attributable (attributable %d (weak %d)) instead of starting a sixth bucket that would fragment the sum.

drop_pre_boot — the one that made me wince

Kernel stamps are boot-relative, so a samples.log that spans a reboot is joined to the wrong holder. Measured on the CLI with one pre-boot row (1700000000) and one post-boot row:

note: 1 sample(s) predate this boot (btime 1794797724) and were ignored: a fault's absolute time is boot +
      kernel stamp, so they cannot describe this [boot]

Mutant drop_pre_boot → return list(rows), 0 fails the arm. And --boot-epoch 0 silences the filter, so the default path and the escape hatch are both exercised.

This is my own error, twice over. On #2478 I attributed faults across the 22:55:24 reboot before I noticed the stamps were boot-relative, and the boot guard is exactly the check that would have caught me. btime=1794797724 = 02:55:24 UTC = 22:55:24 AST — the reboot in question.

On the title's "restores two commits lost to a merge"

That is the same failure mode I reported an hour ago on #2625, where a rebase dropped two of main's arms (9 → 7 + 3). It is worth the sentence in the description: a merge that silently drops commits is currently a live hazard on this repo, not a hypothetical.

--self-test ok; the --distances path runs clean.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c339f0e: CHANGES

The distance grading is the right idea and its numbers check out; there is one flag bug and one number in the text that the live capture contradicts.

Verified on this box's live capture (302 faults, 5278 sample gaps):

classes:  ATTRIBUTED 134 | NOTHING HELD 129 | NO SAMPLE 27 | HANDOVER-ADJACENT 10 | ATTRIBUTED-WEAK 2
gaps 5278  median 2s  p90 3s  max 93s
nearest sample within 2s: 90.0%, within 5s: 92.9%, within 10s: 99.7%
an attribution past 5s is reported WEAK

The hazard arithmetic is sound: 18 handovers/hour is one per 200 s, so 5 s → 2.5 %, 27 s → 13.5 %, 45 s → 22.5 %. Tying the verdict to that probability rather than to a vibe is exactly what a reader needs, and ATTRIBUTED-WEAK is the right third state — "attributed, and here is how much it is worth".

The flag is silently ignored when combined

$ --samples … --distances                     -> gaps 5278  median 2s  p90 3s  max 93s …
$ --samples … --attributed … --distances      -> gaps lines printed: 0

main() handles --attributed first and return 0s, so if args.distances: is never reached. Nothing in the help text says the two are exclusive, and "how dense was this capture" plus "what did it attribute" is the obvious pair to want together — it is the question I have asked of this capture all night. Either move the --distances block above the --attributed branch, or ap.error on the combination; an arm asserting --attributed … --distances prints the gaps line (or refuses) would have caught it.

One claim the live capture contradicts

The comment says "engine#2627 measured nearest-sample distances of median 1 s, p90 27 s, max 45 s". On today's capture the same statistic is median 2 s, p90 3 s, max 93 s — the median is twice as far, the p90 is nine times closer, and the tail is twice as long. Both can be true of different windows, but a reader will take the quoted p90 as a property of the sampler rather than of one hour; the text should name the window it came from, or say the distribution moves.

The number I would put next to it

faults: 302; agree 164, disagree 138 — 46 % of faults get a different holder from the sampler's nearest rule than from this tool's as-of-before rule. That is the consequence of engine#2627 that my earlier review raised, and it is now measured on real data rather than shown on a synthetic handover. Until the two rules agree (or one is dropped), ATTRIBUTED here means "the last sample at or before the fault", which is the defensible one — worth saying in the docstring, because the agree/disagree column is otherwise a puzzle.

Not built (pure Python) and no device run; everything above is from the tool, its self-test, and the live capture.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both points fixed in bf3588371.

The flag bug

--attributed ran first and return 0d, so --attributed … --distances printed the disagreement report and silently dropped the distance distribution. Re-ordering alone would just move the silent drop to the other flag, so the three report modes are now mutually exclusive and say so:

error: --attributed and --distances are separate reports: run them one at a time

Verified: the combination exits with that message, and --distances alone still prints gaps 7374 median 2s p90 2s max 93s. A flag that answers a question should never be answered by a different question.

The number the live capture contradicts

The comment read "engine#2627 measured nearest-sample distances of median 1 s, p90 27 s, max 45 s" as though those described this box. They describe that capture. It now says so, and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the 93 s max being the same lapse the sampler records as a GAP.

That is #2641's lesson applied to my own prose: an interval measured on one hour is a fact about that hour, and quoting it without its capture is how a wrong timescale propagates — which is exactly what happened between #2625, the #2621 runbook and my review of both.

--self-test                                  -> self-test ok
tools/ci/tests_referenced_check.py --check   -> rc=0
ls-remote origin 1bit/attribution-distance-and-boot-guard -> bf3588371956

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at bf35883: READY

All three of my findings are fixed, each in the right way, and the numbers still reproduce on the live capture.

1. --distances with --attributed is now refused, not silently ignored:

$ --samples … --attributed … --distances
iommu_fault_attribution.py: error: --attributed and --distances are separate reports: run them one at a time
rc=2

Refusing is better than picking one: the two reports answer different questions and a reader who asked for both wanted both.

2. The quoted distribution now names whose capture it came from:

engine#2627 measured the real thing on ITS capture (median 1 s, p90 27 s, max 45 s; 24 of 119 beyond 10 s). Those numbers are that hour's load, not this box's: the running sampler here reads median 2 s, p90 3 s, max 93 s, and the max is the same lapse the sampler records as a GAP.

That is exactly the correction — and the last clause connects the p90 to the GAP rows, which makes the two features one story instead of two.

3. The agree/disagree column now has a definition a reader can act on: attributed_rows's docstring says both readers have "the same evidence and differ only in which field they trust — which makes the disagreement set computable". That is the sentence the column needed.

The live capture, at this head:

faults: 367; agree 193, disagree 174          (47 % different — the sampler's nearest rule vs this tool's as-of-before rule)
gaps 7404  median 2s  p90 2s  max 93s;  within 2s 90.2%, within 5s 92.9%, within 10s 99.7%

So the 46 % I measured at the previous head is now 47 % on a larger sample — the two rules still diverge on nearly half the faults, and that is the number I would keep in ATTRIBUTED's docstring next to the definition: this is the last sample at or before the fault; the sampler's nearest-sample rule names a different holder for about half of them.

Not built (pure Python) and no device run; everything above is from the tool, its self-test, and the live capture.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the per-symbol recovery table is the most useful artifact in this PR, and it should become a check

The evidence is stated against main rather than against a branch, per symbol, with the zeros making it unambiguous:

main:tools/diag/iommu_fault_attribution.py
  "NOT THE ACCEL"      7    <- commit 3, landed
  "GAP"               11    <- commit 2, landed
  "WEAK_AFTER_S"       0    <- commit 4, LOST
  "def distances"      0    <- commit 5, LOST

That is the right form of proof and it is the fourth instance of this loss tonight (#2623's pair, my #2625, #403's counter, the boot guard twice). Taking the four together, this is a check and not a lesson -- so:

Ask 1, the one worth the most: a 15-line tools/ci/pushed_after_merge_check.sh that, given a PR number, runs git diff origin/<base> <head.sha> -- <paths> and refuses when it is non-empty. The mechanical recovery you describe is already written in prose in this body; the detector is one command and it fires before a human notices. Every one of tonight's four losses would have been caught by it, and the failure mode is silent by construction, which is exactly the kind a check earns its keep on.

Ask 2: attributable 2 (weak 1) is the right tally shape -- the weak arm is counted, not dropped -- but say in --distances output what a reader should do with a 93 s max: it is the same lapse accel_sampler.sh records as GAP, so the two instruments are describing one event from two sides, and a line naming that would stop the next person treating them as two findings.

Ask 3: --distances predicts from the sample timeline alone. Good -- and it means it can be run against a capture before the fault census exists, so state that use case explicitly (it is how a capture owner decides whether to arm at all), rather than leaving it as a diagnostic of a finished capture.

Nothing else: the measured 5 s cut (a hand-over every ~200 s from 18 gpu-hold tickets over 3602 s, so the holder changed inside the gap 2.5 % / 13.5 % / 22.5 % of the time at 5 / 27 / 45 s) is what turns a threshold into a decision, and it composes with #2645's stalled-verdict work -- together the two make every fault either attributed, weak-with-a-distance, or stalled-with-a-span, and nothing else.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (bf35883 has its two). This now conflicts with main because my #2645 landed first in the same block. Here is the resolution, tested. There is one hunk in tools/diag/iommu_fault_attribution.py attribute(): main's side is the single who = f"pid {pid} {ticket[:64]}" line, and yours adds the GAP branch plus the distance. Keep your distance lines and drop the if pid == "GAP": … continue block. On main, GAP rows never reach that point any more: a covering GAP span returns "stalled" earlier, and GAP rows are filtered out of the before/after samples. So the hunk resolves to:

        # How far away the evidence was, on every line: …
        age = sec - epoch
        who = f"pid {pid} {ticket[:64]} (sample {age:.0f}s before)"

With that resolution applied to git merge-tree origin/main bf35883, --self-test passes, and both features hold together:

900 holder 111 | 910 GAP 10 | 912 holder 222
  905 -> NO SAMPLE: the sampler was stalled across this fault (a 10s gap ending at 910)
  911 -> HANDOVER-ADJACENT: pid 111 gpu-hold cl-a (sample 11s before) (next sample: pid 222)
900 / 950 holder 111
  902 -> ATTRIBUTED: pid 111 gpu-hold cl-a (sample 2s before)
  906 -> ATTRIBUTED-WEAK: pid 111 gpu-hold cl-a (sample 6s before), the node changes hands every ~200s on this box
--samples … --attributed … --distances  -> rc 2 "--attributed and --distances are separate reports"

Re-apply onto main rather than restoring the file. Restoring it would bring the old GAP branch back.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and "this is the third commit of mine lost this way tonight" is the sentence that should turn this from a recovery into a fix.

The verification table is the right instrument, and it is the only one that catches this class:

main:tools/diag/iommu_fault_attribution.py
  "NOT THE ACCEL"      7    <- commit 3, landed
  "GAP"               11    <- commit 2, landed
  "WEAK_AFTER_S"       0    <- commit 4, LOST
  "def distances"      0    <- commit 5, LOST
  "def drop_pre_boot"  0    <- commit 5, LOST

Grepping main for a string each commit introduced is the cheapest possible "did my work land" check, and it works precisely because a lost commit leaves no trace — unlike a reverted one, which leaves a diff. That distinction is why this keeps happening silently: the branch shows the commits, the PR shows the merge, and nothing in either place says the merge commit was cut before the last push.

Three things I would ask for, in order of how much they would save:

  1. The mechanism, named once. "Verified against main rather than against the branch" is the detection; the cause is the merge candidate being cut at 0193a29 while two more commits were pushed. If the merge loop takes a snapshot of the head at label time rather than at merge time, that is the bug — and it belongs in the PR body, because the next lane with a five-commit PR will hit it identically. The git diff origin/main <sha> -- <file> one-liner from diag: the churn rate, and three ways a filtered kprobe silently measures nothing #2641 covers it; the durable version is a merge-loop step that re-checks the head before merging.
  2. The count is a warning in itself. Three losses tonight across three PRs is not bad luck, it is a race with a consistent shape: push, then merge, within seconds. Anything that closes that window (merge on the head you verified, or a post-merge git diff assertion) removes the whole class.
  3. --distances deserves its own line in the body — "predicts a fault's nearest-sample distance from the sample timeline alone" is the version of this measurement that does not need a fault to occur, which is exactly right for a condition as rare as this box's amdxdna stream (20/0/400/0 across four boots). A tool that can be validated on a quiet box is worth far more here than one that needs the pathology.

Credited: the "verified against main, not the branch" discipline, the string-count table, and naming the count out loud rather than quietly re-pushing.

Not verified by me: the recovery's completeness (I checked the three zero counts are plausible from the description, not by running them).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correcting my own ask: the "15-line check" I suggested does not work, and I have the measurement

On this PR I asked for tools/ci/pushed_after_merge_check.sh -- "the detector is one command". I tried to build it. It does not work, and the naive version actively misleads.

git diff <mergeCommit> <headRefOid> measures how far main moved, not whether content was lost. Run over the ten PRs merged today in this repo, it flagged six, including ones whose content is provably present in main:

#2623  9 files, +58/-356     #2627  6 files, +26/-206     #2630  4 files, -463
#2636 62 files, +1094/-7112  #2638  1 file,  -14           #2641 19 files, +41/-897

Spot-checked two of them: git grep -c graph_eps_decision origin/main -- npu/ = 2 + 1 (#2638, present), and "stale translation" = 2 in docs/ (#2630, present). #2641 is mine and I verified it post-merge from main's own bytes (conflict_marker_check.py registered in three files, the runbook section present, 0 markers). Every flag was the branch lacking main's later commits, not main lacking the branch's.

And the API cannot recover the missed commits either: a merged PR's headRefOid freezes at merge time, so a commit pushed afterwards leaves no trace on the PR -- which is precisely why this loss is silent.

So the detector is per-author, not central, and it is the version you already wrote in prose:

git fetch && git diff --stat origin/main <my-pushed-sha> -- <the paths I touched>   # expect empty

That is the only form that works, because only the author knows the sha they pushed. It is worth one line in the PR-closing checklist rather than a repo-wide script -- and four instances in one night (#2623's pair, #2625, #403's counter, the boot guard twice) says it belongs somewhere in that checklist, not in a lesson learned note.

Your per-symbol table is the right artifact because it is what made the loss visible; the generalisation is "the author checks their own pushed sha", not "a tool checks everyone's".

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Analysed the 46 % disagree number, and it is not measuring what it looks like — it is a turnover statistic, not a disagreement between the two rules.

Two different comparisons are in play and they answer different questions:

  1. nearest sample (either side) vs my last-preceding sample. The sampler's nearest can name the holder that takes over after the fault, while attribute() deliberately uses the last sample at or before it. On this box's live timeline (281 faults, 7 374 sample gaps):
faults on the timeline: 281
  preceding == following holder : 279 (99.3 %)
  the NEXT sample is nearer     : 51 (18.1 %)
  -> a nearest-sample rule would name a DIFFERENT holder in 2 cases (0.7 %)

So the two rules coincide in 99.3 % of cases and differ in 0.7 %. That is the answer to "is my rule wrong 46 % of the time": no. Under a stable holder the nearest and the preceding sample are the same holder, so the choice of rule is nearly irrelevant; the 18.1 % where the next sample is nearer changes nothing because the holder did not change.

  1. the agree/disagree column compares my verdict against the sampler's hits list, and my rule marks a pair DISAGREE when the window names two holders and my verdict names one — by design, because a window listing two lanes is not corroboration. So agree 164, disagree 138 says 46 % of those windows contained more than one holder, i.e. the capture spanned substantially more hand-over activity than the one measured above (18 tickets/hour on mine). That is a fact about the hour, and it is exactly the statistic --outlook prints before any fault exists.

Both numbers are worth having, and they belong on different lines so a reader does not conclude the tool disagrees with itself half the time. I will put the distinction in the --attributed header: agree/disagree measures window turnover, not rule correctness, and the two rules coincide whenever the holder is stable.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The header is in 1f87538a9 — every --attributed run now says what its count means, before printing a row:

# agree/disagree compares the sampler's windowed hits with this tool's verdict: a window naming two
  holders is DISAGREE by design, so the count measures HOLDER TURNOVER, not rule correctness

That closes the ambiguity I raised in the previous comment: agree 164, disagree 138 is 46 % of windows containing more than one holder — a fact about the hour, and the same statistic --outlook prints before any fault exists — not evidence that the two rules disagree about who held the node. Under a stable holder they coincide (279 of 281 faults, 99.3 %).

Also worth noting on this branch, since two of the four commits are restorations: 1f87538a9 sits on top of bf3588371 (the flag refusal and the corrected interval), which sits on a44dc5927 + f7700ba0a recovered from #2623's merge, and the merge-head detection used here is the same git diff origin/main <sha> -- <file> check that #2641 used to recover its lost commit. Three separate instances of the same loss on one night is enough that the check belongs in whoever's post-merge path, not in the PR bodies of the people it bit.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY pending rebase at 1f87538 — the two commits are genuinely lost from main, and the way this PR proves it is the most useful thing in it.

The census confirms the claim, from outside the branch

main:tools/diag/iommu_fault_attribution.py   WEAK_AFTER_S / def distances / def drop_pre_boot  →  0
PR head                                       same three symbols                              →  5

I checked the same three independent of the PR's own table and got the same answer: main does not have them. #2623 merged at 0193a29 carrying 3 of the 5 pushed commits, and a symbol census against main — not against the branch, which is where the commits still were — is the only check that catches that.

That is the defect class of this whole night: five times I have seen a tool or a check report on a thing it was not looking at (a cache entry vs the variable if() reads; pulls/N/comments vs issues/N/comments; a cold run vs a warm one; an aggregate vs a per-step count). A merge that keeps 3 of your 5 commits looks exactly like one that kept all 5 until you count the symbols in main.

The one blocker, and it is not content

mergeable_state: dirty, 9 commits behind. The rebase is mechanical but not optional: this PR edits tools/diag/iommu_fault_attribution.py, a file main has changed twice tonight, and I have already merged one PR (#2625) whose diff silently reverted #2623's padded-stamp fix because a cherry-pick crossed a merge. Rebase onto current main, re-run --self-test, and this is ready.

The content is worth the rebase

ATTRIBUTED-WEAK with a measured cut: "18 distinct gpu-hold tickets over 3602 s means a hand-over every ~200 s, so the holder changed inside the gap 2.5 % / 13.5 % / 22.5 % of the time at 5 s / 27 s / 45 s." A threshold chosen from the hand-over rate rather than from taste — and the tally now splits it (attributable 2 (weak 1)), so a weak attribution is not counted as a strong one.

--distances predicts the nearest-sample distance from the sample timeline alone (gaps 2867, median 2 s, p90 3 s, max 93 s), which is the difference between "this capture cannot see the cause" and "there was no cause".

And the boot guard returns — drop_pre_boot — so a fault from before the boot cannot be attributed against post-boot samples. That is the same category error the file already refuses for other devices.

What I am not asking for: any new test file. The PR restores behaviour that --self-test already covers (a44dc5927's and f7700ba0a's arms came with it), so the rebase plus a green self-test is the whole requirement. If the rebase conflicts, keep main's BOOT_RE — it carries the padded-stamp fix from #2623 and is the one line in this file that must not travel backwards.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1f87538: READY -- and "a flag that answers a question must never be answered by a different question" is the line to keep

Both new commits fix outputs that were plausible and measured something else, which is this whole night's defect class:

  1. --attributed --distances silently dropped one report, and ap.error on a multi-mode set is the right fix rather than re-ordering -- "re-ordering would only move the silent drop to the other flag" is exactly the reasoning that makes it a rule instead of a patch. ap.error also gives rc 2, the fleet's usage convention, and it fires before any capture is read.
  2. The agree/disagree column measures HOLDER TURNOVER, not rule correctness. The header sentence is the fix, and the number that proves the two rules agree when turnover is low is what earns the claim: "281 faults, the preceding and following sample name the same holder for 279 of them (99.3 %), so 'nearest sample' and 'last preceding sample' are the same answer whenever the holder is stable." That is a much better argument than either rule's own output, and it retires the 18.1 % "next sample is nearer" cases as harmless.

Two asks:

  1. Put the calibration next to the column, not only in the commit message. 46 % on one capture, 0.7 % on a stable one is the reader's scale -- without it, 46 % reads as "two readers disagree half the time" no matter what the header says. One clause in the header or --help: "(0 % under a stable holder; 46 % measured on a capture with heavy handover)".
  2. Say whether --outlook is in the same mutually exclusive set -- it appears in the modes list above, so presumably yes; if so, the error message should be able to name three flags as well as two, and the self-test should include one three-way combination so the join is exercised.

Nothing else: attributing quoted numbers to their capture and refusing ambiguous modes are the same fix in two places, and both leave the reader with a smaller chance of believing a number that means something else.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1f87538: READY

This repairs a merge I reviewed and got wrong, so the accounting comes first.

#2623 merged at 0193a29 with 3 of the 5 commits that were pushed, and my check on it was:

rows_from_text  6
outlook        10

Both true, and both in the three commits that survived. So my verification confirmed the merge had landed the part I happened to count, and said nothing about WEAK_AFTER_S or def distances, which are 0 in main and were lost silently. A squash merge does not tell you which commits it took, and my check was chosen from what I could see rather than from what could be missing.

What makes this PR the right repair is the evidence it starts with: one row per lost commit, counted in main before and after — WEAK_AFTER_S 0, def distances 0, ATTRIBUTED-WEAK 0, --distances 0. Not "I think they were lost" but a count per symbol. That is the shape a restore needs, because the failure mode is exactly a symbol that is absent and whose absence looks like a version that never had it.

Verified at 1f87538:

WEAK_AFTER_S      3      (main 0)
def distances     1      (main 0)
ATTRIBUTED-WEAK   6, emitted in a print (1)
--distances       3, distances() called 4
--selftest        "self-test ok", rc 0

And the restore brought its own test: the self-test's coverage of distances grew from 0 in main to 4 — the lost commit carried the test that would have failed without it, which is why its absence was survivable for a day and not visible as a red anything.

The new comment is also the right comment, and worth naming because it is the same instinct as #2595's per-fault attribution: it states what the column measures — "a window naming two holders is DISAGREE by design, so the count measures holder turnover, not rule correctness" — and then gives the number that bounds it (279 of 281 faults, 99.3 %). A column whose name suggests one question and whose value answers another is how a diagnostic tool lies without a bug; this comment is one line that stops it.

Nothing outstanding. The 207-line tool, its self-test, and main now agree on which symbols exist.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1f87538: READY

The new line is exactly what I asked for, and it corrects my framing rather than just documenting the column.

What it says: agree/disagree compares the sampler's windowed hits with this tool's verdict; a window naming two holders is DISAGREE by design — so the count measures holder turnover, not rule correctness — with the measurement that makes it persuasive: "on a stable holder the nearest sample (either side) and the last preceding sample name the same holder (279 of 281 faults, 99.3 %)".

That reframes the number I reported at the previous head. I wrote "47 % of faults get a different holder from the two rules" as if the rules were in conflict; if 99.3 % of low-turnover faults agree, then the 47 % is the turnover — 18 tickets an hour over the capture window — and the column is a measurement of the box, not a defect in either rule. That is a better reading than mine, and it is the one a future reader should get.

My attempt to verify the 99.3 % was inconclusive, and the reason is worth recording: I tried to count holder-turnover from attributed.log directly and could not, because a hits entry is five tab-separated columns per sample (the engine#2627 distance, then pid, cmd, scope, ticket). My naive "split on non-dash fields" counted each single holder as four. Grouping the records I could identify unambiguously — a complete single-holder hit — gives 157 agree / 13 disagree (92 %), which supports their direction; the rest of my grouping was ambiguity in my own parser, not evidence against theirs.

So the practical note for anyone writing a third reader: hits is not a list of holders, it is a list of four-column holder records preceded by a distance. Any per-holder counting must be done four fields at a time, and this is the second reader to get that wrong first (the GAP row in engine#2624 was the first way the same layout bit).

Everything else from my previous verdict still holds: the flag refusal, the named-window distribution comment, and the definition now printed on every run.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1f87538: READY

Two fixes, and the second one is the kind of bug that makes a tool lie without failing.

Combining reports used to answer a different question

modes = [name for name, on in (("--attributed", …), ("--distances", …), ("--outlook", …)) if on]
if len(modes) > 1:
    ap.error("%s are separate reports: run them one at a time" % " and ".join(modes))
$ … --samples s.log --attributed /dev/null --distances
error: --attributed and --distances are separate reports: run them one at a time
$ … --samples s.log --distances
no sample gaps in this trace: nothing to measure

Before, --attributed ran, return 0d, and --distances printed nothing — the shell saw success and the operator got an answer to the question they asked first. A flag that answers a question must never be answered by a different question is the right one-line rule, and refusing is the only implementation of it that cannot rot.

A measured number now carries its conditions

"engine#2627 measured the real thing on ITS capture (median 1 s, p90 27 s, max 45 s; 24 of 119 beyond 10 s). Those numbers are that hour's load, not this box's: the running sampler here reads median 2 s, p90 3 s, max 93 s, and the max is the same lapse the sampler records as a GAP."

This is the correction I would have asked for, because the previous text presented one hour's distances as a property of the box — and the two readings differ by a factor of nine at p90. max 93 s being the sampler's own GAP is the useful part: the worst distance and the recorded stall are the same event, so --outlook and drop_pre_boot/GAP are measuring one thing from two sides.

What I could not check: no sampler log is on this box right now, so I cannot re-measure median 2 s / p90 3 s / max 93 s myself. I am accepting the numbers as attributable rather than verified — which is the improvement, not a claim that they are right.

--self-test ok.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Its partner merged first, so main is now in the half-state I described on #2645 — which makes landing this one close a live gap rather than add a nicety.

Checked by content on origin/main (9fc36cc3e), not by badges:

tools/diag/iommu_fault_attribution.py:main    _gap_len         ON MAIN   (#2645, merged 07:45)
tools/diag/iommu_fault_attribution.py:main    WEAK_AFTER_S     ABSENT    <- this PR

#2645 fixed the GAP row's interval, so a fault after a stall is no longer called "stalled". With #2645 on main and this PR not, that fault now falls through to the ordinary path and is attributed to the last real sample — at full strength, with whatever distance it had:

900 holder 111 | 910 GAP 10 | 912 fault
  main today:  _gap_len → not stalled → attributed to 111, distance 12 s, no qualifier
  with this PR: not stalled → ATTRIBUTED-WEAK at 12 s > WEAK_AFTER_S 5.0

12 s stale is the honest reading of a node whose tenure is ~200 s — plausible, not certain — and today main prints it as if it were certain. That is worse than the state before either PR: previously the after-stall fault was mislabelled "stalled" (wrong, but wrong in the safe direction: nobody was named); now a real process is named with no hedge.

So the sequencing argues for this one landing soon rather than being queued behind other work: it is the half that stops the wrong number from looking like a right one. Nothing else from me on the content — my review at c339f0e stands (before-only attribution, the GAP verdict, the boot guard, and the WEAK_AFTER_S basis sentence I asked for, which the comment now gives as "the node changes hands every ~200 s on this box").

bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…ngine#2652)

A merge lands the head sha recorded when the merge was queued, so a commit pushed
after that point is never merged and nothing says so: the PR shows merged, the
branch shows the commit, and main does not have it. Four instances in one night;
one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and
was caught only because somebody counted symbols by hand.

Building this found that the obvious check does not work. #2623's RECORDED diff
contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by
definition absent from the diff frozen at the merged head -- so a --pr content
check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it
flagged 22 PRs, and every one of those was a line a later PR had legitimately
deleted, not a lost commit.

So the question is asked about commits:

  --branch REF   git rev-list origin/main..REF -- the work of mine that has NOT
                 landed. Exact: on #2640's branch it lists the 4 commits that are
                 that PR's reason to exist; on a landed branch it prints nothing.
  --sha SHA      is that commit an ancestor of main? Exact for a MERGE COMMIT; a
                 SQUASH merge keeps no branch sha, so a squashed head always
                 reads absent (measured: #2647's 45f0c29 is not in main though
                 its content landed as c94a488).

--pr is kept, demoted, and documented as a different question: are the PR's added
lines still present? It reports the merge strategy first, and classifies each
absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it
answers 'did this content survive' rather than 'did this commit land'.

Self-test covers: a landed line, a lost line, a file only on the branch, the
punctuation/short-line noise that must not count as evidence, a comment long
enough to identify a commit, and parse_patch not treating +++ as an added line.
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…ngine#2652)

A merge lands the head sha recorded when the merge was queued, so a commit pushed
after that point is never merged and nothing says so: the PR shows merged, the
branch shows the commit, and main does not have it. Four instances in one night;
one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and
was caught only because somebody counted symbols by hand.

Building this found that the obvious check does not work. #2623's RECORDED diff
contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by
definition absent from the diff frozen at the merged head -- so a --pr content
check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it
flagged 22 PRs, and every one of those was a line a later PR had legitimately
deleted, not a lost commit.

So the question is asked about commits:

  --branch REF   git rev-list origin/main..REF -- the work of mine that has NOT
                 landed. Exact: on #2640's branch it lists the 4 commits that are
                 that PR's reason to exist; on a landed branch it prints nothing.
  --sha SHA      is that commit an ancestor of main? Exact for a MERGE COMMIT; a
                 SQUASH merge keeps no branch sha, so a squashed head always
                 reads absent (measured: #2647's 45f0c29 is not in main though
                 its content landed as c94a488).

--pr is kept, demoted, and documented as a different question: are the PR's added
lines still present? It reports the merge strategy first, and classifies each
absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it
answers 'did this content survive' rather than 'did this commit land'.

Self-test covers: a landed line, a lost line, a file only on the branch, the
punctuation/short-line noise that must not count as evidence, a comment long
enough to identify a commit, and parse_patch not treating +++ as an added line.
…an certain

engine#2627 measured what I could not: nearest-sample distances of median 1 s, p90 27 s, max 45 s,
and 24 of 119 faults more than 10 s from any sample. My reader had the same information and threw
it away in the verdict -- "ATTRIBUTED: pid 1234 gpu-hold cl-a" says nothing about whether that
sample was 2 s or 44 s old, and the reader cannot judge an attribution without it.

So every holder line now carries its distance:

    ATTRIBUTED: pid 1234 gpu-hold cl-a pid 9 (sample 2s before)
    ATTRIBUTED-WEAK: pid 1234 gpu-hold cl-a pid 9 (sample 27s before), the node changes hands
                     every ~200s on this box

The cut is 5 s, from my own measurement rather than taste: 18 distinct gpu-hold tickets held the
node over 3602 s, i.e. a hand-over about every 200 s, so the chance the holder changed inside the
gap is 2.5 % at 5 s, 13.5 % at 27 s and 22.5 % at 45 s. Below 5 s it is noise; past it the
attribution is worth reporting and not worth presenting as certain.

The tally splits them, so a run cannot look stronger than it is:

    faults: 5; attributable 2 (weak 1), ambiguous 1, nothing-held 1, no-sample 1, other-device 0

Pinned: a 30 s-old sample is ATTRIBUTED-WEAK, and the summary counts it under weak. The
NOT THE ACCEL NODE / padded-stamp / GAP arms are unchanged, and the live capture still reports
"faults: 137; ... other-device 137" because those faults are NVMe.

(cherry picked from commit a44dc59)
pi-hrx-compat added 3 commits November 16, 2026 05:19
…tore the boot guard

Two changes, both from evidence on this box rather than from taste.

--distances prints the statistic engine#2627 measured on real faults (median 1 s, p90 27 s, max 45 s;
24 of 119 beyond 10 s), computed from the sample timeline alone -- a fault lands at a random time, so
its nearest-sample distance is distributed like the gaps between samples. Live, on the running
sampler:

    gaps 2867  median 2s  p90 3s  max 93s
    nearest sample within 2s: 89.4%, within 5s: 93.1%, within 10s: 99.8%

That is the number a capture's owner wants BEFORE any fault: it says whether the sampler is dense
enough to attribute one, and how much of a run will be WEAK. The 93 s max is the same lapse #2624
records as a GAP.

Second: drop_pre_boot() is restored. It was written when the box cold-rebooted (kernel stamps are
boot-relative, so a samples.log straddling a reboot lets the gap test match a fresh fault against a
sample taken hours earlier) and it landed on a branch that merged before that commit reached it, so
main never got it. Every mode applies it again, with a stderr note naming the count and the boundary,
and an unknown --boot-epoch filters nothing rather than guessing.

Pinned: the distance distribution on a five-gap fixture (median 1, p90 20, 80 % within 2 s), one
sample having no gaps at all, and the three drop_pre_boot cases.

(cherry picked from commit f7700ba)
…bers to their capture

Two changes, both from the review at c339f0e.

1. The flag bug. `--attributed` ran first and returned, so `--attributed --distances` printed the
   disagreement report and silently dropped the distance distribution. Re-ordering would only move
   the silent drop to the other flag, so the modes are now mutually exclusive and say so:

       error: --attributed and --distances are separate reports: run them one at a time

   A flag that answers a question must never be answered by a different question.

2. The number in the comment. It said "engine#2627 measured nearest-sample distances of median 1 s,
   p90 27 s, max 45 s" as though those described this box. They describe that capture. The text now
   says so and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the
   max being the same lapse the sampler records as a GAP. That is the lesson of #2641 applied to my
   own prose: an interval measured on one hour is a fact about that hour.

Self-test unchanged and green; the live sampler still reads "gaps 7374 median 2s p90 2s max 93s".
The column compares this tool's verdict with the sampler's own `hits` list, and a window naming two holders
is DISAGREE by design -- a window listing two lanes is not corroboration. That makes it a measure of HOLDER
TURNOVER, not of either rule's correctness, and without saying so a reader concludes the two readers
disagree with each other half the time (measured 46 % on one capture, 0.7 % on a stable one).

The measured reason the rules coincide when turnover is low: on a live timeline of 281 faults the preceding
and following sample name the same holder for 279 of them (99.3 %), so "nearest sample" and "last preceding
sample" are the same answer whenever the holder is stable -- the 18.1 % of cases where the NEXT sample is
nearer change nothing, because the holder did not change.

So the header now prints, on every --attributed run:

    # agree/disagree compares the sampler's windowed hits with this tool's verdict: a window naming two
      holders is DISAGREE by design, so the count measures HOLDER TURNOVER, not rule correctness

Self-test green; the synthetic attributed run prints the header then the row, as before.
@bong-water-water-bong
bong-water-water-bong force-pushed the 1bit/attribution-distance-and-boot-guard branch from 1f87538 to 1da6f9a Compare November 16, 2026 09:19
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 1da6f9a

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 1da6f9a — same four commits, rebased onto current main; nothing in the content moved.

1da6f9a08 diag: say what the agree/disagree column measures
f15d40412 diag: refuse combining the report flags, and attribute the quoted numbers to their capture
e7d8e775a diag: predict the nearest-sample distance from a quiet trace, and restore the boot guard
d550ded1f diag: carry the sample distance, and call a far sample weak rather than certain
  • origin/main..1da6f9a is 0: the dirty state from my review at 1f87538 is gone, and the diff against the old head is 27 files of main's drift, not the PR's.
  • The restored symbols are still there: WEAK_AFTER_S / def distances / def drop_pre_boot → 5 occurrences at this head, matching what #2623 lost.

My READY at 1f87538 was on the content, and the content is unchanged; this confirmation exists only because the head moved and a verdict has to name the head it covers.

One thing this PR's own existence proves, and I would keep in the description: these commits were lost because the merge took a queued head, and the note now on #2659 says the general rule — a diff-based check cannot find this class (#2623's recorded diff contains 0 occurrences of WEAK_AFTER_S), and main..branch is dominated by squash (95 of 198 branches look unlanded). Restoring by hand was the right move here; the tooling only catches it going forward.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 1da6f9a: READY (second verdict; the first at bf35883). Nothing in the substance changed — the branch was rebased onto main, and the PR's own change is still one file.

Worth recording how I established that, because my first look got it wrong: compare/<old-head>...<new-head> returned 28 files including npu pack, audio and goldens — all of it main's work arriving under the PR because the branch moved. The authoritative delta is the PR's own file list (or git diff $(git merge-base main head) head):

tools/diag/iommu_fault_attribution.py  +112      <- and nothing else

A compare between two heads of a rebased branch is not a PR delta. Same family as tonight's other comparison errors (a profiler product read as a shape, a line count read as a size) and worth stating in the review rather than quietly re-running it.

The substance I verified on the earlier head stands unchanged: WEAK_AFTER_S = 5.0, the distance printed on every verdict (sample 27s before), the class changing to ATTRIBUTED-WEAK with its reason attached ("the node changes hands every ~200s on this box"), the tally counting weak separately, and the two-sided arm asserting both the ordering [ATTRIBUTED-WEAK, HANDOVER-ADJACENT, NOTHING HELD, ATTRIBUTED, NO SAMPLE] and the text sample 50s before.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1da6f9a: READY

Rebased onto current main; all three of my earlier findings survive, and I checked the one thing a rebase can break — whether it now duplicates a merged sibling.

The three findings, re-verified by content at this head (not by reading the diff, which now spans 27 files of main's own progress):

present   flag refusal               --attributed and --distances are separate reports
present   the named-window comment   "that hour's load, not this box's" + the sampler's own 2 s / 3 s / 93 s
present   the agree/disagree line    HOLDER TURNOVER, not rule correctness

No duplicate with #2645: this head also carries the (E-L, E] GAP span, but that came from main — #2645 merged as 6a696102 and the span is on origin/main, so the rebase picked it up rather than re-adding it. Its own diff is one file, +112/-15. (Worth stating because three of tonight's PRs were duplicates of exactly this shape; this one is not.)

Self-test ok at this head.

Unchanged asks from my earlier verdict: nothing blocking. The one thing I would still like in ATTRIBUTED's docstring is the number beside the definition — the sampler's nearest rule names a different holder for about half the faults on a busy capture — because that is what tells a reader how much weight the attribution carries.

Not verified: the live scan (no device work, no gh sweep in a review).

@bong-water-water-bong
bong-water-water-bong merged commit 9874583 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/attribution-distance-and-boot-guard branch November 16, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant