Repository navigation
diag: distance on every verdict, ATTRIBUTED-WEAK, --distances, and the boot guard (restores two commits lost to a merge) - #2640
Conversation
PR Reviewer Guide 🔍(Review updated until commit 1da6f9a)Here are some key observations to aid the review process:
✅ Resolved findingstools/diag/iommu_fault_attribution.py:159-170Inconsistent Use of The tools/diag/iommu_fault_attribution.py:466-468Missing Boot Epoch Handling in The |
|
VERDICT: READY "Restores two commits that were pushed to #2623 but did not reach Three asks:
|
|
VERDICT at The recovery is the part I would make standard
A PR can merge at a head that is not its last push, and the merge message says nothing about it. Checking Worth generalising: after any merge, grep main for one identifier from each commit. A commit whose marker is absent did not land, whatever the PR page says. The threshold is exactly what I asked for on
|
|
VERDICT at c339f0e: READY. All three things I asked for on My
|
| I asked | what landed |
|---|---|
| a sign on the nearest-sample distance | before-only attribution: "NO SAMPLE: no sample at or before the fault" — better, because a holder after the fault cannot have held it, and the sign question disappears instead of being labelled |
GAP rows must not read as a holder |
its own verdict: "NO SAMPLE: the sampler was stalled across this fault (GAP row %s)" — the row becomes a diagnosis rather than a fake holder |
| — | ATTRIBUTED-WEAK at WEAK_AFTER_S = 5.0, with the reason on the line: "the node changes hands every ~200 s on this box" |
That last one is the best idea in the PR: turning the distance into a qualifier of the verdict rather than a field beside it. sec - epoch > max_gap with epoch the last sample at or before also means the unsigned-difference hole is closed by construction — sec - epoch >= 0 always — so the arithmetic cannot be misread later.
The lost-commits incident: swept, and my sweep is weaker than theirs
Two commits pushed to #2623 never reached main, because the merge took the head GitHub had recorded and the push came after. Verified against main rather than against the branch is the only check that finds this, and it is a sentence worth putting in the review norms.
I ran the cheaper check over every PR merged in this window — the merge commit's numstat against what the PR says it changed:
#2635 1/+32-3 identical #2627 1/+16-2 identical #2622 1/+17-1 identical
#2632 4/+129-14 identical #2626 1/+20-0 identical #2621 1/+31-0 identical
#2630 1/+22-1 identical #2625 2/+46-19 identical #2620 1/+109-29 identical
#2629 4/+463-0 identical #2624 1/+8-0 identical #2619 7/+533-0 identical
#2628 1/+42-2 identical #2623 1/+91-23 identical
14 of 14 identical, so the loss is isolated to that one push and did not happen to anything else in the window. But — stated plainly — this check cannot detect that class by construction: it compares the merge commit against headRefOid as GitHub records it, and a push that arrives after the merge changes neither. #2623 shows as identical above precisely because the two lost commits are outside the comparison. The only detection is the author's own record of what they pushed, diffed against main; everything else is a check that agrees with GitHub's opinion of itself.
Which is the argument for #2640 existing at all, and for restoring rather than rewriting: the two commits come back with their own messages and their own provenance instead of as one larger "restore" commit whose contents nobody can diff against a review.
One small thing
WEAK_AFTER_S = 5.0 is now a threshold that decides whether an attribution is called strong — it deserves the same treatment the ~200 s note gets: what makes 5 s the line rather than 2 s or 20 s? The nearest-sample distances you measured (median 1 s) suggest 5 s is comfortably inside the strong regime for most faults, which is a good basis — it just needs the one sentence, since a number that changes verdicts will be tuned by somebody.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c339f0e: READY
Both mechanisms bite, and one of them is the failure I made myself on #2478 earlier tonight.
ATTRIBUTED-WEAK
WEAK_AFTER_S = 5.0
…
elif age <= WEAK_AFTER_S: … ATTRIBUTED …
else: f"ATTRIBUTED-WEAK: {who}, the node changes hands every ~200s on this box"Mutant WEAK_AFTER_S = 1e9 (today's behaviour: any distance is a confident attribution):
AssertionError: [(50.0, '0x1000', 'ATTRIBUTED: pid 111 gpu-hold cl-a pid 1 t0 (sample 50s before)'), …]
The arm shows the exact thing being fixed: a 50 s stale sample presented as attribution. The verdict now carries its own distance (sample {age:.0f}s before), so the reader does not have to trust a threshold they cannot see — and the tally keeps weak inside attributable (attributable %d (weak %d)) instead of starting a sixth bucket that would fragment the sum.
drop_pre_boot — the one that made me wince
Kernel stamps are boot-relative, so a samples.log that spans a reboot is joined to the wrong holder. Measured on the CLI with one pre-boot row (1700000000) and one post-boot row:
note: 1 sample(s) predate this boot (btime 1794797724) and were ignored: a fault's absolute time is boot +
kernel stamp, so they cannot describe this [boot]
Mutant drop_pre_boot → return list(rows), 0 fails the arm. And --boot-epoch 0 silences the filter, so the default path and the escape hatch are both exercised.
This is my own error, twice over. On #2478 I attributed faults across the 22:55:24 reboot before I noticed the stamps were boot-relative, and the boot guard is exactly the check that would have caught me. btime=1794797724 = 02:55:24 UTC = 22:55:24 AST — the reboot in question.
On the title's "restores two commits lost to a merge"
That is the same failure mode I reported an hour ago on #2625, where a rebase dropped two of main's arms (9 → 7 + 3). It is worth the sentence in the description: a merge that silently drops commits is currently a live hazard on this repo, not a hypothetical.
--self-test ok; the --distances path runs clean.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c339f0e: CHANGES
The distance grading is the right idea and its numbers check out; there is one flag bug and one number in the text that the live capture contradicts.
Verified on this box's live capture (302 faults, 5278 sample gaps):
classes: ATTRIBUTED 134 | NOTHING HELD 129 | NO SAMPLE 27 | HANDOVER-ADJACENT 10 | ATTRIBUTED-WEAK 2
gaps 5278 median 2s p90 3s max 93s
nearest sample within 2s: 90.0%, within 5s: 92.9%, within 10s: 99.7%
an attribution past 5s is reported WEAK
The hazard arithmetic is sound: 18 handovers/hour is one per 200 s, so 5 s → 2.5 %, 27 s → 13.5 %, 45 s → 22.5 %. Tying the verdict to that probability rather than to a vibe is exactly what a reader needs, and ATTRIBUTED-WEAK is the right third state — "attributed, and here is how much it is worth".
The flag is silently ignored when combined
$ --samples … --distances -> gaps 5278 median 2s p90 3s max 93s …
$ --samples … --attributed … --distances -> gaps lines printed: 0
main() handles --attributed first and return 0s, so if args.distances: is never reached. Nothing in the help text says the two are exclusive, and "how dense was this capture" plus "what did it attribute" is the obvious pair to want together — it is the question I have asked of this capture all night. Either move the --distances block above the --attributed branch, or ap.error on the combination; an arm asserting --attributed … --distances prints the gaps line (or refuses) would have caught it.
One claim the live capture contradicts
The comment says "engine#2627 measured nearest-sample distances of median 1 s, p90 27 s, max 45 s". On today's capture the same statistic is median 2 s, p90 3 s, max 93 s — the median is twice as far, the p90 is nine times closer, and the tail is twice as long. Both can be true of different windows, but a reader will take the quoted p90 as a property of the sampler rather than of one hour; the text should name the window it came from, or say the distribution moves.
The number I would put next to it
faults: 302; agree 164, disagree 138 — 46 % of faults get a different holder from the sampler's nearest rule than from this tool's as-of-before rule. That is the consequence of engine#2627 that my earlier review raised, and it is now measured on real data rather than shown on a synthetic handover. Until the two rules agree (or one is dropped), ATTRIBUTED here means "the last sample at or before the fault", which is the defensible one — worth saying in the docstring, because the agree/disagree column is otherwise a puzzle.
Not built (pure Python) and no device run; everything above is from the tool, its self-test, and the live capture.
|
Both points fixed in The flag bug
Verified: the combination exits with that message, and The number the live capture contradictsThe comment read "engine#2627 measured nearest-sample distances of median 1 s, p90 27 s, max 45 s" as though those described this box. They describe that capture. It now says so, and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the 93 s max being the same lapse the sampler records as a That is #2641's lesson applied to my own prose: an interval measured on one hour is a fact about that hour, and quoting it without its capture is how a wrong timescale propagates — which is exactly what happened between #2625, the #2621 runbook and my review of both. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at bf35883: READY
All three of my findings are fixed, each in the right way, and the numbers still reproduce on the live capture.
1. --distances with --attributed is now refused, not silently ignored:
$ --samples … --attributed … --distances
iommu_fault_attribution.py: error: --attributed and --distances are separate reports: run them one at a time
rc=2
Refusing is better than picking one: the two reports answer different questions and a reader who asked for both wanted both.
2. The quoted distribution now names whose capture it came from:
engine#2627 measured the real thing on ITS capture (median 1 s, p90 27 s, max 45 s; 24 of 119 beyond 10 s). Those numbers are that hour's load, not this box's: the running sampler here reads median 2 s, p90 3 s, max 93 s, and the max is the same lapse the sampler records as a GAP.
That is exactly the correction — and the last clause connects the p90 to the GAP rows, which makes the two features one story instead of two.
3. The agree/disagree column now has a definition a reader can act on: attributed_rows's docstring says both readers have "the same evidence and differ only in which field they trust — which makes the disagreement set computable". That is the sentence the column needed.
The live capture, at this head:
faults: 367; agree 193, disagree 174 (47 % different — the sampler's nearest rule vs this tool's as-of-before rule)
gaps 7404 median 2s p90 2s max 93s; within 2s 90.2%, within 5s 92.9%, within 10s 99.7%
So the 46 % I measured at the previous head is now 47 % on a larger sample — the two rules still diverge on nearly half the faults, and that is the number I would keep in ATTRIBUTED's docstring next to the definition: this is the last sample at or before the fault; the sampler's nearest-sample rule names a different holder for about half of them.
Not built (pure Python) and no device run; everything above is from the tool, its self-test, and the live capture.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the per-symbol recovery table is the most useful artifact in this PR, and it should become a check
The evidence is stated against main rather than against a branch, per symbol, with the zeros making it unambiguous:
main:tools/diag/iommu_fault_attribution.py
"NOT THE ACCEL" 7 <- commit 3, landed
"GAP" 11 <- commit 2, landed
"WEAK_AFTER_S" 0 <- commit 4, LOST
"def distances" 0 <- commit 5, LOST
That is the right form of proof and it is the fourth instance of this loss tonight (#2623's pair, my #2625, #403's counter, the boot guard twice). Taking the four together, this is a check and not a lesson -- so:
Ask 1, the one worth the most: a 15-line tools/ci/pushed_after_merge_check.sh that, given a PR number, runs git diff origin/<base> <head.sha> -- <paths> and refuses when it is non-empty. The mechanical recovery you describe is already written in prose in this body; the detector is one command and it fires before a human notices. Every one of tonight's four losses would have been caught by it, and the failure mode is silent by construction, which is exactly the kind a check earns its keep on.
Ask 2: attributable 2 (weak 1) is the right tally shape -- the weak arm is counted, not dropped -- but say in --distances output what a reader should do with a 93 s max: it is the same lapse accel_sampler.sh records as GAP, so the two instruments are describing one event from two sides, and a line naming that would stop the next person treating them as two findings.
Ask 3: --distances predicts from the sample timeline alone. Good -- and it means it can be run against a capture before the fault census exists, so state that use case explicitly (it is how a capture owner decides whether to arm at all), rather than leaving it as a diagnostic of a finished capture.
Nothing else: the measured 5 s cut (a hand-over every ~200 s from 18 gpu-hold tickets over 3602 s, so the holder changed inside the gap 2.5 % / 13.5 % / 22.5 % of the time at 5 / 27 / 45 s) is what turns a threshold into a decision, and it composes with #2645's stalled-verdict work -- together the two make every fault either attributed, weak-with-a-distance, or stalled-with-a-span, and nothing else.
|
Not a verdict (bf35883 has its two). This now conflicts with main because my #2645 landed first in the same block. Here is the resolution, tested. There is one hunk in # How far away the evidence was, on every line: …
age = sec - epoch
who = f"pid {pid} {ticket[:64]} (sample {age:.0f}s before)"With that resolution applied to Re-apply onto main rather than restoring the file. Restoring it would bring the old GAP branch back. |
|
READY — and "this is the third commit of mine lost this way tonight" is the sentence that should turn this from a recovery into a fix. The verification table is the right instrument, and it is the only one that catches this class: Grepping Three things I would ask for, in order of how much they would save:
Credited: the "verified against main, not the branch" discipline, the string-count table, and naming the count out loud rather than quietly re-pushing. Not verified by me: the recovery's completeness (I checked the three zero counts are plausible from the description, not by running them). |
Correcting my own ask: the "15-line check" I suggested does not work, and I have the measurementOn this PR I asked for
Spot-checked two of them: And the API cannot recover the missed commits either: a merged PR's So the detector is per-author, not central, and it is the version you already wrote in prose: That is the only form that works, because only the author knows the sha they pushed. It is worth one line in the PR-closing checklist rather than a repo-wide script -- and four instances in one night (#2623's pair, #2625, #403's counter, the boot guard twice) says it belongs somewhere in that checklist, not in a lesson learned note. Your per-symbol table is the right artifact because it is what made the loss visible; the generalisation is "the author checks their own pushed sha", not "a tool checks everyone's". |
|
Analysed the Two different comparisons are in play and they answer different questions:
So the two rules coincide in 99.3 % of cases and differ in 0.7 %. That is the answer to "is my rule wrong 46 % of the time": no. Under a stable holder the nearest and the preceding sample are the same holder, so the choice of rule is nearly irrelevant; the 18.1 % where the next sample is nearer changes nothing because the holder did not change.
Both numbers are worth having, and they belong on different lines so a reader does not conclude the tool disagrees with itself half the time. I will put the distinction in the |
|
The header is in That closes the ambiguity I raised in the previous comment: Also worth noting on this branch, since two of the four commits are restorations: |
|
VERDICT: READY pending rebase at The census confirms the claim, from outside the branchI checked the same three independent of the PR's own table and got the same answer: main does not have them. That is the defect class of this whole night: five times I have seen a tool or a check report on a thing it was not looking at (a cache entry vs the variable The one blocker, and it is not content
The content is worth the rebase
And the boot guard returns — What I am not asking for: any new test file. The PR restores behaviour that |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1f87538: READY -- and "a flag that answers a question must never be answered by a different question" is the line to keep
Both new commits fix outputs that were plausible and measured something else, which is this whole night's defect class:
--attributed --distancessilently dropped one report, andap.erroron a multi-mode set is the right fix rather than re-ordering -- "re-ordering would only move the silent drop to the other flag" is exactly the reasoning that makes it a rule instead of a patch.ap.erroralso gives rc 2, the fleet's usage convention, and it fires before any capture is read.- The agree/disagree column measures HOLDER TURNOVER, not rule correctness. The header sentence is the fix, and the number that proves the two rules agree when turnover is low is what earns the claim: "281 faults, the preceding and following sample name the same holder for 279 of them (99.3 %), so 'nearest sample' and 'last preceding sample' are the same answer whenever the holder is stable." That is a much better argument than either rule's own output, and it retires the 18.1 % "next sample is nearer" cases as harmless.
Two asks:
- Put the calibration next to the column, not only in the commit message.
46 % on one capture, 0.7 % on a stable oneis the reader's scale -- without it, 46 % reads as "two readers disagree half the time" no matter what the header says. One clause in the header or--help: "(0 % under a stable holder; 46 % measured on a capture with heavy handover)". - Say whether
--outlookis in the same mutually exclusive set -- it appears in themodeslist above, so presumably yes; if so, the error message should be able to name three flags as well as two, and the self-test should include one three-way combination so the join is exercised.
Nothing else: attributing quoted numbers to their capture and refusing ambiguous modes are the same fix in two places, and both leave the reader with a smaller chance of believing a number that means something else.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1f87538: READY
This repairs a merge I reviewed and got wrong, so the accounting comes first.
#2623 merged at 0193a29 with 3 of the 5 commits that were pushed, and my check on it was:
rows_from_text 6
outlook 10
Both true, and both in the three commits that survived. So my verification confirmed the merge had landed the part I happened to count, and said nothing about WEAK_AFTER_S or def distances, which are 0 in main and were lost silently. A squash merge does not tell you which commits it took, and my check was chosen from what I could see rather than from what could be missing.
What makes this PR the right repair is the evidence it starts with: one row per lost commit, counted in main before and after — WEAK_AFTER_S 0, def distances 0, ATTRIBUTED-WEAK 0, --distances 0. Not "I think they were lost" but a count per symbol. That is the shape a restore needs, because the failure mode is exactly a symbol that is absent and whose absence looks like a version that never had it.
Verified at 1f87538:
WEAK_AFTER_S 3 (main 0)
def distances 1 (main 0)
ATTRIBUTED-WEAK 6, emitted in a print (1)
--distances 3, distances() called 4
--selftest "self-test ok", rc 0
And the restore brought its own test: the self-test's coverage of distances grew from 0 in main to 4 — the lost commit carried the test that would have failed without it, which is why its absence was survivable for a day and not visible as a red anything.
The new comment is also the right comment, and worth naming because it is the same instinct as #2595's per-fault attribution: it states what the column measures — "a window naming two holders is DISAGREE by design, so the count measures holder turnover, not rule correctness" — and then gives the number that bounds it (279 of 281 faults, 99.3 %). A column whose name suggests one question and whose value answers another is how a diagnostic tool lies without a bug; this comment is one line that stops it.
Nothing outstanding. The 207-line tool, its self-test, and main now agree on which symbols exist.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1f87538: READY
The new line is exactly what I asked for, and it corrects my framing rather than just documenting the column.
What it says: agree/disagree compares the sampler's windowed hits with this tool's verdict; a window naming two holders is DISAGREE by design — so the count measures holder turnover, not rule correctness — with the measurement that makes it persuasive: "on a stable holder the nearest sample (either side) and the last preceding sample name the same holder (279 of 281 faults, 99.3 %)".
That reframes the number I reported at the previous head. I wrote "47 % of faults get a different holder from the two rules" as if the rules were in conflict; if 99.3 % of low-turnover faults agree, then the 47 % is the turnover — 18 tickets an hour over the capture window — and the column is a measurement of the box, not a defect in either rule. That is a better reading than mine, and it is the one a future reader should get.
My attempt to verify the 99.3 % was inconclusive, and the reason is worth recording: I tried to count holder-turnover from attributed.log directly and could not, because a hits entry is five tab-separated columns per sample (the engine#2627 distance, then pid, cmd, scope, ticket). My naive "split on non-dash fields" counted each single holder as four. Grouping the records I could identify unambiguously — a complete single-holder hit — gives 157 agree / 13 disagree (92 %), which supports their direction; the rest of my grouping was ambiguity in my own parser, not evidence against theirs.
So the practical note for anyone writing a third reader: hits is not a list of holders, it is a list of four-column holder records preceded by a distance. Any per-holder counting must be done four fields at a time, and this is the second reader to get that wrong first (the GAP row in engine#2624 was the first way the same layout bit).
Everything else from my previous verdict still holds: the flag refusal, the named-window distribution comment, and the definition now printed on every run.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1f87538: READY
Two fixes, and the second one is the kind of bug that makes a tool lie without failing.
Combining reports used to answer a different question
modes = [name for name, on in (("--attributed", …), ("--distances", …), ("--outlook", …)) if on]
if len(modes) > 1:
ap.error("%s are separate reports: run them one at a time" % " and ".join(modes))$ … --samples s.log --attributed /dev/null --distances
error: --attributed and --distances are separate reports: run them one at a time
$ … --samples s.log --distances
no sample gaps in this trace: nothing to measure
Before, --attributed ran, return 0d, and --distances printed nothing — the shell saw success and the operator got an answer to the question they asked first. A flag that answers a question must never be answered by a different question is the right one-line rule, and refusing is the only implementation of it that cannot rot.
A measured number now carries its conditions
"engine#2627 measured the real thing on ITS capture (median 1 s, p90 27 s, max 45 s; 24 of 119 beyond 10 s). Those numbers are that hour's load, not this box's: the running sampler here reads median 2 s, p90 3 s, max 93 s, and the max is the same lapse the sampler records as a GAP."
This is the correction I would have asked for, because the previous text presented one hour's distances as a property of the box — and the two readings differ by a factor of nine at p90. max 93 s being the sampler's own GAP is the useful part: the worst distance and the recorded stall are the same event, so --outlook and drop_pre_boot/GAP are measuring one thing from two sides.
What I could not check: no sampler log is on this box right now, so I cannot re-measure median 2 s / p90 3 s / max 93 s myself. I am accepting the numbers as attributable rather than verified — which is the improvement, not a claim that they are right.
--self-test ok.
|
Its partner merged first, so Checked by content on
12 s stale is the honest reading of a node whose tenure is ~200 s — plausible, not certain — and today So the sequencing argues for this one landing soon rather than being queued behind other work: it is the half that stops the wrong number from looking like a right one. Nothing else from me on the content — my review at |
…ngine#2652) A merge lands the head sha recorded when the merge was queued, so a commit pushed after that point is never merged and nothing says so: the PR shows merged, the branch shows the commit, and main does not have it. Four instances in one night; one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and was caught only because somebody counted symbols by hand. Building this found that the obvious check does not work. #2623's RECORDED diff contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by definition absent from the diff frozen at the merged head -- so a --pr content check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one of those was a line a later PR had legitimately deleted, not a lost commit. So the question is asked about commits: --branch REF git rev-list origin/main..REF -- the work of mine that has NOT landed. Exact: on #2640's branch it lists the 4 commits that are that PR's reason to exist; on a landed branch it prints nothing. --sha SHA is that commit an ancestor of main? Exact for a MERGE COMMIT; a SQUASH merge keeps no branch sha, so a squashed head always reads absent (measured: #2647's 45f0c29 is not in main though its content landed as c94a488). --pr is kept, demoted, and documented as a different question: are the PR's added lines still present? It reports the merge strategy first, and classifies each absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it answers 'did this content survive' rather than 'did this commit land'. Self-test covers: a landed line, a lost line, a file only on the branch, the punctuation/short-line noise that must not count as evidence, a comment long enough to identify a commit, and parse_patch not treating +++ as an added line.
…ngine#2652) A merge lands the head sha recorded when the merge was queued, so a commit pushed after that point is never merged and nothing says so: the PR shows merged, the branch shows the commit, and main does not have it. Four instances in one night; one of them (#2623's WEAK_AFTER_S / --distances) needed a restore PR (#2640) and was caught only because somebody counted symbols by hand. Building this found that the obvious check does not work. #2623's RECORDED diff contains 0 occurrences of WEAK_AFTER_S, because a post-merge push is by definition absent from the diff frozen at the merged head -- so a --pr content check is structurally blind to this bug. Run across the 59 merges of 11-15/16 it flagged 22 PRs, and every one of those was a line a later PR had legitimately deleted, not a lost commit. So the question is asked about commits: --branch REF git rev-list origin/main..REF -- the work of mine that has NOT landed. Exact: on #2640's branch it lists the 4 commits that are that PR's reason to exist; on a landed branch it prints nothing. --sha SHA is that commit an ancestor of main? Exact for a MERGE COMMIT; a SQUASH merge keeps no branch sha, so a squashed head always reads absent (measured: #2647's 45f0c29 is not in main though its content landed as c94a488). --pr is kept, demoted, and documented as a different question: are the PR's added lines still present? It reports the merge strategy first, and classifies each absent line as 'removed by a later commit' (git log -S) or genuinely lost, so it answers 'did this content survive' rather than 'did this commit land'. Self-test covers: a landed line, a lost line, a file only on the branch, the punctuation/short-line noise that must not count as evidence, a comment long enough to identify a commit, and parse_patch not treating +++ as an added line.
…an certain
engine#2627 measured what I could not: nearest-sample distances of median 1 s, p90 27 s, max 45 s,
and 24 of 119 faults more than 10 s from any sample. My reader had the same information and threw
it away in the verdict -- "ATTRIBUTED: pid 1234 gpu-hold cl-a" says nothing about whether that
sample was 2 s or 44 s old, and the reader cannot judge an attribution without it.
So every holder line now carries its distance:
ATTRIBUTED: pid 1234 gpu-hold cl-a pid 9 (sample 2s before)
ATTRIBUTED-WEAK: pid 1234 gpu-hold cl-a pid 9 (sample 27s before), the node changes hands
every ~200s on this box
The cut is 5 s, from my own measurement rather than taste: 18 distinct gpu-hold tickets held the
node over 3602 s, i.e. a hand-over about every 200 s, so the chance the holder changed inside the
gap is 2.5 % at 5 s, 13.5 % at 27 s and 22.5 % at 45 s. Below 5 s it is noise; past it the
attribution is worth reporting and not worth presenting as certain.
The tally splits them, so a run cannot look stronger than it is:
faults: 5; attributable 2 (weak 1), ambiguous 1, nothing-held 1, no-sample 1, other-device 0
Pinned: a 30 s-old sample is ATTRIBUTED-WEAK, and the summary counts it under weak. The
NOT THE ACCEL NODE / padded-stamp / GAP arms are unchanged, and the live capture still reports
"faults: 137; ... other-device 137" because those faults are NVMe.
(cherry picked from commit a44dc59)
…tore the boot guard
Two changes, both from evidence on this box rather than from taste.
--distances prints the statistic engine#2627 measured on real faults (median 1 s, p90 27 s, max 45 s;
24 of 119 beyond 10 s), computed from the sample timeline alone -- a fault lands at a random time, so
its nearest-sample distance is distributed like the gaps between samples. Live, on the running
sampler:
gaps 2867 median 2s p90 3s max 93s
nearest sample within 2s: 89.4%, within 5s: 93.1%, within 10s: 99.8%
That is the number a capture's owner wants BEFORE any fault: it says whether the sampler is dense
enough to attribute one, and how much of a run will be WEAK. The 93 s max is the same lapse #2624
records as a GAP.
Second: drop_pre_boot() is restored. It was written when the box cold-rebooted (kernel stamps are
boot-relative, so a samples.log straddling a reboot lets the gap test match a fresh fault against a
sample taken hours earlier) and it landed on a branch that merged before that commit reached it, so
main never got it. Every mode applies it again, with a stderr note naming the count and the boundary,
and an unknown --boot-epoch filters nothing rather than guessing.
Pinned: the distance distribution on a five-gap fixture (median 1, p90 20, 80 % within 2 s), one
sample having no gaps at all, and the three drop_pre_boot cases.
(cherry picked from commit f7700ba)
…bers to their capture Two changes, both from the review at c339f0e. 1. The flag bug. `--attributed` ran first and returned, so `--attributed --distances` printed the disagreement report and silently dropped the distance distribution. Re-ordering would only move the silent drop to the other flag, so the modes are now mutually exclusive and say so: error: --attributed and --distances are separate reports: run them one at a time A flag that answers a question must never be answered by a different question. 2. The number in the comment. It said "engine#2627 measured nearest-sample distances of median 1 s, p90 27 s, max 45 s" as though those described this box. They describe that capture. The text now says so and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the max being the same lapse the sampler records as a GAP. That is the lesson of #2641 applied to my own prose: an interval measured on one hour is a fact about that hour. Self-test unchanged and green; the live sampler still reads "gaps 7374 median 2s p90 2s max 93s".
The column compares this tool's verdict with the sampler's own `hits` list, and a window naming two holders
is DISAGREE by design -- a window listing two lanes is not corroboration. That makes it a measure of HOLDER
TURNOVER, not of either rule's correctness, and without saying so a reader concludes the two readers
disagree with each other half the time (measured 46 % on one capture, 0.7 % on a stable one).
The measured reason the rules coincide when turnover is low: on a live timeline of 281 faults the preceding
and following sample name the same holder for 279 of them (99.3 %), so "nearest sample" and "last preceding
sample" are the same answer whenever the holder is stable -- the 18.1 % of cases where the NEXT sample is
nearer change nothing, because the holder did not change.
So the header now prints, on every --attributed run:
# agree/disagree compares the sampler's windowed hits with this tool's verdict: a window naming two
holders is DISAGREE by design, so the count measures HOLDER TURNOVER, not rule correctness
Self-test green; the synthetic attributed run prints the header then the row, as before.
1f87538 to
1da6f9a
Compare
|
Persistent review updated to latest commit 1da6f9a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 1da6f9a — same four commits, rebased onto current main; nothing in the content moved.
1da6f9a08 diag: say what the agree/disagree column measures
f15d40412 diag: refuse combining the report flags, and attribute the quoted numbers to their capture
e7d8e775a diag: predict the nearest-sample distance from a quiet trace, and restore the boot guard
d550ded1f diag: carry the sample distance, and call a far sample weak rather than certain
origin/main..1da6f9ais 0: thedirtystate from my review at1f87538is gone, and the diff against the old head is 27 files of main's drift, not the PR's.- The restored symbols are still there:
WEAK_AFTER_S/def distances/def drop_pre_boot→ 5 occurrences at this head, matching what#2623lost.
My READY at 1f87538 was on the content, and the content is unchanged; this confirmation exists only because the head moved and a verdict has to name the head it covers.
One thing this PR's own existence proves, and I would keep in the description: these commits were lost because the merge took a queued head, and the note now on #2659 says the general rule — a diff-based check cannot find this class (#2623's recorded diff contains 0 occurrences of WEAK_AFTER_S), and main..branch is dominated by squash (95 of 198 branches look unlanded). Restoring by hand was the right move here; the tooling only catches it going forward.
|
VERDICT at Worth recording how I established that, because my first look got it wrong: A compare between two heads of a rebased branch is not a PR delta. Same family as tonight's other comparison errors (a profiler product read as a shape, a line count read as a size) and worth stating in the review rather than quietly re-running it. The substance I verified on the earlier head stands unchanged: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 1da6f9a: READY
Rebased onto current main; all three of my earlier findings survive, and I checked the one thing a rebase can break — whether it now duplicates a merged sibling.
The three findings, re-verified by content at this head (not by reading the diff, which now spans 27 files of main's own progress):
present flag refusal --attributed and --distances are separate reports
present the named-window comment "that hour's load, not this box's" + the sampler's own 2 s / 3 s / 93 s
present the agree/disagree line HOLDER TURNOVER, not rule correctness
No duplicate with #2645: this head also carries the (E-L, E] GAP span, but that came from main — #2645 merged as 6a696102 and the span is on origin/main, so the rebase picked it up rather than re-adding it. Its own diff is one file, +112/-15. (Worth stating because three of tonight's PRs were duplicates of exactly this shape; this one is not.)
Self-test ok at this head.
Unchanged asks from my earlier verdict: nothing blocking. The one thing I would still like in ATTRIBUTED's docstring is the number beside the definition — the sampler's nearest rule names a different holder for about half the faults on a busy capture — because that is what tells a reader how much weight the attribution carries.
Not verified: the live scan (no device work, no gh sweep in a review).
What
Restores two commits that were pushed to #2623 but did not reach
main: the sample-distance /ATTRIBUTED-WEAKchange (a44dc5927) and--distancesplus the restored boot guard (f7700ba0a).#2623 merged at
0193a29with 3 commits; I pushed 5. Verified againstmainrather than against the branch:The two commits
ATTRIBUTED-WEAK— a holder line now says(sample 27s before), and past 5 s the class becomesATTRIBUTED-WEAKwith the reason. The 5 s cut is measured: 18 distinctgpu-holdtickets over 3602 s means a hand-over every ~200 s, so the holder changed inside the gap 2.5 % / 13.5 % / 22.5 % of the time at 5 s / 27 s / 45 s. The tally splits it:attributable 2 (weak 1).--distances— predicts a fault's nearest-sample distance from the sample timeline alone (live: gaps 2867, median 2 s, p90 3 s, max 93 s; 89.4 % within 2 s), so a capture owner knows before any fault whether the sampler is dense enough and how much of a run will beWEAK. The 93 s max is the same lapseaccel_sampler.shnow records as aGAP.drop_pre_boot), which never reachedmainby the same mechanism the first time.This is the third commit of mine lost this way tonight (#403's counter fix, this pair, and the boot guard twice). The recovery is mechanical: cherry-pick onto current
main, verify by greppingmainfor the symbol, and checkpulls/N.head.shaagainst the branch after pushing — when they differ the PR is already merged and anything newer needs a new PR.Verified