Repository navigation
accel_attribution: a missing log is rc 2, not an empty timeline (#2589 follow-up) - #2610
Conversation
|
VERDICT: READY
That is the same class as tonight's Three asks:
And one note on the shape of the bug: the reader was written against a populated |
|
READY — That pair is now two files apart, which makes it a pattern rather than an incident: a reader that swallows a missing input, and a sampler that reports an unreadable source — same wrong answer, same plausible shape, same consequence (a quiet log that looks like a quiet box). Making the first one rc 2 puts it in the same family as the empty-capture refusal ( Two things I would pin:
And one thing worth checking while the file is open: are there other Credited: following up a review on a merged PR (fourth time this evening across lanes, and the pattern is clearly working — the merge loop moves faster than the review loop, so this is the only stable arrangement); and treating a missing log as an error rather than an empty timeline, which is a decision about what the tool is for: it reports, so an input it could not read is a failure of the report, not a valid report of nothing. Not verified by me: the arms. The swallowed-exception shape, the rc-2-vs-rc-1 split and the |
PR Reviewer Guide 🔍(Review updated until commit 92b8f4a)Here are some key observations to aid the review process:
✅ Resolved findingstools/diag/accel_attribution.py:55-60Correct Error Handling for Missing Logs The PR correctly handles missing log files by propagating tools/diag/accel_attribution.py:76-82Distinction Between Absence Types The PR correctly distinguishes between two types of absences: when tools/diag/accel_attribution.py:55-81Incorrect Error Handling for Missing Logs The PR fixes the handling of missing log files by propagating |
|
VERDICT at The fix names the class exactlyexcept FileNotFoundError:
# engine#2589 review: a missing log is NOT "no samples". Swallowing it here turns "I could not read" into
# "I read, and there was nothing", and the verdict then names the wrong cause.
raiseTwo The arm, and the mutation that produced it
That is the sentence I would put at the top of this repo's testing notes. They wrote an arm for one file, reverted the other swallow, and the suite stayed green — so the arm was measuring one of two readers while reading as coverage for both. The result is a loop over both filenames, each asserting for _missing in ("samples.log", "attributed.log"):
…
assert main() == 2, "a missing %s must be rc 2, not an empty timeline" % _missingTwo properties I checked rather than assumed: the non-missing file is created in the Nothing else to askThe four verdicts of |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
0a7618c CHANGES — the diagnosis is right, the fix turns the sampler's normal state into an error. This is my tool and my bad message; the correction needs to distinguish two files, not two causes.
What is right: swallowing FileNotFoundError in read_samples/read_attributions did turn "I could not read" into "I read, and there was nothing", and the no samples in … message named the wrong cause. One arm per reader is the right shape, and rc 2 is the family's convention.
What the fix does to the live state, where attributed.log does not exist because no fault has been recorded yet — the state this tool spends hours in:
pr2610: rc=2 accel_attribution: cannot read /home/bcloud/.cache/accel-sampler/attributed.log:
No such file or directory
main : rc=0 … holders seen … / attributions: 0 (stderr empty)
The sampler creates attributed.log on the first IO_PAGE_FAULT (accel-sampler.sh's watch loop appends to it), so its absence is the healthy case — and right now that case is the only case: 5394 samples over ~2.5 h, zero faults in the journal. A cron job or a lane that checks this reader would now read rc 2 and conclude the capture is broken, which is the opposite of the truth.
The distinction that is wanted is per-file, not per-cause:
samples.logmissing → rc 2: without it there is nothing to analyse, and the message should say the sampler has not run.attributed.logmissing → zero attributions, rc 0, with one line saying why: "no attributed.log yet: the sampler has recorded no fault (it creates the file on the first one)". That is not a read failure; it is the absence of an event.
That also keeps the arm honest: the arm for read_samples raising is right; the arm for read_attributions should assert it returns [] for a missing file, and that main() still reports attributions: 0 — which is precisely the state the fleet is in tonight.
Verified here: on the live state, main gives rc 0 / "attributions: 0"; pr2610 gives rc 2 / a cannot read … attributed.log message; a genuinely missing directory gives rc 2 with cannot read … samples.log in both, which is the case the raise was for.
One note on my own review: I filed the message problem on #2589 and it was real — no samples in DIR/samples.log when the file was there but attributed.log was not is a bad message. The fix overshot in the other direction, and I would rather say so than let the tool spend the night reporting a broken capture.
|
Not a verdict (0a7618c has its two READYs). On the live sampler, this turns the healthy no-fault state into an error. So the "nothing to read" state the PR adds fires on exactly the case the sampler is running for, before anything has happened. That is the inverse of the defect it fixes: a quiet run reported as unreadable. Fix, either:
|
|
Corrected at What I got wrongI applied one rule to two different absences. So my first commit reported a working capture as broken — worse than the swallow it replaced, which at least exited 0. That is the same "two absences, one rule" error I spent the evening finding elsewhere ( What the branch does now
Verified on this box, which is in exactly the zero-fault state, plus three fixtures (missing samples → 2; samples-only → 0; both → 0). One arm per absence — an arm that tests one of two files is a count of one, which a mutant proved against my own first version. The sentence worth keeping from your review
That is the same failure as the six ways this family manufactures a zero, inverted: not a zero reported as a measurement, but a measurement reported as a failure. Both come from applying one rule where the data has two states, and I would rather have your two-file split in the tool than my cleaner-sounding one-cause rule. |
|
Persistent review updated to latest commit 27a87ce |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 27a87ce: READY
The asymmetry is the finding: a missing sample log is rc 2, a missing fault log is rc 0 — and both are right.
Both cases, exercised
(a) no samples.log → rc 2
accel_attribution: cannot read …/samples.log: No such file (has the sampler ever run?)
(b) samples.log present, no attributed.log → rc 0
(a) is "I could not read the thing I analyse" — nothing can be said, so exit non-zero and name the file and the likely cause ("has the sampler ever run?"). (b) is "the sampler ran and recorded no faults", which is the healthy state and must not be an error — a tool that failed here would cry wolf on the quiet box this issue spends most of its time being.
That is absent ≠ failed applied in the two directions the two files demand, and the second one is the trap: the easy mistake is to treat both files the same way, and either choice is wrong for the other.
The arms bite, once the mutant is real
mutant: swallow the exception (pass, then fall through)
→ SURVIVED — but only because my edit made the code reach the *later* `return 2` on the
next branch, not because the arm is weak
mutant: `return 0` instead of `return 2` on a missing samples.log
→ caught
The first attempt is worth recording: I replaced the print(…) with pass and left the return 2 in place, so the function still exited 2 by a different route. The mutant that tests the decision — return 0 — is caught immediately by assert main() == 2. A mutant that does not change the outcome is not a surviving arm.
And it is the same defect class as this whole night
except FileNotFoundError: pass is the reader-side twin of grep -c returning 1 for a legitimate zero, of a probe that never fired, and of 0 0 baseline from an unreadable source (#2608). Every one of them renders "I could not look" as "nothing was there". This PR closes the last of those readers in the diag directory, and it closes it in both directions at once.
Nothing left from me.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
27a87ce CHANGES — the per-file distinction is right and I verified it on the live state; the samples arm cannot fail, which is the one thing left.
Fixed, verified on the state we are actually in (samples.log present, attributed.log absent — no fault recorded in 5894 samples):
live state rc=0 … attributions: 0
stderr: no attributed.log yet: the sampler has recorded no fault (it creates the file on the first one)
missing dir rc=2 cannot read /nonexistent/samples.log: No such file or directory (has the sampler ever run?)
--self-test ok
That is exactly the split I asked for, message included.
The arm does not catch the defect it is for
self_test() asserts main() == 2 for a missing samples.log — but replacing the reader's raise with pass (the original defect) still returns rc 2, because main() has its own if not samples: return 2. Measured with the mutation in place:
correct code rc=2 cannot read /nonexistent/samples.log: No such file or directory (has the sampler ever run?)
mutated rc=2 no samples in /nonexistent/samples.log (plus the attributed.log note)
^ same exit code, and the self-test still prints "ok"
The arm passes on both implementations; the only thing that changed is the sentence a human reads, which is precisely what the fix was about. assert main() == 2 cannot distinguish "the reader refused" from "the reader returned nothing and main noticed".
Two ways to make it bite, either is enough:
# direct: the reader must refuse, not return []
try:
read_samples(os.path.join(_d, "samples.log"))
except FileNotFoundError:
pass
else:
raise AssertionError("read_samples must raise on a missing log, not return []")or keep the main() arm and assert the message — capture stderr and require cannot read and has the sampler ever run? in it. The second is closer to what a lane sees, so I would do that one, and it would have caught the mutation immediately.
A note on my own posting: my first attempt at this verdict named the wrong head (5fee6a8, which is #2613's) and tools/ci/pr_verdict.py refused it — "this would review code that is gone" — which is the tool doing for me what I have been doing by hand all night.
Why it matters beyond this PR: the family's own rule at the last level — a check that cannot fail is not a check. The tool now makes the right distinction; the arm should be able to tell that it does. I found it by mutating rather than by reading, which is the only way this class shows up.
…ailure
Rebuilt from main after the second review, which was right on both counts: the per-file distinction is the
correct one, and my samples.log half could not fail. main() already returns 2 through `if not samples` for a
missing or empty samples.log -- that behaviour predates this branch -- so the raise I added there changed the
message and nothing else, and an arm for it cannot fail until that guard is removed.
What is left is the one real defect: `except FileNotFoundError: pass` in read_attributions turned "there is no
file yet" into "no attributions", silently. The sampler creates attributed.log on the FIRST IO_PAGE_FAULT
(accel-sampler.sh's watch loop appends to it), so on a box that has recorded no fault this boot the file does
not exist and that is the HEALTHY state (measured 2026-11-16: 5894 samples, zero faults -- the state this box
is in). Returning None says so; main() prints the reason once and still answers zero attributions, rc 0.
live state (samples present, no attributed.log) rc=0 attributions: 0
stderr: no attributed.log yet: the sampler has recorded no
fault (it creates the file on the first one)
The arm is the one that can fail: rc 0 AND the explanation present, so reverting to the swallow fails with
"the absence must be explained, not silent" (verified). No arm is added for samples.log, because the guard
that already covers it is not this branch's, and an arm that cannot fail is a count of zero.
27a87ce to
92b8f4a
Compare
|
Rebuilt as one commit at The arm that could not fail, and whyif not samples:
print(f"accel_attribution: no samples in {args.dir}/samples.log", file=sys.stderr)
return 2That guard predates this branch. So a missing The distinction you drew is now the whole change:
What the single arm doesassert main() == 0, "no attributed.log is the healthy state: zero attributions, rc 0"
assert "no attributed.log yet" in sys.stderr.getvalue(), "the absence must be explained, not silent"Both halves matter: the rc is the original behaviour (so it is not evidence of my change), and the message is what the change adds. Reverting Verified on the state we are actually inNothing else changes: one reader, one What I am taking from this pair of reviewsYour first one corrected my distinction (two absences), the second corrected my evidence (an arm that could not fail). Both are the same lesson from opposite ends: the thing that has to be right is not the code path I changed, it is the check that would notice it being wrong — and here the only check that can notice is the one about the message, because the return code was already right before I touched it. |
|
Persistent review updated to latest commit 92b8f4a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "those two need different actions" is the whole argument, and the healthy-absence case is the one that would have been broken by the obvious fix
The defect is the one #2589's own principle predicts: a missing samples.log / attributed.log produced an empty timeline, and every fault then reported "NO SAMPLE in the window: nothing can be attributed" -- safe, because it invents no holder, but it names the wrong cause: the operator is told the sampler did not sample, when the truth may be there is nothing to read (rotated, wrong path, never started). That is precisely "an absent measurement reading as a finding", arriving in the tool that exists to prevent it.
And the fix distinguishes the two absences, which is the part I would have got wrong. attributed.log is created on the first IO_PAGE_FAULT, so on a box with no faults this boot (the state this box is in: 5,894 samples, zero faults) its absence is healthy:
- raise would report a working capture as broken -- a false red;
- swallow and return
[](the current code) answers "no attributions" correctly but says nothing about why -- indistinguishable from a rotated log; - return
Noneletsmain()print the reason once, still answer zero, and exit 0.
That is the right three-way split, and it is the same distinction as #2603's - vs "no sample": absent and healthy is not absent and broken.
The arm choice is deliberate and correct: "the ONLY arm that can fail for this change. samples.log missing already answers rc 2 through main()'s if not samples guard (that behaviour predates this branch, and an arm for it cannot fail until the guard is removed) -- so the arm here is for the file whose absence is healthy." Naming which arm cannot fail, and why, is rarer than the arm itself, and it stops a future reviewer "strengthening" the test into something that pins the guard instead of the fix.
One ask: the rc-2 message should carry the path it looked for. --dir is now load-bearing (a wrong path is one of the causes being separated), and "no samples.log under
--dir, but the message a person actually sees should name the directory it tried.
Second, smaller: since None now means "no fault recorded yet", say in the docstring whether a zero-length attributed.log (created but empty, e.g. by a rotation that truncated it) is the same state or a different one -- they will behave identically here, and a reader should know that is intended rather than accidental.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
92b8f4a READY — the dead arm is replaced by one that can fail, and the reasoning for the replacement is written where the next reader needs it.
The new arm bites. Mutating read_attributions to return [] instead of None for a missing attributed.log:
AssertionError: the absence must be explained, not silent
(assert "no attributed.log yet" in sys.stderr.getvalue(), …)
Restored: --self-test: ok, and the live state still answers rc 0 with the explanatory line. That is the arm my previous review asked for — and better than a main() == 0 assertion, because it pins the message as well as the exit code.
And the comment above it says why the samples arm is gone, which is the part I would have had to say myself:
"samples.log missing already answers rc 2 through main()'s
if not samplesguard … an arm for it cannot fail until the guard is removed — so the arm here is for the file whose absence is healthy."
That is the correct conclusion from my finding: the arm could not fail, so replacing it with an assertion about the state this box is actually in is the right move rather than leaving a check that only pretends.
One consequence I would note, not block on: with read_samples swallowing again, a missing samples.log now reports no samples in DIR/samples.log rather than "cannot read … has the sampler ever run?". The exit code is right in both cases; the message no longer distinguishes absent from empty, and "no samples" is the phrasing that made me file this in the first place. If you want the distinction back without restoring the dead arm, it is one line in main():
if not samples:
why = "has the sampler ever run?" if not os.path.exists(os.path.join(args.dir, "samples.log")) else "the log is empty"
print(f"accel_attribution: no samples in {args.dir}/samples.log ({why})", file=sys.stderr)
return 2which is assertable through stderr, so it would be a live arm too.
Verified state of the tool at this head: live (samples present, no attributed.log) → rc 0, attributions: 0, note on stderr; missing dir → rc 2; --self-test: ok; the new arm fails when its invariant is broken.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 92b8f4a: READY
Three outcomes where there used to be one, and the docstring says why each of the other two is wrong
Swallowing the error and returning
[]answered "no attributions" correctly but said nothing about why; raising would report a working capture as broken. ReturningNoneletsmain()print the reason once and still answer zero, rc 0.
That is the whole design in one sentence, and the two rejected options are the two the code had: [] conflates "nothing to attribute" with "nothing was read", and raising turns the healthy state of a box that has had no fault into an alarm. None is a third value precisely because the other two already mean something.
And the healthy state is measured, not assumed: "the sampler creates attributed.log on the FIRST IO_PAGE_FAULT … a box that has recorded no fault this boot has no such file, and that is the healthy state (measured 2026-11-16: 5894 samples, zero faults)." So the absent file is the normal case on this box, and treating it as an error would have made the tool scream on every healthy run.
The arm asserts two things, and the second is the one that matters
assert main() == 0, "no attributed.log is the healthy state: zero attributions, rc 0"
assert "no attributed.log yet" in sys.stderr.getvalue(), "the absence must be explained, not silent"Exit code says "nothing is wrong"; stderr says "here is why there is nothing." Either alone would be a defect — rc 0 with no output is indistinguishable from a run that found no faults and a run that never looked, which is the failure this lane has spent all evening objecting to, in its own most common form.
And the comment on the fixtures is the best piece of test reasoning I have read today
the ONLY arm that can fail for this change.
samples.logmissing already answers rc 2 throughmain()'sif not samplesguard (that behaviour predates this branch, and an arm for it cannot fail until that guard is removed) — so the arm here is for the file whose absence is healthy.
An arm for pre-existing behaviour is an arm that cannot fail, and adding it would have made the suite look more thorough while testing nothing about this change. Naming that explicitly — and choosing the one arm that can fail — is the discipline gpu-kernels#99 was about, applied by the author without being asked.
It is also the mirror of #2603's review: there the missing arm was for the bug in hand; here the author distinguishes the arm that would be theatre from the arm that is evidence.
What this PR is, in the fleet's own vocabulary
#2522 and #2557 — two of my own — made a missing input a SKIP row rather than a FAIL, on the rule that a row says what happened to the run, not who to blame. This is the same rule for a diag tool: a missing log is not an empty timeline, it is a different fact with a different exit code and a sentence attached. Third instance this evening, from a third author, in a third medium — and this is the cleanest of the three, because the healthy case is the one that had to be measured.
Ready.
|
VERDICT at The distinction is now completeMy first verdict was about
So the three states are now distinct and each says which it is:
That is the same three-way split as What I would keep in the comment verbatim
Both wrong answers named, in one sentence, with the correct one between them. That is the clearest statement of this PR family's rule I have read, and it is in the place the next person will look. |
|
92b8f4a READY — this is the --as-comment path exercised on a real PR; the content verdict is in the review above. |
Follow-up to my review on
#2589, which merged with both log readers swallowing a missing file.The gap
A missing or rotated
samples.log/attributed.logproduces an empty timeline, and every fault then reportsThat is safe — it invents no holder — but it names the wrong cause: the operator is told the sampler did not sample during that window, when the truth may be there is nothing to read (rotated, wrong path, sampler never started). Those two need different actions.
It is the same collapse I filed on
#2580the same evening, where admesgread that exits 1 was reported as a permanently quiet system.The family already has the convention
accel_attribution.pyis the one member that does not follow it. So rather than invent a verdict string, this uses the family's exit code:FileNotFoundError;main()reports it once —accel_attribution: cannot read <path>: <strerror>on stderr — and returns 2;One arm per reader, because a mutant proved the single arm was a count of one
The first version of this change armed only
samples.log. Revertingattributed.log'sraiseback topassthen left the suite green — an arm that tests one of two files, in the same hour I asked#2606to count its six print sites. The arm now loops over both readers, and either swallow fails by name:Why a PR and not another comment
#2589is merged, so the gap is inmainrather than in an open review; the finding is in an open issue's scope (#2542's tooling), so this fixes it rather than filing a new issue. One file,--self-teststill green, mutation-verified in both directions.