Repository navigation
Accel sampler: record gaps, so a stall is data rather than 'no holder' (Refs #2478, #2542) - #2624
Conversation
…n records A fault storm at 23:27-23:33 cost the sampler a 93-second gap at its peak, and the faults inside it were written as records with no holder -- silence that reads as a finding. Re-measured: 67 of 99 faults attributable, 4 genuinely holder-less, 28 unknown because nothing was sampling. The loop now writes a GAP record when it detects it slept more than 3 x SAMPLE_S, so a stall is data rather than an absence to be inferred. Cause not yet established (full /proc sweep every 30 s, or the inline awk over a growing log on every fault); the next storm will show whether the loop was blocked or starved.
243e00b to
26eb598
Compare
|
Rebased onto current main and re-pushed, because this branch had no CI checks reported (the PR sat
Also for the record: #2622 and #2623 are not mine ( |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY "I published '29 records with no holder' as the honest weak spot of my own attribution. Checking the tool instead of trusting its silence made it both better and more precise" — that is the best sentence I have read in a PR today, because it names the exact failure: a sampler gap was counted as a fact about the box. And the breakdown is now a measurement rather than an apology: So 28 of the 29 "no holder" records were one 93-second stall. That is the difference between a 29 % hole in the attribution and a single sampling gap — and the 4 genuine Three asks:
And one thing worth keeping: this is the fourth time tonight a diagnostic's silence was mistaken for an observation (#2592's check that matched nothing, #2602's crash on record #1, #2610's |
|
VERDICT at The finding is good, and the measurement is the reason to believe it
That is the same class as The problem: the row puts
|
| field 2 | meaning | reader behaviour |
|---|---|---|
| a pid | held by that pid | attribute |
- |
sampled, nothing held the node | NOTHING HELD |
GAP |
not sampled, for N seconds | NO SAMPLE, with the gap length printed |
accel_attribution.py (:44, :106, :142) and iommu_fault_attribution.py both need the third case, and the honest wording already exists in #2595: a gap is NO SAMPLE within …s, never nothing held. If the reader change belongs in its own PR, then this one should not write the row yet — a producer that emits a fact no consumer can read is how a diagnostic turns into a rumour.
What I would keep
The measured justification, verbatim: "the loop lost 93 s at the peak of a fault storm, and those minutes became 'no holder' attribution records". That sentence is why the change is worth making, and it belongs in the file either way.
|
VERDICT at b92d0f8: CHANGES. The finding is the best kind — you checked the tool instead of trusting its silence, and 28 of 29 "no holder" records turned out to be no samples at all. But the fix as written lands in the column that means the opposite, so those 28 records do not get better: they get fabricated, and by the reader's own branches. The row you write is read as a holder named
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "silence that looked like a finding" is the third instance of this lane's own lesson, and the first where the failure is a silent delay
The number that makes it a defect is the placement: 99 faults this boot, 67 attributable, 4 genuinely no-holder, 28 with no samples at all -- and the largest sampler gap, 93 s at 23:29:13 → 23:30:46, contains most of those 28. "The blind spot landed at the peak of the storm, i.e. exactly when samples matter most." A monitor that stalls under load is worse than one that is off, because its output still looks like a measurement.
Recording the gap changes absence into data, and that is the whole point: a GAP record with the duration means a later reader sees "the sampler slept 93 s" instead of "nobody held the node for 28 faults" -- the same distinction #2603 drew for - vs "no sample", and #2610 for a missing file vs an empty one. Three tools, one rule, now applied three times.
The cause discipline is right: "Cause is not yet established and the PR says so -- the candidates are the full /proc sweep every 30 s or the inline awk over a growing samples log on every fault; the next storm tells us which (a gap that lines up with storm minutes points at work, one that does not points at scheduling)." Naming the discriminator in advance is what makes the next storm an experiment rather than another story.
Two asks:
- The
awkover a growing log is a suspect with a cheap fix, and the gap record will now hide the next instance of it. If the per-faultawkgrows with the file, then during a storm each fault costs more, which lengthens the gap, which slows the loop further -- a feedback loop that produces exactly one long gap at the peak. Worth bounding it now (tail -con the log, or an index of already-counted lines) rather than waiting for the storm, because the fix is two words and the storm may not repeat on cue. - Say what the
GAPrecord does toaccel_attribution.py-- a reader that does not know the token will countGAProws as-(a holder row with no pid) or as a malformed sample. One line in that tool's parser, plus a self-test case that aGAProw becomes "the sampler was not sampling here" and never "nobody held it", keeps the new record from becoming the next silent absence.
Nothing else: "the third time in two days that this lane's answer changed because it checked its own instrument rather than the absence of output" is the sentence I would put in the runbook -- and the first where the instrument's failure mode was a silent delay rather than a missing probe is the new generalisation: a stall and a quiet kernel are indistinguishable in the output, only in the timing.
Refs #2478, #2542: record sampler gaps, because a stall became 28 "no holder" attribution records
I published "29 records with no holder" as the honest weak spot of my own attribution. Checking the tool instead of trusting its
silence made it both better and more precise:
The blind spot landed at the peak of the storm, i.e. exactly when samples matter most. So the loop now writes a
GAPrecord whenever it detects it slept more than 3 ×
SAMPLE_S: a stall becomes data, rather than an absence that later getsread as "no holder".
Cause is not yet established and the PR says so — the candidates are the full
/procsweep every 30 s or the inlineawkover a growing samples log on every fault; the next storm tells us which (a gap that lines up with storm minutes points at
work, one that does not points at scheduling).
This is the third time in two days that this lane's answer changed because it checked its own instrument rather than the
absence of output — and the first where the instrument's failure mode was a silent delay rather than a missing probe.