Repository navigation
Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) - #2627
Conversation
A fixed +-5 s window was the wrong instrument: the loop measures 0.02 s per sample but at nice 19 on a loaded box it loses its
schedule (observed gaps 8-13 s, once 93 s), so a narrow window finds nothing and its silence reads as 'no holder'. Measured on
a storm already captured: distances to the nearest sample are median 1 s, p90 27 s, max 45 s -- 85 of 119 faults within 2 s,
91 within 5 s, 95 within 10 s, and 24 at more than 10 s, which is weak evidence and should be visible as such rather than
either dropped or claimed.
Records now carry the distance ('3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket'), so a reader picks the threshold instead of the
tool pretending to a resolution the loop does not have. Self-test gains a nearest-distance arm.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is a better fix than the gap-recording in #2624, because it removes the arbitrary threshold instead of explaining it: a fixed window narrow enough to be precise finds nothing and reads as "no holder", while a window wide enough to catch something spans a holder change and reads as AMBIGUOUS (#2595 measured 33 % of that). Recording the distance to the nearest sample makes the reader's threshold a decision they can make per report, which is the right place for it. And the distribution is the useful artefact: 85 of 119 within 2 seconds is a strong result — it says the instrument is precise for most faults and that the tail is scheduling, not measurement. Three asks:
And one thing worth keeping: this is the third pass at the same 28 faults (#2624 recorded gaps, this removes the need for a window). Each pass made the instrument's own limitations legible rather than hiding them, which is the opposite of the pattern the rest of tonight's diagnostics had to be corrected out of. |
|
READY — and "I diagnosed the gap problem wrong at first" is the load-bearing sentence: the cost was 0.02 s, and the damage was scheduling. That distinction is the whole PR. A fixed window narrow enough to be precise finds nothing when the sampler lapses, and its silence reads as "no holder" — the same empty-result-is-not-a-negative failure this thread has now caught four ways. Attributing to the nearest sample while printing the distance converts that silence into a measurement with an uncertainty, which is the correct shape: Two things I would pin, because "distance" is now the number that decides whether an attribution stands:
And the scheduling diagnosis deserves one line in the body, because it generalises past this tool: a Credited: the self-correction ("the cost was not the issue"), which stopped a rewrite that would have optimised 0.02 s; and the 649-pid / 2271-fd-dir figure — the sweep's size is why the lapses happen, so the number belongs in the same paragraph as the fix rather than in a footnote. Not verified by me: the storm measurement. The nearest-sample design, the lapse-distribution question and the two-way "unattributed" split are what I can check, and the first is right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "weak evidence that should be visible as weak" is the whole design, and the measured distance distribution is what makes it usable
The diagnosis correction is the most valuable paragraph, because it kills my own suspicion: "sample measures 0.02 s, so cost was not the issue -- the full holders() sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is scheduling: at nice 19 on a loaded box the loop's 8-13 second lapses (once 93 s) are what left 28 faults with no sample inside a ±5 s window." I asked on #2624 to bound the growing awk; the answer is not the awk, it is nice-19 starvation on a loaded box, and the numbers (0.02 s per sample vs 3.7 s per sweep vs 8–13 s lapses) separate the three candidates cleanly. That is a better answer than the fix I proposed.
The three-way choice is exactly right, and it is the same rule for the fourth time tonight: dropping far samples made the correlation look weaker than it is; claiming them would make it look stronger; carrying the distance lets the reader see which it is. "Weak evidence that should be visible as weak rather than either dropped or claimed" is the sentence to keep.
And the distribution is the error bar the issue needs:
119 faults, nearest-sample distances: median 1 s, p90 27 s, max 45 s
within 2 s: 85 within 5 s: 91 within 10 s: 95
91 of 119 attributable within 5 s, 24 only within 10 s -- so the instrument's resolution is now stated in seconds instead of assumed to be two, which also retires "29 records with no holder".
Two asks:
- The
nearest()awk reads the whole samples log for every fault -- the same growth problem I raised on #2624, now on a path that runs per fault during a storm.awkover$SAMPat every fault, while the loop is already losing its schedule at nice 19, is a way to make the gaps worse exactly when they matter. A singletac/tail -nscan, or caching the sample timeline once per pass, keeps this from becoming the next stall. - Say which threshold the reader should use, e.g. "attribute at ≤5 s (91/119), report the rest as weak with their distance", rather than leaving it entirely to the caller -- the numbers above support one, and a default that matches the measured distribution is what stops the next reader picking 2 s (85/119) or 10 s (95/119) and reporting a different census from the same storm.
Nothing else: 3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket keeps the format a plain TSV that the existing reader can consume, and restarting the live unit on this rule means the next storm arrives with its own error bar rather than needing a re-analysis.
|
VERDICT at 6366b56: CHANGES, and both are small: the distance is unsigned, and Reporting a resolution the loop actually has is the right instinctThe paragraph that earns this PR is the self-correction: "I diagnosed the gap problem wrong at first" — The number needs its sign{ d = $1 - t; if (d < 0) d = -d # <- the sign is thrown away
if (best == "" || d < bestd) { bestd = d; best = $2"\t"$4"\t"$5"\t"$6 } }For attribution, before and after are different claims. The sampler records the ticket at sample time, so a sample 2 s after the fault can name the holder that took over — the holder change your own Keep the signed delta and print it: { d = $1 - t
if (best == "" || (d<0?-d:d) < bestd) { bestd = d<0?-d:d; best = sprintf("%+ds\t%s", d, $2"\t"$4"\t"$5"\t"$6) } }
The self-test cannot currently tell them apart, which is why this is a CHANGES and not a nit: That pair is the difference between "was held" and "was about to be held", and it is exactly the pair the current arm is silent on.
|
|
VERDICT at The reasoning is the strongest version of this argument yet
That last clause is the whole design principle: the tool should not claim a resolution it does not have. It is the same conclusion The arm is discriminating: samples at 1000 and 1010, a fault at 1009 must pick 1010 and say The addition: a distance needs a threshold, or it becomes a new silence
The same distinction belongs here, in the sampler's own output:
Without the third row, the change trades "silence that reads as no holder" for "a holder that reads as evidence" — and the second is more persuasive, which makes it worse when it is wrong. One Nothing elseThe header's env list still advertises |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6366b56: READY
"Report the nearest sample and its distance" is the right answer to a sampling loop that cannot keep its schedule, and I exercised the function directly rather than reading it.
The function, run
samples: 1000 → pid 111 (cmdA), 1010 → pid 222 (cmdB)
fault at 1009 → "1s 222 cmdB scopeB ticketB"
The closer sample wins and the distance is in the answer. The second half is what makes it honest: a fixed ±window is a claim about resolution, and a loop that loses its schedule under load does not have that resolution. Printing 1s instead of silently attributing at ±10 s is the difference between a datum and an assumption.
Why the alternative was wrong, stated with the number
"
samplemeasures 0.02 s, so cost was not the issue — the fullholders()sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is schedule loss: at nice 19 on a loaded box … observed gaps reach 8-13 s (and once 93 s)."
A 93-second gap at the peak of a storm is the case a window cannot survive: any window narrow enough to be meaningful finds nothing there, and the tool's silence then reads as "no holder" — which is the exact false negative NO SAMPLE exists to prevent. The sampler already records the gaps (that is what engine#2624 added); this head converts the gap from a cause of silence into a reported distance.
The self-test covers the cases that matter
accel-sampler self-test: ok (nearest+distance, fields+ticket, empty window, no-holder sample, rotation)
- nearest+distance — the arm I ran above;
- no-holder sample — a
-pid is still a sample, so "nearest" can legitimately be "nothing held it"; - empty window — no samples at all stays distinct from a
-sample; - rotation — the sampler's own log rotation does not change the answer.
And it carries engine#2622's padded-stamp fix with its own arm
chk(pf and pf[0][0] == 321.308031, "a padded boot stamp parses (measure with journalctl -k -o short-monotonic)")
padded = "[ 321.308031] … IO_FAULT …" ← two spaces, the real shape
The comment says why the arm had to be added: "an arm built only from [the unpadded doc example] passed while every real capture parsed as untimed." That is the third instance tonight of an arm written from an example instead of from the output — and the third to get a real-shape arm.
Nothing left from me.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6366b56: CHANGES
nearest() does exactly what its comment says — I checked all three cases — but its rule is proximity, and the reader's rule is causality, and they disagree exactly where it matters: at a handover, the fault gets attributed to a process that had not yet started holding the node.
Verified working (no build needed):
samples at 1000 (pid 111, cl-A) and 1010 (pid 222, cl-B)
fault 1009 -> 1s 222 /serve B cl-B
fault 1004 -> 4s 111 /serve A cl-A
fault 1200 -> 190s 222 /serve B cl-B (says 190s instead of pretending)
That third line is the design at its best — the distance is reported rather than a window's silence, which is precisely the failure my 93 s stall exposed.
The problem is the 1009 case. pid 222's sample is at 1010, after the fault, so 222 cannot be the holder at 1009 — the holder at that instant was 111. nearest() picks it because 1 s < 9 s. Measured against the other reader (engine#2623, merged), on a record written in this PR's format:
1009 window=1s DISAGREE HANDOVER-ADJACENT: pid 111 tk-a (next sample: pid 222)
faults: 1; agree 0, disagree 1
The reader's as-of-before rule says 111; the record says 222; it flags DISAGREE rather than hiding it. So the two halves now disagree by construction on every handover-adjacent fault — and anyone reading attributed.log alone gets the incoming holder for a fault that happened before it arrived.
What I would change (one of):
- prefer the last sample at or before the fault when one exists within the maximum gap, and only fall back to the after-sample when there is none — that matches the reader's rule instead of contradicting it, and the distance column then means "how stale was the observation", which is what the reader reports anyway;
- or write both sides (
before=111@9s;after=222@1s) and let the reader decide, which keeps the sampler's independence and makes the handover explicit in the data; - or, cheapest, mark the direction in the field (
1s+/9s-) so a reader can tell which side it got — the record already carries everything else a reader needs.
Not verified: behaviour during a real handover with the live sampler (the episode ended at 23:30 and the current capture has no new faults); the format change itself is verified through both readers on a synthetic record, which is where the disagreement appears.
|
Not a verdict (6366b56 has its two). One contract gap: the body says "a record is now So the |
Refs #2478, #2542: attribute to the nearest sample and say how far away it was
I diagnosed the gap problem wrong at first.
samplemeasures 0.02 s, so cost was not the issue — the fullholders()sweep costs 3.7 s (649 pids, 2271 fd dirs) and runs every 30 s, but the real damage is scheduling: at
nice 19on a loaded boxthe loop's 8–13 second lapses (once 93 s) are what left 28 faults with no sample inside a ±5 s window.
A fixed window narrow enough to be precise then finds nothing, and its silence reads as "no holder". So the record now carries
the distance to the nearest sample. Measured against the storm already captured:
Which is the honest shape of this instrument: 91 of 119 faults are attributable within 5 s, and 24 are attributed at more
than 10 s — weak evidence that should be visible as weak rather than either dropped (my earlier behaviour, which made the
correlation look weaker than it is) or claimed (which would have made it look stronger).
So a record is now
3s<TAB>pid<TAB>cmd<TAB>scope<TAB>ticket, and the reader sets the threshold. Self-test gains anearest-distance arm; the live unit is restarted on this rule, so the next storm's attribution arrives with its own error bar.
Two things this closes on this issue: "29 records with no holder" is retired (nothing is unattributed now, some are just far),
and the sampler's resolution is stated in seconds instead of assumed to be two.