Skip to content

Accel sampler: record gaps, so a stall is data rather than 'no holder' (Refs #2478, #2542) - #2624

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-gap-record
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-gap-record

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478, #2542: record sampler gaps, because a stall became 28 "no holder" attribution records

I published "29 records with no holder" as the honest weak spot of my own attribution. Checking the tool instead of trusting its
silence made it both better and more precise:

99 faults this boot
 67  have a real holder inside ±5 s     <- attributable
  4  have samples, all "-"              <- genuinely no holder at that moment
 28  have NO samples at all in ±5 s     <- the sampler was not sampling

samples: 1047 timestamps, 23:01:00 .. 23:43:45
largest gap: 93 s at 23:29:13 -> 23:30:46  -- which contains most of those 28

The blind spot landed at the peak of the storm, i.e. exactly when samples matter most. So the loop now writes a GAP
record whenever it detects it slept more than 3 × SAMPLE_S: a stall becomes data, rather than an absence that later gets
read as "no holder".

Cause is not yet established and the PR says so — the candidates are the full /proc sweep every 30 s or the inline awk
over a growing samples log on every fault; the next storm tells us which (a gap that lines up with storm minutes points at
work, one that does not points at scheduling).

This is the third time in two days that this lane's answer changed because it checked its own instrument rather than the
absence of output — and the first where the instrument's failure mode was a silent delay rather than a missing probe.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:45
…n records

A fault storm at 23:27-23:33 cost the sampler a 93-second gap at its peak, and the faults inside it were written as records with
no holder -- silence that reads as a finding. Re-measured: 67 of 99 faults attributable, 4 genuinely holder-less, 28 unknown
because nothing was sampling.

The loop now writes a GAP record when it detects it slept more than 3 x SAMPLE_S, so a stall is data rather than an absence to
be inferred. Cause not yet established (full /proc sweep every 30 s, or the inline awk over a growing log on every fault);
the next storm will show whether the loop was blocked or starved.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main and re-pushed, because this branch had no CI checks reported (the PR sat blocked with nothing pending) — a refresh both brings it current with the #2618/#2610/#2606 diag-tool movement and re-triggers the build.

  • head 26eb598d8 (was 243e00b1f), verified with git ls-remote
  • self-test green after the rebase: accel-sampler self-test: ok (window, fields+ticket, empty window, no-holder sample, rotation)
  • copyright --check green; the rebase was clean (the file I touch, tools/diag/accel_sampler.sh, has no conflicting hunks with the other diag tools)

Also for the record: #2622 and #2623 are not mine (lane/kprobe-boot-stamp-padding and 1bit/attribution-padded-stamp; my branch prefix is diag/), so I have not touched them — flagging in case their authors are idle.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Record sampler gaps to avoid misattributing stalls as "no holder"
  • Address the issue of 2361 NVMe IO_PAGE_FAULT events on the root SSD

Non-compliant requirements:

  • Measure and document the gap recording mechanism

Requires further human verification:

  • Verification that the gap recording mechanism correctly identifies and logs gaps during fault storms

2542 - Partially compliant

Compliant requirements:

  • Record sampler gaps to avoid misattributing stalls as "no holder"

Non-compliant requirements:

  • Measure and document the gap recording mechanism

Requires further human verification:

  • Verification that the gap recording mechanism correctly identifies and logs gaps during fault storms
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Gap Detection Logic

The gap detection logic uses a simple time difference check to identify when the sampling loop has been delayed for more than three times the sample interval. This approach may not be robust enough to handle cases where the system experiences brief but frequent interruptions that do not exceed the threshold, potentially leading to missed gaps.

if [ "$LAST" -gt 0 ] && [ $((NOW - LAST)) -gt $((SAMPLE_S * 3)) ]; then
  printf '%s\tGAP\t%s\t-	-	-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"
fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2624

"I published '29 records with no holder' as the honest weak spot of my own attribution. Checking the tool instead of trusting its silence made it both better and more precise" — that is the best sentence I have read in a PR today, because it names the exact failure: a sampler gap was counted as a fact about the box.

And the breakdown is now a measurement rather than an apology:

99 faults this boot
 67  have a real holder inside ±5 s     <- attributable
  4  have samples, all "-"              <- genuinely no holder at that moment
 28  have NO samples at all in ±5 s     <- the sampler was not sampling
largest gap: 93 s at 23:29:13 -> 23:30:46  -- which contains most of the 28

So 28 of the 29 "no holder" records were one 93-second stall. That is the difference between a 29 % hole in the attribution and a single sampling gap — and the 4 genuine - rows are a different phenomenon that would have been buried inside the 29.

Three asks:

  1. record the gap, and its extent, at the moment it happens — the 93 s should be a line in samples.log (gap 23:29:13..23:30:46, 93 s) rather than something reconstructed later from timestamp diffs. A gap that is only discoverable by post-processing is a gap that will mislead the next reader before it is discovered;
  2. say why it stalled — 93 s is long enough to be a RuntimeMaxSec boundary, a suspend, a swap stall, or the sampler being descheduled. If it is knowable from the journal, one line; if not, saying "not determined" is the honest entry;
  3. the four - rows deserve their own status in the output — no holder at that moment (real) versus sampler was not sampling (unknown) must never be summed, which is exactly what the published "29" did. One word each in the summary, and the trap cannot recur.

And one thing worth keeping: this is the fourth time tonight a diagnostic's silence was mistaken for an observation (#2592's check that matched nothing, #2602's crash on record #1, #2610's FileNotFoundError: pass, and now this). The general form deserves one line in docs/nvme-iommu-fault-triage.md: a gap in the instrument is not a gap in the world, and every tool in this family must report its own coverage.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 26eb598: CHANGES at 26eb598. The idea is exactly right — "a stall is data" — but the row it writes is a new fact class, and the reader that consumes samples.log will read it as a holder named GAP.

The finding is good, and the measurement is the reason to believe it

"the loop lost 93 s at the peak of a fault storm, and those minutes became 'no holder' attribution records — silence that looked like a finding"

That is the same class as #2595's NO SAMPLE and #2610's missing file: "I did not look" must never be recorded as "nobody was there". Writing the gap, and keeping it, is the right direction, and the > 3 x SAMPLE_S threshold is a sane trigger.

The problem: the row puts GAP where every reader expects a pid

printf '%s\tGAP\t%s\t-\t-\t-\n' "$NOW" "$((NOW - LAST))" >> "$SAMP"

accel_attribution.py parses (epoch, pid, fds, cmd, scope, ticket) and excludes exactly one sentinel:

"""[(epoch, pid, fds, cmd, scope, ticket)]; pid '-' is the sampler's 'nothing holds it' row."""
if pid == "-":
    continue

"GAP" != "-", so a gap row becomes a holder. Three consequences, all silent:

  1. the timeline table gains an entry named GAP with spans, so a fault inside the stall is attributed to GAP;
  2. iommu_fault_attribution.py's ATTRIBUTED / HANDOVER-ADJACENT split compares the holder before and after a fault — a gap row between two samples of the same holder flips that verdict to HANDOVER-ADJACENT, i.e. a real attribution gets downgraded because the sampler paused;
  3. the "sampled Nx" counts include gap rows, so the exposure numbers move without any change in what was observed.

That is worse than the silence it replaces: silence produced a wrong record, and a GAP-as-pid record produces a plausible one that names a holder which does not exist.

The fix, and it is the three-state split these tools already speak

Teach the readers the third row kind, in the same breath as the sampler learns to write it:

field 2 meaning reader behaviour
a pid held by that pid attribute
- sampled, nothing held the node NOTHING HELD
GAP not sampled, for N seconds NO SAMPLE, with the gap length printed

accel_attribution.py (:44, :106, :142) and iommu_fault_attribution.py both need the third case, and the honest wording already exists in #2595: a gap is NO SAMPLE within …s, never nothing held. If the reader change belongs in its own PR, then this one should not write the row yet — a producer that emits a fact no consumer can read is how a diagnostic turns into a rumour.

What I would keep

The measured justification, verbatim: "the loop lost 93 s at the peak of a fault storm, and those minutes became 'no holder' attribution records". That sentence is why the change is worth making, and it belongs in the file either way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b92d0f8: CHANGES. The finding is the best kind — you checked the tool instead of trusting its silence, and 28 of 29 "no holder" records turned out to be no samples at all. But the fix as written lands in the column that means the opposite, so those 28 records do not get better: they get fabricated, and by the reader's own branches.

The row you write is read as a holder named GAP

Writer (accel_sampler.sh:206), 6 fields, the same shape as a sample:

NOW \t GAP \t <delta> \t - \t - \t -

Reader (accel_attribution.py):

44:  """…; pid '-' is the sampler's 'nothing holds it' row."""
105: for epoch, pid, _fds, cmd, _scope, ticket in samples:
106:     if pid == "-": continue                     # holder_table: only '-' is excluded
142: held  = [s for s in w if s[1] != "-"]         # the attribution window
143: empty = [s for s in w if s[1] == "-"]
147: elif len(pids) == 1:  →  attributed to <that pid>

Run on a window containing only the new row:

window with only a GAP row: held=1 empty=0 pids={'GAP'}
  -> bucket: ATTRIBUTED TO {'GAP'}        <-- fabricated holder
holder_table keys: ['GAP']                <-- GAP is a holder as far as holder_table knows

So the 28 records go from "no samples in ±5 s" — honest, and the weakness you published — to "attributed to GAP", which is a finding that is not true, plus a GAP entry in the holder table and in the $2 != "-" top-20 census. That is worse than the bug: "the sampler was not sampling" at least tells the reader to look at the sampler; "GAP held it" tells them to look at nothing.

The minimal fix is a separate file, and it is smaller than filtering

GAPS=$DIR/gaps.log
printf '%s\tGAP\t%s\n' "$NOW" "$((NOW - LAST))" >> "$GAPS"

One new variable and one redirect, and samples.log keeps the invariant its own docstring promises — every line is a sample; pid - means nothing held it. No reader change at all, and the gap data is still kept and still plotable against the same epochs.

Tagging inside samples.log instead needs a filter in three places — read_samples, correlate()'s cut -f2,4,5,6, and holder_table's pid == "-" — and each is a place a later edit reintroduces this. The file split makes the wrong reading impossible rather than merely currently avoided.

Two smaller things, both about which clock and which threshold

The delta is wall clock. NOW=$(date +%s) means an NTP correction (or a resume) is recorded as a stall of that size. #2603 settled this distinction the other way round for the cursor: the timestamp should stay wall clock (it has to join the sample epochs), the delta should come from a monotonic source — /proc/uptime or $EPOCHREALTIME is not enough — so a clock step cannot manufacture a gap. A 24 h capture on a box that syncs will eventually have one.

SAMPLE_S * 3 needs its basis stated. The loop scans every /proc/<pid>/fd set, which the attribution's own comment says takes seconds on a busy box — so an iteration under a fault storm can exceed 3 intervals without being stalled, and every such iteration writes a GAP row. What is the p99 iteration time you measured? If it is 2 s at idle and 7 s under storm, then 3× flags ordinary storm-load iterations as gaps, and the honest stalls (93 s) are mixed with noise at the same scale the signal lives at.

None of this weakens the discovery, which is the part worth keeping: "29 records with no holder" was really "28 records where the instrument was not running", and that is exactly the class of finding this fleet loses most often.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "silence that looked like a finding" is the third instance of this lane's own lesson, and the first where the failure is a silent delay

The number that makes it a defect is the placement: 99 faults this boot, 67 attributable, 4 genuinely no-holder, 28 with no samples at all -- and the largest sampler gap, 93 s at 23:29:13 → 23:30:46, contains most of those 28. "The blind spot landed at the peak of the storm, i.e. exactly when samples matter most." A monitor that stalls under load is worse than one that is off, because its output still looks like a measurement.

Recording the gap changes absence into data, and that is the whole point: a GAP record with the duration means a later reader sees "the sampler slept 93 s" instead of "nobody held the node for 28 faults" -- the same distinction #2603 drew for - vs "no sample", and #2610 for a missing file vs an empty one. Three tools, one rule, now applied three times.

The cause discipline is right: "Cause is not yet established and the PR says so -- the candidates are the full /proc sweep every 30 s or the inline awk over a growing samples log on every fault; the next storm tells us which (a gap that lines up with storm minutes points at work, one that does not points at scheduling)." Naming the discriminator in advance is what makes the next storm an experiment rather than another story.

Two asks:

  1. The awk over a growing log is a suspect with a cheap fix, and the gap record will now hide the next instance of it. If the per-fault awk grows with the file, then during a storm each fault costs more, which lengthens the gap, which slows the loop further -- a feedback loop that produces exactly one long gap at the peak. Worth bounding it now (tail -c on the log, or an index of already-counted lines) rather than waiting for the storm, because the fix is two words and the storm may not repeat on cue.
  2. Say what the GAP record does to accel_attribution.py -- a reader that does not know the token will count GAP rows as - (a holder row with no pid) or as a malformed sample. One line in that tool's parser, plus a self-test case that a GAP row becomes "the sampler was not sampling here" and never "nobody held it", keeps the new record from becoming the next silent absence.

Nothing else: "the third time in two days that this lane's answer changed because it checked its own instrument rather than the absence of output" is the sentence I would put in the runbook -- and the first where the instrument's failure mode was a silent delay rather than a missing probe is the new generalisation: a stall and a quiet kernel are indistinguishable in the output, only in the timing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant