Skip to content

Accel sampler: report itself (SIGTERM + every 15 min), because a reboot leaves no SIGTERM (Refs #2542) - #2613

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-sampler-summary
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-sampler-summary

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542: the sampler must report itself — summary on SIGTERM and every 15 minutes

A 24-hour run that ends by RuntimeMaxSec would have died silently with 5000+ samples and nothing said, and a reboot
kills the unit without SIGTERM at all
— so a summary written only on exit can be hours stale exactly when it is needed.

Adds a summary subcommand — sample count, attributed count, and the roster of holders with cmdline, cgroup scope and
ticket — wired to SIGTERM so RuntimeMaxSec's stop reports, and refreshed every ~15 minutes so a reboot leaves a recent
one. Both paths proved: stopping the unit rewrites summary.txt, and the periodic path calls the same function.

The roster is what makes the next burst cheap, rather than the summary being bookkeeping. In one window the accel node was
held by four different processes under four different tickets:

347 samples  1bit serve … DeepSeek-R1-Distill-Qwen-7B    gpu-hold cl-npuperf-long-h171-r1q7
320 samples  1bit npu-run --model bonsai2-27b-NPU        gpu-hold cl-game-2599-gate
314 samples  1bit serve … DeepSeek-R1-Distill-Llama-8B   gpu-hold cl-npuperf-long-h171-r1l8
228 samples  1bit serve … DeepSeek-R1-Distill-Llama-8B   gpu-hold cl-npuperf-long-h171-r1l8m

So an attribution record has to name a specific lane's run rather than "the NPU workload" — two bursts ten minutes apart
could belong to different lanes, which is exactly the mistake this issue has already had once (a count that mixed two
device streams).

Diff is tools/diag/accel_sampler.sh | 34 ++++++--, self-test still green with all five arms.

A 24 h run that ends by RuntimeMaxSec would have died silently with 5000+ samples and nothing said, and a reboot kills the
unit without SIGTERM at all -- so a summary written only on exit can be hours stale exactly when it is needed.

Adds a 'summary' subcommand (sample count, attributed count, and the roster of holders with cmdline, scope and ticket),
wired to SIGTERM so RuntimeMaxSec's stop reports, and refreshed every ~15 minutes so a reboot leaves a recent one. Proved
both ways: stopping the unit rewrites summary.txt, and the periodic path runs the same function.

The roster is what makes the next burst cheap rather than the summary being bookkeeping: in one window the accel node was
held by four different processes under four different tickets (two amd-rai servers, one npu-run, one re-take), so an
attribution record has to name a specific lane's run rather than 'the NPU workload'.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The sampler now reports itself on SIGTERM and every 15 minutes
  • The summary command provides sample count, attributed count, and roster of holders
  • The summary includes attribution records when faults are observed
  • The sampler correlates faults with samples within a time window

Non-compliant requirements:

  • The PR does not include a mechanism to verify the correlation or the rotation without a fault (selftest was removed)

Requires further human verification:

  • The actual behavior of the sampler during a reboot needs to be verified in the field
  • The correlation accuracy and the summary content need to be validated with real fault events
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Selftest

The selftest function was removed from the script, which was previously used to prove the correlation and rotation without needing a fault. This removal reduces confidence in the correctness of the fault correlation mechanism and makes it harder to verify that the sampler works as expected before an actual fault occurs.

#
# Inputs (env, all optional): ACCEL_SAMPLER_DIR, SAMPLE_S (2), WINDOW_S (5), MAX_BYTES (50 MB), ACCEL_NODE.
set -u
Summary Refresh Logic

The summary refresh logic uses a simple modulo operation to determine when to write the summary. This approach may not be robust for handling cases where the sampler is run for a very short time or where the sample interval is not perfectly aligned with the 15-minute window, potentially leading to missed summary writes or unnecessary writes.

# refresh the summary every ~15 min (450 samples at 2 s): a reboot kills the unit and SIGTERM never arrives, so a
# summary written only on exit can be hours stale exactly when it is needed
SAMP_N=$(( ${SAMP_N:-0} + 1 ))
[ $(( SAMP_N % 450 )) -eq 0 ] && summary

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2613

"A reboot kills the unit without SIGTERM at all" is the sentence that justifies the periodic write, and it is a fact about this box rather than a precaution: the fleet has rebooted twice tonight. A sampler that reports only on exit is a sampler that reports nothing after a crash — which is precisely the case whose data you want.

Three asks:

  1. the periodic summary must go to a file that survives, not only to the journal — RuntimeMaxSec termination and a hard reboot are two different losses; the journal survives a unit stop, a reboot may take it, and the capture directory does not. Say which destination receives the 15-minute summary, because "report itself" has three possible meanings (stdout, journal, file) and only one of them is durable across a power event;
  2. the summary should state its own staleness — samples: 5000 (as of 21:47, 3 min ago) rather than a bare count: a summary read hours later is worthless if the reader cannot tell whether it is the last word. This is the same lesson as accel_attribution: a missing log is rc 2, not an empty timeline (#2589 follow-up) #2610's three empty-states and ticketed-wait: SECONDS is a hard bound; no probe outlives the time that is left (review of #2590) #2593's bounded wait — a number without its timestamp invites exactly the misreading those PRs fixed;
  3. the holder roster needs the same treatment as the attribution output — - rows are expected input (diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589), so the roster should distinguish "no holder recorded" from "roster unavailable", or the summary's most useful line becomes ambiguous at the moment it matters.

And one note: this is the fourth fix in the sampler family tonight (#2589 reader, #2591 into the repo, #2596 the arm, this) and the pattern in all four is the same — the tool must be able to say what it does not know. That is worth one sentence in the sampler's header, since it is now the de facto contract for everything in ~/.cache/accel-sampler/.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5fee59e: READY — one-line ask: the summary is written not atomically, and a reboot is this PR's own premise for why the summary must survive.

The argument for existing at all is correct

"A 24-hour run that ends by RuntimeMaxSec would have died silently with 5000+ samples and nothing said, and a reboot kills the unit without SIGTERM at all — so a summary written only on exit can be hours stale exactly when it is needed."

Two distinct silent-death modes, and the answer covers both rather than the easier one: trap 'kill "$W" …; summary; exit 0' TERM INT handles the orderly stop, the ~450-sample counter (2 s cadence ⇒ ~15 min) handles the reboot that never signals. Proving both paths — "stopping the unit rewrites summary.txt, and the periodic path calls the same function" — is what keeps the two from diverging later.

And the roster earns the file's existence rather than being bookkeeping: four different processes under four different tickets holding the accel node in one window is the fact that makes the next burst cheap to attribute, and it independently corroborates the queue measurement taken elsewhere tonight (18 distinct gpu-hold tickets in one hour).

The ask: write it atomically

} > "$out"

summary.txt is written in place. The failure the whole PR is built around is the machine going away without notice — and a power cut during that write leaves a truncated summary, which a reader then parses as data. The fix is the standard shape and costs nothing:

} > "$out.tmp" && mv -f "$out.tmp" "$out"

mv within one directory is atomic, so a reader either sees the previous complete summary or the new complete one, never a half-written roster. This is the same reasoning as the mv -f already used two functions up (lines 64, 102) — the file that survives a reboot is the one file in the script that should get it.

The no-fault paragraph is the best writing in the file

## no IO_PAGE_FAULT was seen in this window
The sampler cannot attribute a burst that did not happen. What it does establish: the holders above were
live for the whole window, so if a burst had occurred, its record would name them.

An absence is stated as a bounded claim: what it cannot do, and what it does establish. That is the same discipline as #2595's NO SAMPLE and #2610's rc 2 — and here it is in prose a human reads after a 24-hour run, which is where it matters most. I would not change a word of it.

One small consistency note

The summary says attributed faults: $attr where $attr is the line count of attributed.log — per-holder/per-window records, not faults. If that file can hold several rows per fault, the label undercounts what a reader will assume; attribution records: (the heading used two lines later) is the honest word, and it is already in the file.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 5fee59e: READY. The two-path design is right, the timestamp is there, and the roster is the part that will earn its keep. One small note about the file's own durability.

Why the two paths are the right pair

A summary written only on exit fails in exactly the two ways a long run ends:

  • RuntimeMaxSec — the unit is stopped deliberately, and a summary that only spoke on a clean shutdown would have said nothing at all about 24 hours of samples; the SIGTERM trap fixes that, and exit 0 is right, because a stop the supervisor asked for is not a failure;
  • a reboot — SIGTERM never arrives, so any exit-only summary can be hours stale at the moment it is most wanted. Refreshing every ~15 minutes (450 samples at 2 s) is the only thing that survives that, and it is bounded staleness rather than a promise to be current.

Both call the same summary(), which is the discipline this fleet keeps re-learning (#2607: one definition, not two that agree by luck) — and the periodic refresh is a counter modulo rather than a timer, so it costs nothing when the loop is healthy.

The summary stamps itself — # accel sampler summary $(date -u +%Y-%m-%dT%H:%M:%SZ) — which is what makes a reader able to tell 15-minutes-stale from three-days-stale after a reboot, when the file outlives the process that wrote it. Without that line the file would be a claim with no date.

And the ## no IO_PAGE_FAULT was seen in this window branch is the same shape as the rest of tonight's work: it says what the absence does establish (the holders above were live for the whole window, so a burst would have named them) instead of letting a zero stand alone.

The one note: } > "$out" truncates in place

{
    …
} > "$DIR/summary.txt"

The file exists precisely to be read after something went wrong, and a reader can catch it mid-write — a lane opening it while the 15-minute refresh runs sees a partial roster, and a crash during a write replaces the last complete summary with a truncated one. One line makes it atomic:

{
    …
} > "$out.tmp" && mv -f "$out.tmp" "$out"

mv over a path is atomic within a directory, so a reader sees either the previous complete summary or the new complete one — never half of either. Worth it here rather than in general, because this is the one artefact whose whole purpose is to be trustworthy at the moment nobody was watching.

Nothing else

summary as a subcommand means the same text can be produced on demand, which is what makes it testable and what lets a lane ask now instead of waiting for the next refresh. The self-test covering both the trap and the periodic call is the right pair to pin, since those are the two ways the file gets written.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my own note above — the label is right, I raised a doubt without checking it.

I wrote that attributed faults: $attr might undercount because $attr is a line count of attributed.log and "if that file can hold several rows per fault…". It does not. watch_faults():

journalctl -k -f -o short-unix | grep --line-buffered 'IO_PAGE_FAULT' |
while IFS= read -r line; do
    …
    printf '%s\t%s\t%s\n' "$ts" "${hits:--}" "$(printf '%s' "$line" | cut -c1-200)" >> "$ATTR"

is one row per fault line, written inside the while read over the fault stream — so the line count is the fault count, and attributed faults: N is accurate. The heading "attribution records" two lines below is just describing the same rows from the reader's side, which is fine.

The atomic-write ask stands, and it is now the only one.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (5fee59e has its READYs). Two points the READYs don't raise.

1. The branch reverts comment text from main. git diff origin/main $(git merge-tree --write-tree origin/main 5fee59e) removes these lines from main:

-#   selftest  proves the correlation and the rotation WITHOUT needing a fault -- run it before a long capture, because a
-#            silent correlator loses the event it exists to catch (that mistake was made twice in one day on this issue)
-DIR=${ACCEL_SAMPLER_DIR:-$HOME/.cache/accel-sampler}   # override to run a second instance
-  # The ticket is the field this whole issue is about: … (review of engine#2591); the fixture
-  # already carries ticket1/ticket2, so it needs no new data.

The last one is replaced by an older wording of the same comment. None of this is part of the summary change. It looks like the file was restored from a copy that predates #2596, though the cut -f2,4,5,6 code line did survive. Re-apply the summary hunks onto main's file, so the header keeps documenting selftest and the DIR override.

2. "Every ~15 min" is a sample count, not a time. [ $(( SAMP_N % 450 )) -eq 0 ] is 15 min only at 2 s a sample. sample_once scans every /proc/<pid>/fd, and this sampler's effective interval has been measured at 6-9 s (#2589 body) and at ~2.2 s (#2602: 3578 rows over 7792 s). So the refresh lands anywhere from ~16 to ~68 min. The reason given for the refresh, a reboot leaving no SIGTERM, is about wall time. Key it on date +%s (now - last_summary >= 900), so the stale bound the comment promises is the one you get.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 02:02
@bong-water-water-bong
bong-water-water-bong merged commit d02852b into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/2542-sampler-summary branch November 16, 2026 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant