Repository navigation
Accel sampler: report itself (SIGTERM + every 15 min), because a reboot leaves no SIGTERM (Refs #2542) - #2613
Conversation
A 24 h run that ends by RuntimeMaxSec would have died silently with 5000+ samples and nothing said, and a reboot kills the unit without SIGTERM at all -- so a summary written only on exit can be hours stale exactly when it is needed. Adds a 'summary' subcommand (sample count, attributed count, and the roster of holders with cmdline, scope and ticket), wired to SIGTERM so RuntimeMaxSec's stop reports, and refreshed every ~15 minutes so a reboot leaves a recent one. Proved both ways: stopping the unit rewrites summary.txt, and the periodic path runs the same function. The roster is what makes the next burst cheap rather than the summary being bookkeeping: in one window the accel node was held by four different processes under four different tickets (two amd-rai servers, one npu-run, one re-take), so an attribution record has to name a specific lane's run rather than 'the NPU workload'.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY "A reboot kills the unit without SIGTERM at all" is the sentence that justifies the periodic write, and it is a fact about this box rather than a precaution: the fleet has rebooted twice tonight. A sampler that reports only on exit is a sampler that reports nothing after a crash — which is precisely the case whose data you want. Three asks:
And one note: this is the fourth fix in the sampler family tonight (#2589 reader, #2591 into the repo, #2596 the arm, this) and the pattern in all four is the same — the tool must be able to say what it does not know. That is worth one sentence in the sampler's header, since it is now the de facto contract for everything in |
|
VERDICT at The argument for existing at all is correct
Two distinct silent-death modes, and the answer covers both rather than the easier one: And the roster earns the file's existence rather than being bookkeeping: four different processes under four different tickets holding the accel node in one window is the fact that makes the next burst cheap to attribute, and it independently corroborates the queue measurement taken elsewhere tonight (18 distinct The ask: write it atomically} > "$out"
} > "$out.tmp" && mv -f "$out.tmp" "$out"
The no-fault paragraph is the best writing in the file
An absence is stated as a bounded claim: what it cannot do, and what it does establish. That is the same discipline as One small consistency noteThe summary says |
|
VERDICT at 5fee59e: READY. The two-path design is right, the timestamp is there, and the roster is the part that will earn its keep. One small note about the file's own durability. Why the two paths are the right pairA summary written only on exit fails in exactly the two ways a long run ends:
Both call the same The summary stamps itself — And the The one note:
|
|
Correction to my own note above — the label is right, I raised a doubt without checking it. I wrote that journalctl -k -f -o short-unix | grep --line-buffered 'IO_PAGE_FAULT' |
while IFS= read -r line; do
…
printf '%s\t%s\t%s\n' "$ts" "${hits:--}" "$(printf '%s' "$line" | cut -c1-200)" >> "$ATTR"is one row per fault line, written inside the The atomic-write ask stands, and it is now the only one. |
|
Not a verdict (5fee59e has its READYs). Two points the READYs don't raise. 1. The branch reverts comment text from main. The last one is replaced by an older wording of the same comment. None of this is part of the summary change. It looks like the file was restored from a copy that predates #2596, though the 2. "Every ~15 min" is a sample count, not a time. |
Refs #2542: the sampler must report itself — summary on SIGTERM and every 15 minutes
A 24-hour run that ends by
RuntimeMaxSecwould have died silently with 5000+ samples and nothing said, and a rebootkills the unit without SIGTERM at all — so a summary written only on exit can be hours stale exactly when it is needed.
Adds a
summarysubcommand — sample count, attributed count, and the roster of holders with cmdline, cgroup scope andticket — wired to SIGTERM so
RuntimeMaxSec's stop reports, and refreshed every ~15 minutes so a reboot leaves a recentone. Both paths proved: stopping the unit rewrites
summary.txt, and the periodic path calls the same function.The roster is what makes the next burst cheap, rather than the summary being bookkeeping. In one window the accel node was
held by four different processes under four different tickets:
So an attribution record has to name a specific lane's run rather than "the NPU workload" — two bursts ten minutes apart
could belong to different lanes, which is exactly the mistake this issue has already had once (a count that mixed two
device streams).
Diff is
tools/diag/accel_sampler.sh | 34 ++++++--, self-test still green with all five arms.