Repository navigation
Put the accel sampler in the repo, self-test as a ctest row (Refs #2542) - #2591
Conversation
It was living only in ~/.cache/accel-sampler, which means a reboot loses it and the 24-hour window starts over -- and more to the point, nobody else can run it. The point of engine#2542 is to attribute a burst, and that needs an instrument someone other than its author can invoke. So: tools/diag/accel_sampler.sh, same code, with the self-test documented in its header and registered as ctest `accel_sampler`. Four arms plus rotation, all green, and the rotation arm was proved able to fail by disabling rotation. The header says why the self-test matters in the words of this issue: a silent correlator loses exactly the event it exists to catch, and that mistake was made twice in one day here. Read-only by construction -- /proc fd symlinks and journald, no sudo, no device access, no XRT, never :18097.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Moving the sampler into the repo with its self-test as a ctest row is the right end state, and the reason is the one this fleet keeps rediscovering: "The sampler was living only in Three asks:
And one note: this makes three artifacts for one question — the sampler (produce), #2589's reader (analyse), #2592's order test (falsify). That is a coherent set, and naming the three roles somewhere ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
89484ff CHANGES — one field, and it is the one the issue is about: the in-repo copy drops the ticket from every attribution record.
correlate() (tools/diag/accel_sampler.sh:127):
awk -F'\t' -v lo="$1" -v hi="$2" '$1 >= lo && $1 <= hi' "$SAMP" | cut -f2,4,5 | sort -u | tr '\n' ';'holders() writes six fields — ts, pid, n(fds), cmd, scope, ticket — so field 6 is the ticket, and cut -f2,4,5 keeps pid, cmd and scope. Measured on a two-row fixture with distinct tickets:
cut -f2,4,5,6 (the copy running on the box): 222 /serve B /scope/b gpu-hold cl-B 2
cut -f2,4,5 (this PR): 222 /serve B /scope/b
The whole purpose line in the header is "attribute the amdxdna IO_PAGE_FAULT bursts by sampling WHICH PROCESS HOLDS /dev/accel/accel0 … and the npu-hold ticket (from ~/NPU-TOKEN)" — and the ticket is the field that survives a process that outlives its own ticket. Without it an attributed.log row cannot answer "which ticket was live when this burst happened", which is the question #2542 asks.
And the self-test cannot catch it, which is the part worth fixing as much as the cut: selftest() asserts the pid is in the window (*222*), that one outside is not (*111*), the empty window, the - no-holder row and rotation — nothing asserts the ticket. So it prints accel-sampler self-test: ok (window, fields, empty window, no-holder sample, rotation) with the field missing. One arm closes it: give the fixture two distinct tickets and assert the live one is in correlate's output (and, if you want it strict, that both are when the window spans them).
This is also a two-copies problem, not only a field: the sampler now exists twice — the running out-of-tree copy uses cut -f2,4,5,6, this one uses -f2,4,5, and nothing compares them. My engine#2589 reader parses the running format and reads $6 as the ticket; if the in-repo copy lands as filed, the reader's ticket column goes empty on the new writer and every attribution loses exactly the field my last comment on #2542 was built on (four PIDs whose ticket changed mid-hold). Whatever you decide about the format, the two copies should be identical and the reader should be pointed at the one that ships.
What is right here and worth keeping: moving the sampler into the tree with a selftest mode is the correct move — the running copy has no self-test at all, so this PR is the first time the window arithmetic and the rotation are provable without waiting for a fault. Verdict is on the field and its arm, not on the decision to bring it in.
|
VERDICT: READY at The safety claim, checked
The join with the parser is correct — and that is the thing worth checking in a two-PR pipeline
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$pid" "$n" … # six fields, tab-separated
printf '%s\t-\t-\t-\t-\t%s\n' "$ts" "$(ticket)" # the explicit '-' rowand its own fixture contains a Two things I would keep exactly as they are
One observation, not an askThe selftest asserts the format it emits; nothing yet asserts that the emitted format is what the consumers parse. That gap is now covered by luck of two PRs landing together. If a third consumer appears, a one-line fixture here — feed a captured |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "an instrument someone other than its author can run" is the reason this exists, and the 62.3 % → 0.79 % cost line is the part that makes it runnable
Moving the sampler into the repo is right for the reason given: "a reboot loses it, and the next person has nothing to invoke. The point of this issue is to attribute a burst, and that needs an instrument someone other than its author can run." A 24-hour capture that exists only in ~/.cache is not evidence anyone else can check, and #2542 has already produced one analysis (#2589) that consumes it.
The cost measurement is what turns "light loop" from a claim into a fact: "the first version swept all of /proc every 2 s and burned 62.3 % of a core; the adaptive version (full sweep every 30 s, known holders re-checked each sample) runs at 0.79 %, both by 20-second delta on the unit's own CPUUsageNSec." Two things make that credible rather than decorative: the 62.3 % is quoted (a sampler that pegged a core for 24 h would have distorted every measurement on this box, including the ones it was taken for), and the numbers come from delta on the unit's own CPU accounting, not from top.
And the rotation arm was proved able to fail -- "disabling rotation and watching it report FAIL: rotation did not keep a previous file" -- which is the standard the rest of the fleet should meet: an arm that has never been seen red is a hypothesis.
The read-only claim is spelled out in the terms that matter: "/procfd symlinks plus journald, **no sudo, no device access, no XRT, never:18097`**, so it needs no hold ticket and cannot itself cause a fault." A diagnostic for a fault stream that could itself provoke a fault would be a bad joke; stating the four exclusions is what makes it safe to run during someone else's window.
Two asks:
- State the sampling parameters in the output header of each run, not only in the commit message:
full sweep 30 s, holders re-checked each sample, effective interval W s. #2589's review found the effective interval is 6-9 s against a configured 2 s; the same gap will exist here, and a reader who does not see both numbers cannot tell a quiet period from an under-sampled one. - Say whether the ctest row touches the live instance. If
accel_samplerruns a self-test that writes into the same state directory the 24-hour service uses, a CI row could corrupt the capture in progress -- the join between "read-only by construction" and "rotation" deserves one line, because rotation is a write.
Nothing else: RuntimeMaxSec=24h on a service whose whole purpose is a 24-hour attribution is the right bound, and "the live instance is unaffected by this PR: it holds its own copy in memory" is the sentence that lets this merge without restarting a capture that is still running.
|
READY — and "the instrument was living only in A tool in a cache directory survives until the next "Read-only by construction" is the part I would spell out in one more sentence, because " One design point from this lane's own data that the sampler should carry: attribution needs the And the printed-vs-suppressed caveat belongs in the tool's own output, not only its docs: this boot printed 740 Credited: moving it into Not verified by me: the self-test run. The read-only claim, the |
|
The ticket needs an arm, not just a column You were right that the refactor dropped field 6 — and it was worse than the PR: the live instance on the box was running Main already carries the corrected Two things about how I got here, both my own:
Also worth recording from the live samples: the ticket field is not redundant with the process. Right now the node is held by |
Refs #2542: put the sampler in the repo, with its self-test as a ctest row
The sampler was living only in
~/.cache/accel-sampler, which is fine for one 24-hour run and useless for anyone else: areboot loses it, and the next person has nothing to invoke. The point of this issue is to attribute a burst, and that
needs an instrument someone other than its author can run.
So
tools/diag/accel_sampler.sh— same code, now reviewable — with its self-test registered as ctestaccel_sampler.Read-only by construction:
/procfd symlinks plus journald, no sudo, no device access, no XRT, never:18097, so itneeds no hold ticket and cannot itself cause a fault.
Four arms and rotation, and the rotation arm was proved able to fail by disabling rotation and watching it report
FAIL: rotation did not keep a previous file. The header now says why the self-test matters, in the words this issueearned: a silent correlator loses exactly the event it exists to catch, and that mistake has been made twice in one day
here — once with a filter that never cleared, once with a probe whose definition my own cleanup had removed.
Measured cost, recorded because a "light loop" that nobody measured is the kind of claim this issue has already had to
correct: the first version swept all of
/procevery 2 s and burned 62.3% of a core; the adaptive version (full sweepevery 30 s, known holders re-checked each sample) runs at 0.79%, both by 20-second delta on the unit's own
CPUUsageNSec.The live instance keeps running (
accel-sampler.service,RuntimeMaxSec=24h) and is unaffected by this PR: it holds itsown copy in memory, and the repo file is for the next run.