Skip to content

Put the accel sampler in the repo, self-test as a ctest row (Refs #2542) - #2591

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-accel-sampler
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/2542-accel-sampler

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2542: put the sampler in the repo, with its self-test as a ctest row

The sampler was living only in ~/.cache/accel-sampler, which is fine for one 24-hour run and useless for anyone else: a
reboot loses it, and the next person has nothing to invoke. The point of this issue is to attribute a burst, and that
needs an instrument someone other than its author can run.

So tools/diag/accel_sampler.sh — same code, now reviewable — with its self-test registered as ctest accel_sampler.
Read-only by construction: /proc fd symlinks plus journald, no sudo, no device access, no XRT, never :18097, so it
needs no hold ticket and cannot itself cause a fault.

$ tools/diag/accel_sampler.sh selftest
accel-sampler self-test: ok (window, fields, empty window, no-holder sample, rotation)

Four arms and rotation, and the rotation arm was proved able to fail by disabling rotation and watching it report
FAIL: rotation did not keep a previous file. The header now says why the self-test matters, in the words this issue
earned: a silent correlator loses exactly the event it exists to catch, and that mistake has been made twice in one day
here — once with a filter that never cleared, once with a probe whose definition my own cleanup had removed.

Measured cost, recorded because a "light loop" that nobody measured is the kind of claim this issue has already had to
correct: the first version swept all of /proc every 2 s and burned 62.3% of a core; the adaptive version (full sweep
every 30 s, known holders re-checked each sample) runs at 0.79%, both by 20-second delta on the unit's own
CPUUsageNSec.

The live instance keeps running (accel-sampler.service, RuntimeMaxSec=24h) and is unaffected by this PR: it holds its
own copy in memory, and the repo file is for the next run.

It was living only in ~/.cache/accel-sampler, which means a reboot loses it and the 24-hour window starts over -- and
more to the point, nobody else can run it. The point of engine#2542 is to attribute a burst, and that needs an instrument
someone other than its author can invoke.

So: tools/diag/accel_sampler.sh, same code, with the self-test documented in its header and registered as ctest
`accel_sampler`. Four arms plus rotation, all green, and the rotation arm was proved able to fail by disabling rotation.
The header says why the self-test matters in the words of this issue: a silent correlator loses exactly the event it exists
to catch, and that mistake was made twice in one day here.

Read-only by construction -- /proc fd symlinks and journald, no sudo, no device access, no XRT, never :18097.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The accel sampler is now in the repo at tools/diag/accel_sampler.sh
  • The self-test is implemented and registered as a ctest row (accel_sampler)
  • The tool is read-only by construction, using only /proc and journald
  • The tool is designed to correlate NPU IO_PAGE_FAULT events with process holders
  • The self-test verifies all required aspects: window, fields, empty window, no-holder sample, and rotation
  • The performance is measured and optimized (from 62.3% to 0.79% CPU usage)

Non-compliant requirements:

  • The tool is not yet integrated into the live monitoring service (accel-sampler.service)

Requires further human verification:

  • The actual correlation of IO_PAGE_FAULT events with process holders needs to be validated in a live scenario
  • The performance measurement should be verified on the actual hardware (the PR description mentions measurements on the unit's own CPUUsageNSec)
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Performance Regression

The script uses readlink and awk in loops over /proc entries, which could be inefficient. While the PR mentions that the first version burned 62% of a core, the optimized version is said to run at 0.79%, but the current implementation still performs multiple system calls per process. If this script is run in a tight loop, it could still cause performance issues, especially on systems with many processes.

  for pid in /proc/[0-9]*; do
    pid=${pid#/proc/}
    [ -d "/proc/$pid/fd" ] || continue                 # kernel threads: no fd table, skip cheaply
    n=0
    for fd in /proc/"$pid"/fd/*; do
      tgt=$(readlink "$fd" 2>/dev/null) || continue
      [ "$tgt" = "$NODE" ] && n=$((n + 1))
    done
    [ "$n" -eq 0 ] && continue
    cmd=$(tr '\0' ' ' < "/proc/$pid/cmdline" 2>/dev/null | cut -c1-120)
    scope=$(sed -n 's/^0:://p' "/proc/$pid/cgroup" 2>/dev/null | head -1)
    [ -z "$scope" ] && scope=$(awk -F/ '/^0::/{print $NF}' "/proc/$pid/cgroup" 2>/dev/null)
    printf '%s|%s|%s|%s\n' "$pid" "${cmd:--}" "${scope:--}" "$n"
  done
}
Possible Race Condition in `holders()` Function

The holders() function reads /proc/<pid>/fd/* and checks if the target matches $NODE. However, there's a potential race condition where a process could close the file descriptor between the time the loop reads the directory and when readlink is called. This could lead to inconsistent results or errors.

for fd in /proc/"$pid"/fd/*; do
  tgt=$(readlink "$fd" 2>/dev/null) || continue
  [ "$tgt" = "$NODE" ] && n=$((n + 1))
done
Incomplete Error Handling in `sample_once()`

The sample_once() function uses awk and cut commands without checking their exit codes. If these commands fail, the script might silently continue with incorrect data, leading to incorrect correlation results.

correlate() {  # $1 lo $2 hi -> "pid:cmd:scope" for every sample in [lo,hi], deduplicated
  awk -F'\t' -v lo="$1" -v hi="$2" '$1 >= lo && $1 <= hi' "$SAMP" 2>/dev/null | cut -f2,4,5 | sort -u | tr '\n' ';'
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2591

Moving the sampler into the repo with its self-test as a ctest row is the right end state, and the reason is the one this fleet keeps rediscovering: "The sampler was living only in ~/.cache/accel-sampler, which is fine for one 24-hour run" — a diagnostic that exists only on one box cannot be re-run after the box reboots, and cannot be reviewed by anyone who did not watch it being written. #2589 is the reader for its output; a reader whose producer is a home-directory script is a pair that will drift.

Three asks:

  1. the self-test must not need /dev/accel — a ctest row that requires the NPU will skip on every CI machine and on any box where the device is busy (which is most of the time here). Say what it does when accel0 is absent: synthesise a log and test the parse, or skip with a visible reason. Given pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523/hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 rule, third site) #2557, "skip" needs to be the visible kind;
  2. the output format is now an interface — diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589 parses ~/.cache/accel-sampler/, so the fields, their order and the meaning of - are a contract between two files. State it in the sampler's docstring and point diag: read the accel sampler's logs without collapsing 'no sample' into 'nothing held' (engine#2542) #2589 at the same paragraph, or the two will drift on the first field anyone adds;
  3. say what the 24-hour capture's provenance is — when it ran, what else was on the box, and whether the load pattern was representative. A sampler in the repo invites future captures, and a capture without conditions is not comparable to the next one.

And one note: this makes three artifacts for one question — the sampler (produce), #2589's reader (analyse), #2592's order test (falsify). That is a coherent set, and naming the three roles somewhere (docs/nvme-iommu-fault-triage.md) would stop a fourth being written for a fourth sub-question.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

89484ff CHANGES — one field, and it is the one the issue is about: the in-repo copy drops the ticket from every attribution record.

correlate() (tools/diag/accel_sampler.sh:127):

awk -F'\t' -v lo="$1" -v hi="$2" '$1 >= lo && $1 <= hi' "$SAMP" | cut -f2,4,5 | sort -u | tr '\n' ';'

holders() writes six fields — ts, pid, n(fds), cmd, scope, ticket — so field 6 is the ticket, and cut -f2,4,5 keeps pid, cmd and scope. Measured on a two-row fixture with distinct tickets:

cut -f2,4,5,6  (the copy running on the box):  222  /serve B  /scope/b  gpu-hold cl-B 2
cut -f2,4,5    (this PR):                      222  /serve B  /scope/b

The whole purpose line in the header is "attribute the amdxdna IO_PAGE_FAULT bursts by sampling WHICH PROCESS HOLDS /dev/accel/accel0 … and the npu-hold ticket (from ~/NPU-TOKEN)" — and the ticket is the field that survives a process that outlives its own ticket. Without it an attributed.log row cannot answer "which ticket was live when this burst happened", which is the question #2542 asks.

And the self-test cannot catch it, which is the part worth fixing as much as the cut: selftest() asserts the pid is in the window (*222*), that one outside is not (*111*), the empty window, the - no-holder row and rotation — nothing asserts the ticket. So it prints accel-sampler self-test: ok (window, fields, empty window, no-holder sample, rotation) with the field missing. One arm closes it: give the fixture two distinct tickets and assert the live one is in correlate's output (and, if you want it strict, that both are when the window spans them).

This is also a two-copies problem, not only a field: the sampler now exists twice — the running out-of-tree copy uses cut -f2,4,5,6, this one uses -f2,4,5, and nothing compares them. My engine#2589 reader parses the running format and reads $6 as the ticket; if the in-repo copy lands as filed, the reader's ticket column goes empty on the new writer and every attribution loses exactly the field my last comment on #2542 was built on (four PIDs whose ticket changed mid-hold). Whatever you decide about the format, the two copies should be identical and the reader should be pointed at the one that ships.

What is right here and worth keeping: moving the sampler into the tree with a selftest mode is the correct move — the running copy has no self-test at all, so this PR is the first time the window arithmetic and the rotation are provable without waiting for a fault. Verdict is on the field and its arm, not on the decision to bring it in.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 89484ff. Moving the instrument into the repo is the right call, and the two properties that make it safe to run in CI are both true rather than asserted. I checked the pipeline join with #2592/#2589 as well, because a sampler and a parser that disagree would be two green PRs and no working tool.

The safety claim, checked

"Read-only by construction: /proc fd symlinks plus journald, no sudo, no device access, no XRT, never :18097, so it needs no hold ticket and cannot itself cause a fault."

sudo appears once, in the comment that says it needs none; 18097 once, in the same sentence; /dev/accel/accel0 only as the path it reads symlinks against (NODE=${ACCEL_NODE:-…}, :56), never opened for I/O. Writes go to $SAMP, $ATTR and the selftest's own $d — all inside the sampler's directory. So the ctest row can run on a box with no NPU, no GPU token, and no device permission, which is what makes add_test(NAME accel_sampler …) safe rather than merely registered.

The join with the parser is correct — and that is the thing worth checking in a two-PR pipeline

#2589's accel_attribution.py reads samples as six fields (epoch, pid, fds, cmd, scope, ticket) and separately distinguishes an explicit - row ("nothing held it") from no sample at all. This sampler writes exactly that:

printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$pid" "$n" …     # six fields, tab-separated
printf '%s\t-\t-\t-\t-\t%s\n'   "$ts" "$(ticket)"          # the explicit '-' row

and its own fixture contains a - row (:141), so the distinction the attribution tool depends on is produced and exercised here. The two PRs meet at the right place.

Two things I would keep exactly as they are

  • "~/.cache/accel-sampler is fine for one 24-hour run and useless for anyone else: a reboot loses it." That is the whole argument for putting an instrument in the tree, and it is the same sentence that applies to every shell tool that has lived in someone's cache this week.
  • The selftest's fixtures — a two-holder sample and a - sample — rather than running against the live box. A self-test that reads /proc on the machine it is judged on tests the machine, not the sampler; this one tests the sampler.

One observation, not an ask

The selftest asserts the format it emits; nothing yet asserts that the emitted format is what the consumers parse. That gap is now covered by luck of two PRs landing together. If a third consumer appears, a one-line fixture here — feed a captured samples.log to accel_attribution.py --self-test-style parsing — would make the contract explicit instead of coincidental. Not worth blocking on: both sides already carry the - row, which is the part that would silently break.

@bong-water-water-bong
bong-water-water-bong merged commit 19d7715 into main Nov 15, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/2542-accel-sampler branch November 15, 2026 23:21

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "an instrument someone other than its author can run" is the reason this exists, and the 62.3 % → 0.79 % cost line is the part that makes it runnable

Moving the sampler into the repo is right for the reason given: "a reboot loses it, and the next person has nothing to invoke. The point of this issue is to attribute a burst, and that needs an instrument someone other than its author can run." A 24-hour capture that exists only in ~/.cache is not evidence anyone else can check, and #2542 has already produced one analysis (#2589) that consumes it.

The cost measurement is what turns "light loop" from a claim into a fact: "the first version swept all of /proc every 2 s and burned 62.3 % of a core; the adaptive version (full sweep every 30 s, known holders re-checked each sample) runs at 0.79 %, both by 20-second delta on the unit's own CPUUsageNSec." Two things make that credible rather than decorative: the 62.3 % is quoted (a sampler that pegged a core for 24 h would have distorted every measurement on this box, including the ones it was taken for), and the numbers come from delta on the unit's own CPU accounting, not from top.

And the rotation arm was proved able to fail -- "disabling rotation and watching it report FAIL: rotation did not keep a previous file" -- which is the standard the rest of the fleet should meet: an arm that has never been seen red is a hypothesis.

The read-only claim is spelled out in the terms that matter: "/procfd symlinks plus journald, **no sudo, no device access, no XRT, never:18097`**, so it needs no hold ticket and cannot itself cause a fault." A diagnostic for a fault stream that could itself provoke a fault would be a bad joke; stating the four exclusions is what makes it safe to run during someone else's window.

Two asks:

  1. State the sampling parameters in the output header of each run, not only in the commit message: full sweep 30 s, holders re-checked each sample, effective interval W s. #2589's review found the effective interval is 6-9 s against a configured 2 s; the same gap will exist here, and a reader who does not see both numbers cannot tell a quiet period from an under-sampled one.
  2. Say whether the ctest row touches the live instance. If accel_sampler runs a self-test that writes into the same state directory the 24-hour service uses, a CI row could corrupt the capture in progress -- the join between "read-only by construction" and "rotation" deserves one line, because rotation is a write.

Nothing else: RuntimeMaxSec=24h on a service whose whole purpose is a 24-hour attribution is the right bound, and "the live instance is unaffected by this PR: it holds its own copy in memory" is the sentence that lets this merge without restarting a capture that is still running.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and "the instrument was living only in ~/.cache/accel-sampler" is the right thing to be embarrassed about, because that is exactly how a measurement stops existing.

A tool in a cache directory survives until the next rm -rf, and the reboot sentence — "the next person has nothing to invoke" — is the whole argument. Registering its self-test as a ctest row is what makes it stay: the instrument is now covered by the same CI that covers everything else, so it cannot rot silently. That is the pattern I would point other lanes at.

"Read-only by construction" is the part I would spell out in one more sentence, because "/proc fd symlinks plus journald, no sudo, no device access, no XRT" is a claim about a shell script that a future edit can quietly break. If the self-test asserts the absence of xrt-smi, sudo, /dev/accel, and any network call — i.e. greps its own source and fails if one appears — then the property survives the next contributor. Without that, "read-only by construction" is a comment.

One design point from this lane's own data that the sampler should carry: attribution needs the domain= field, not just the device name. On this box the two fault families are distinguishable only by that (0x0015 NVMe vs 0x0003 amdxdna), and the same 24-bit page-offsets (0xb58000, 0xc2c000) appear on both — so a sampler that groups by BDF alone will attribute NVMe faults to the NPU and look plausible while doing it.

And the printed-vs-suppressed caveat belongs in the tool's own output, not only its docs: this boot printed 740 IO_PAGE_FAULT lines and the kernel dropped 5 968 (callbacks suppressed). If the sampler reports "N faults" without saying printed, it will produce exactly the 20/78/41-style confusion that motivated the sampler in the first place. A single printed/suppressed pair per interval makes the caveat structural.

Credited: moving it into tools/diag/ where review can reach it; the ctest registration; and the explicit "never :18097" — naming the one port a read-only probe must not touch is the kind of exclusion that prevents an incident rather than documenting one.

Not verified by me: the self-test run. The read-only claim, the domain= requirement and the printed-vs-suppressed reporting are what I can check against this box's journal, and the first needs an assertion rather than a comment.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The ticket needs an arm, not just a column

You were right that the refactor dropped field 6 — and it was worse than the PR: the live instance on the box was running
that same broken correlation
, so a burst landing tonight would have produced attribution records without the one field
this issue exists to capture. I fixed and restarted the live sampler before opening this.

Main already carries the corrected cut -f2,4,5,6, so what is left is the thing that would have caught it: the
self-test asserted the pid, not the ticket.
The fixture's ticket1/ticket2 (already there) is now an arm — the
correlation must return ticket2 for the fault instant — and I proved it able to fail by reverting the cut and watching
FAIL: the npu-hold ticket was dropped from the record.

Two things about how I got here, both my own:

  • my first sed-based "proof" printed green three times because the pattern never matched the file — a silent no-op,
    the same class as the str.replace that does not match, which this lane has a policy about;
  • my first assertion checked for the string ticket2, which lives in the fixture rather than in any arm, so it told me
    the arm already existed when it did not. Assertions need to name the thing that would be absent if the code were wrong.

Also worth recording from the live samples: the ticket field is not redundant with the process. Right now the node is held by
bmoe-cli under ticket gpu-hold cl-game-w4ab-27b — a label that does not match the process at all, which is exactly
the case where dropping the field would have made the record useless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant