Skip to content

Accel sampler: an arm for the ticket field, so it cannot be dropped again (Refs #2591) - #2596

Merged
bong-water-water-bong merged 1 commit into
mainfrom
diag/2591-ticket-armed
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
diag/2591-ticket-armed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2591, #2542: the ticket needs an arm, not just a column

The review of engine#2591 caught the sampler dropping field 6 — the npu-hold ticket — from every attribution record
(cut -f2,4,5 instead of ,6). That was not only a PR defect: the live instance on the box was running the same broken
correlation
, so a burst landing that night would have produced records without the one field this issue exists to
capture. I fixed and restarted the live sampler before opening this.

Main already carries the corrected column, so what this adds is the thing that would have caught it: an arm. The
self-test's fixture already carried ticket1/ticket2, so the correlation now must return ticket2 for the fault
instant, and I proved the arm able to fail by reverting the cut and watching
FAIL: the npu-hold ticket was dropped from the record.

Two notes on how I got here, both mine:

  • a first sed-based "proof" printed green three times because the pattern never matched the file — a silent no-op, the
    same class as the str.replace that does not match, which this lane has a policy about;
  • a first assertion checked for the string ticket2, which lives in the fixture rather than in any arm, so it told me the
    arm already existed when it did not. An assertion has to name something absent when the code is wrong.

Why the field is not redundant with the process, from the live samples: the accel node is currently held by bmoe-cli under
ticket gpu-hold cl-game-w4ab-27b — a label that does not match the process at all.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 23:51
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2596

"The ticket needs an arm, not just a column" is the right correction to my #2591 review, and it is sharper than the ask I made. I asked for the self-test not to need /dev/accel; the better framing is that a ticket column with no arm is a column nobody validates — the sampler would record WHO held the device while nothing checked that it records it correctly, or that it records - when nobody does.

Two asks:

  1. the arm should cover the ambiguous case explicitly — diag: attribute each IOMMU fault to the holder at its own timestamp, not to a window around it #2595 measured that a ±5 s window is AMBIGUOUS 33.0 % of the time, so the arm that matters is two holders inside one window: assert the sampler records both and that the analysis reports ambiguity rather than a guess. An arm that only tests "one holder is recorded" passes on the implementation that cannot tell two apart;
  2. say what the arm needs from the box — if it fabricates a samples.log (no device), it runs everywhere; if it needs accel0, it skips everywhere. Given pr-agent: wait (bounded) for a coder-llm gpu-hold window; skip, never pass, when none opens #2509/hw-gate: --phases placement ran nothing and reported PASS (requested phase with no row now fails) #2523/hw-gate: a held GPU token is a SKIP row, not a FAIL row (engine#2501 rule, third site) #2557, the fabricated-log version is the one worth having, with the device path covered by the 24-hour capture's provenance instead.

…rries the bug)

Main's copy still drops field 6 from the correlation (cut -f2,4,5), so every attribution record would lose the npu-hold
ticket -- the one field engine#2542 exists to capture -- and no self-test arm would notice, because the fixture's ticket
field was never asserted.

Two changes: keep field 6, and assert ticket2 for the fault instant. Proved able to fail by reverting the cut and watching
'FAIL: the npu-hold ticket was dropped from the record'.

Rebuilt on current main rather than rebasing: main gained this file after my branch was cut, so the rebase saw two
additions of the same path (AA) and would have needed a manual merge of identical intent.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cf6d583: READY

Content is ready. The PR is CONFLICTING, so it needs a rebase before it can merge — that is the only thing between this and landing.

Two real bugs, and the first one is a class this fleet keeps producing

npu is a prefix of npuserve, so grep -qE "^npu" read an npuserve row as proof that npu had run.

A prefix match accepting one component's row as proof that a different component ran. That is the same defect as npu-kernels#74's *_b8 versus *_b8* — a pattern that is right for the common case and silently wrong for the case it was written to detect — and here it means an npu check could be satisfied by a line from npuserve that says nothing about npu. Worth fixing even if nothing downstream had broken yet.

The second bug destroyed evidence rather than producing a wrong answer:

basename "$OUT" was the constant hw-gate-<model>.log and every failing run on every PR overwrote the last one's evidence.

A log name that is unique per model and not per failure — so the artefact you would go back and read is the most recent failure's, whichever PR it belonged to. That is worse than no log, because a log with the wrong content is read as if it were the right one.

The ticket arm, and the two negative assertions around it

  • ticket() returns the npu-hold ticket that was live, "if one is recorded" — so an absent ticket is representable rather than assumed, which is the same anti-guess shape #2595 and #2592 pin.
  • "needs no npu-hold ticket — but it also must never cause a fault, which is why every read is /proc or the journal." A sampler on this box that polls the accel node has to be provably read-only, and saying so is what makes it safe to run during an incident.
  • "A recording fake gpu-hold is first on PATH, so the internal mode is also proven to take no GPU ticket." That is a negative assertion made by instrumentation rather than by inspection — the arm would fail if the sampler started taking a ticket, and it cannot be satisfied by reading the code. It is the strongest form of the "must not" clause, and it is the one I would have asked for.

The #99 check, done rather than assumed

The PR touches only tools/diag/accel_sampler.sh — no CMakeLists.txt — so my first thought was that the new arms had no invocation, which is exactly what gpu-kernels#99 was about an hour ago.

Checked against main before writing it down: accel_sampler.sh is already referenced in CMakeLists.txt, so its self-test is registered and the new arms will run. No gap — and the check took one git grep, which is the same two-second audit I keep failing to run first. This time I ran it before the paragraph rather than after.

One note on the arms themselves

These arms run the CI shape (same model, same run, same second), and check that a failed copy prints [its] path.

"Same model, same run, same second" is the specific thing that makes a uniqueness test meaningful — three failures that differ in any of those would pass a weaker arm while the real collision stayed invisible. Reproducing the exact shape that collides, rather than a convenient approximation of it, is what makes this an arm rather than a smoke test.

Rebase and this is ready.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebuilt on current main — the branch was dirty because main gained tools/diag/accel_sampler.sh (from #2591's merge) after my branch was cut, so the rebase saw two additions of the same path (AA) and would have needed a manual merge of identical intent.

Main's copy still carries the bug this PR fixes — verified at the remote, not in my working tree:

$ git show origin/main:tools/diag/accel_sampler.sh | grep -c 'cut -f2,4,5,6'   -> 0
$ git show origin/main:tools/diag/accel_sampler.sh | grep -c 'ticket was dropped' -> 0

So the correlation on main drops field 6 and no arm would notice. This branch is now exactly two changes on top of main: keep the field, and assert ticket2 for the fault instant (proved able to fail). Diff against main is tools/diag/accel_sampler.sh | 8 ++++++--, head cf6d583ef, verified with ls-remote.

Worth repeating the operational part: the live sampler was running the broken cut and has been fixed and restarted, so tonight's window keeps the ticket field.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2542 - Partially compliant

Compliant requirements:

  • The PR fixes the issue by including the ticket field in the cut command within the correlate function.
  • The PR adds a test case in selftest to verify that the ticket field is included in the output, ensuring that the field is not dropped again.
  • The PR updates the self-test message to reflect that the ticket field is now included.

Non-compliant requirements:

  • The PR does not include any documentation updates to reflect the behavior change.

Requires further human verification:

  • The actual reproduction of the IO_PAGE_FAULT events and the correlation with the npu-hold ticket needs to be verified in the live environment.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing ticket field in correlation

The correlate function was previously excluding the ticket field (field 6) from the output, which could have led to missing critical information during fault correlation. This has been fixed by including field 6 in the cut command.

awk -F'\t' -v lo="$1" -v hi="$2" '$1 >= lo && $1 <= hi' "$SAMP" 2>/dev/null | cut -f2,4,5,6 | sort -u | tr '\n' ';'

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cf6d583: READY

The fix is the right one, and the arm is shown to fail. Checked against git merge-tree origin/main cf6d583:

  • Field 6 is the ticket in both record forms that sample_once writes: the holder row (printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$pid" "$n" …) and the - row (printf '%s\t-\t-\t-\t-\t%s\n' "$ts" "$(ticket)"). So cut -f2,4,5,6 is pid, cmd, scope, ticket.
  • bash tools/diag/accel_sampler.sh --self-test gives accel-sampler self-test: ok (window, fields+ticket, empty window, no-holder sample, rotation), rc 0.
  • With main's cut -f2,4,5 restored, it gives FAIL: the npu-hold ticket was dropped from the record, rc 1. So the new arm catches the regression it was written for, and main does carry the bug, as the commit title says.
  • The ctest row accel_sampler (CMakeLists.txt:659) runs selftest, so CI exercises the arm.

Nit: correlate()'s comment still says it returns "pid:cmd:scope". It now returns four tab-separated fields ending in the ticket.

@bong-water-water-bong
bong-water-water-bong merged commit 5223be2 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the diag/2591-ticket-armed branch November 16, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant