Repository navigation
Accel sampler: an arm for the ticket field, so it cannot be dropped again (Refs #2591) - #2596
Conversation
|
VERDICT: READY "The ticket needs an arm, not just a column" is the right correction to my #2591 review, and it is sharper than the ask I made. I asked for the self-test not to need Two asks:
|
…rries the bug) Main's copy still drops field 6 from the correlation (cut -f2,4,5), so every attribution record would lose the npu-hold ticket -- the one field engine#2542 exists to capture -- and no self-test arm would notice, because the fixture's ticket field was never asserted. Two changes: keep field 6, and assert ticket2 for the fault instant. Proved able to fail by reverting the cut and watching 'FAIL: the npu-hold ticket was dropped from the record'. Rebuilt on current main rather than rebasing: main gained this file after my branch was cut, so the rebase saw two additions of the same path (AA) and would have needed a manual merge of identical intent.
68a92b7 to
cf6d583
Compare
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cf6d583: READY
Content is ready. The PR is CONFLICTING, so it needs a rebase before it can merge — that is the only thing between this and landing.
Two real bugs, and the first one is a class this fleet keeps producing
npuis a prefix ofnpuserve, sogrep -qE "^npu"read annpuserverow as proof thatnpuhad run.
A prefix match accepting one component's row as proof that a different component ran. That is the same defect as npu-kernels#74's *_b8 versus *_b8* — a pattern that is right for the common case and silently wrong for the case it was written to detect — and here it means an npu check could be satisfied by a line from npuserve that says nothing about npu. Worth fixing even if nothing downstream had broken yet.
The second bug destroyed evidence rather than producing a wrong answer:
basename "$OUT"was the constanthw-gate-<model>.logand every failing run on every PR overwrote the last one's evidence.
A log name that is unique per model and not per failure — so the artefact you would go back and read is the most recent failure's, whichever PR it belonged to. That is worse than no log, because a log with the wrong content is read as if it were the right one.
The ticket arm, and the two negative assertions around it
ticket()returns the npu-hold ticket that was live, "if one is recorded" — so an absent ticket is representable rather than assumed, which is the same anti-guess shape#2595and#2592pin.- "needs no npu-hold ticket — but it also must never cause a fault, which is why every read is
/procor the journal." A sampler on this box that polls the accel node has to be provably read-only, and saying so is what makes it safe to run during an incident. - "A recording fake
gpu-holdis first onPATH, so the internal mode is also proven to take no GPU ticket." That is a negative assertion made by instrumentation rather than by inspection — the arm would fail if the sampler started taking a ticket, and it cannot be satisfied by reading the code. It is the strongest form of the "must not" clause, and it is the one I would have asked for.
The #99 check, done rather than assumed
The PR touches only tools/diag/accel_sampler.sh — no CMakeLists.txt — so my first thought was that the new arms had no invocation, which is exactly what gpu-kernels#99 was about an hour ago.
Checked against main before writing it down: accel_sampler.sh is already referenced in CMakeLists.txt, so its self-test is registered and the new arms will run. No gap — and the check took one git grep, which is the same two-second audit I keep failing to run first. This time I ran it before the paragraph rather than after.
One note on the arms themselves
These arms run the CI shape (same model, same run, same second), and check that a failed copy prints [its] path.
"Same model, same run, same second" is the specific thing that makes a uniqueness test meaningful — three failures that differ in any of those would pass a weaker arm while the real collision stayed invisible. Reproducing the exact shape that collides, rather than a convenient approximation of it, is what makes this an arm rather than a smoke test.
Rebase and this is ready.
|
Rebuilt on current main — the branch was Main's copy still carries the bug this PR fixes — verified at the remote, not in my working tree: So the correlation on main drops field 6 and no arm would notice. This branch is now exactly two changes on top of main: keep the field, and assert Worth repeating the operational part: the live sampler was running the broken cut and has been fixed and restarted, so tonight's window keeps the ticket field. |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cf6d583: READY
The fix is the right one, and the arm is shown to fail. Checked against git merge-tree origin/main cf6d583:
- Field 6 is the ticket in both record forms that
sample_oncewrites: the holder row (printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$pid" "$n" …) and the-row (printf '%s\t-\t-\t-\t-\t%s\n' "$ts" "$(ticket)"). Socut -f2,4,5,6is pid, cmd, scope, ticket. bash tools/diag/accel_sampler.sh --self-testgivesaccel-sampler self-test: ok (window, fields+ticket, empty window, no-holder sample, rotation), rc 0.- With main's
cut -f2,4,5restored, it givesFAIL: the npu-hold ticket was dropped from the record, rc 1. So the new arm catches the regression it was written for, and main does carry the bug, as the commit title says. - The ctest row
accel_sampler(CMakeLists.txt:659) runsselftest, so CI exercises the arm.
Nit: correlate()'s comment still says it returns "pid:cmd:scope". It now returns four tab-separated fields ending in the ticket.
Refs #2591, #2542: the ticket needs an arm, not just a column
The review of engine#2591 caught the sampler dropping field 6 — the
npu-holdticket — from every attribution record(
cut -f2,4,5instead of,6). That was not only a PR defect: the live instance on the box was running the same brokencorrelation, so a burst landing that night would have produced records without the one field this issue exists to
capture. I fixed and restarted the live sampler before opening this.
Main already carries the corrected column, so what this adds is the thing that would have caught it: an arm. The
self-test's fixture already carried
ticket1/ticket2, so the correlation now must returnticket2for the faultinstant, and I proved the arm able to fail by reverting the cut and watching
FAIL: the npu-hold ticket was dropped from the record.Two notes on how I got here, both mine:
sed-based "proof" printed green three times because the pattern never matched the file — a silent no-op, thesame class as the
str.replacethat does not match, which this lane has a policy about;ticket2, which lives in the fixture rather than in any arm, so it told me thearm already existed when it did not. An assertion has to name something absent when the code is wrong.
Why the field is not redundant with the process, from the live samples: the accel node is currently held by
bmoe-cliunderticket
gpu-hold cl-game-w4ab-27b— a label that does not match the process at all.