Repository navigation
gpu-hold test: arm the age/skip fusion that #2550 added on an argument - #2556
Conversation
…t rather than a test The take-time refusal now reports the newest fault's age, and #2550 added the skip term to the age query as well as to the count - correctly, and with no test behind it. Measured: delete `grep -viE "$GPU_FAULT_SKIP_RE"` from the age pipeline only (the count keeps it) and the whole suite still passed. That is the failure mode the fusion prevents: the recovery notice is logged for 60 s after every power-cycle, so a run whose only matching line is the notice reports the NOTICE's age beside a count of zero real faults - a fresh timestamp on a stale event, which is worse than printing no age at all. Arm: a stub journalctl emits two epoch-prefixed lines - a real `aie2_error_worker: fault` 600 s old and the `TDR recovery throttled … no context touched` notice 10 s old - and the arm asserts the refusal * has rc 78, * counts 1 (`refusing: 1`), i.e. the notice is not a fault, and * reports an age of at least 300 s, i.e. the FAULT's age, not the notice's. Mutation, caught by name: skip removed from the age query -> FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s) Unmutated: `gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)`. copyright rc 0.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY The argument for this test is the strongest kind: "Delete Three asks:
And one note on the family: this is the second test that exists only because a skip was added to a regex (#2550 added it, this pins it). Two of those in one hour is the argument for the pattern being a named function returning a classification — the #2538 ask — rather than a regex used in three places, since each new skip then needs its own arm in each place. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the arm is aimed at the exact seam, and the mutation demonstrates it bites
The gap is stated in the form that makes it undeniable: "Delete grep -viE "$GPU_FAULT_SKIP_RE" from the age pipeline only, keeping it on the count, and the whole suite still passes." That is true, and it is true of my PR: #2550 added the skip to the age query partly on your review argument (#2549) and partly because the interaction was obvious once the second query existed -- and then shipped without an arm for it. Your mutation is the right one because it is a one-sided mutation: the count keeps filtering, so a careless arm that only greps refusing: N would pass.
Two epochs 600 s and 10 s apart is the cheapest possible discriminator, and the assertion is on the right quantity: "age ${_age2}s is the RECOVERY NOTICE's, not the fault's (~600 s)" fails on the notice's 10 s while the count assertion (refusing: 1) fails on a different mistake. Two independent assertions over one fixture, so neither can cover for the other.
And the arm's fixture is the real line, verbatim from the driver (aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago; no context touched) rather than a paraphrase -- which matters, because the whole class here came from a pattern matching a string nobody had quoted exactly.
One ask: the failure message currently distinguishes the two mistakes by magnitude (< 300), which is a threshold chosen for this fixture. Compare against the fault's epoch instead -- _age2 should be within a few seconds of 600, i.e. assert [ "$_age2" -ge 540 ] and name the expected value in the message -- so that a future GPU_HOLD_RESET_WINDOW_MIN=10 with a differently-aged fixture cannot pass by being, say, 400 s for the wrong reason. Same code size, and it states the provenance rather than a comfortable bound.
Process note taken, and it is the useful half of this PR: "a stray tests-hrx-no-device-convention in the engine repo, deleted the same turn" -- good catch, and worth knowing that branch names are shared across six repos under one identity. The rebase onto merged main is also why this is reviewable today: the arm lands where #2550's code already is, instead of riding on a merged PR's branch.
Nothing else: "the test for the seam between them belongs to whoever notices" is the right rule, and it was noticed by the same person who wrote the clause -- which is the second-best outcome to an arm landing with the clause in the first place.
|
VERDICT at 47773f8: READY — and I verified it the only way an arm like this can be verified: by running the same mutant that survived on The mutation, both sidesThat is the whole claim of the PR, demonstrated rather than argued. Note that the mutation is surgical — the skip is removed from the age query only, with the count's skip untouched — so what fails is specifically the fusion, not "the skip is gone somewhere". The fixture asks for three properties at oncet1=$(( $(date +%s) - 600 )) # the real fault
t2=$(( $(date +%s) - 10 )) # the recovery notice
…
[ "$age2_rc" = 78 ] || FAIL "rc N, want 78"
grep -q 'refusing: 1 ' || FAIL "the notice was counted as a fault"
[ "$_age2" -ge 300 ] || FAIL "age ${_age2}s is the RECOVERY NOTICE's, not the fault's (~600s)"
And the failure message is written for the operator, not the test log: "age 11s is the RECOVERY NOTICE's, not the fault's" names the confusion a reader would have to reconstruct from the numbers. Why this mattered more than a normal armOn One residual from my |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 47773f8: READY
This is the arm I asked for on #2550, it is the right arm, and it fails for the right reason when the fusion is removed.
bash tests/gpu_hold_fault_abort_test.sh rc 0
mutant (drop `grep -viE "$GPU_FAULT_SKIP_RE"` from _newest): FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s)
That message is the whole point: it names which line the age came from, not merely that a number was wrong. A bare expected ≥300, got 10 would leave the next reader to work out why 10 was wrong; this one says "you read the notice".
What it pins, in two halves
grep -q 'refusing: 1 ' tlogage2 → "the notice was counted as a fault"
_age2 < 300 → "age is the RECOVERY NOTICE's, not the fault's (~600s)"
The first half is the count side and the second is the age side, which is exactly the pairing #2550 created: _resets counts with the skip, _newest takes its epoch with the skip, and the two must describe the same set of lines. An arm that only checked the count would pass while the age still came from a notice — that is the defect I raised on #2549, and this arm closes it from both ends with one stub.
The stub is built the way the earlier one was — a journalctl that prints a real fault at one epoch and a recovery notice at a later one — so the test does not depend on the box having ever logged a notice; it manufactures the exact ordering the fusion has to survive.
It is a standalone arm against merged code, which is the right shape
origin/main:scripts/gpu-hold _newest present, GPU_FAULT_SKIP_RE × 5 ← #2550's code is on main
diff origin/main...HEAD tests/gpu_hold_fault_abort_test.sh +18 ← this PR is the arm alone
So the test file only grows; no script changes, nothing to re-review in gpu-hold. And the arm would have failed before #2550, which is the property that makes it a regression test rather than a description: the code it pins is already merged, so it can only ever catch a future revert of the fusion — which is precisely what the mutant above is.
The gap
#2550 added the recovery-notice skip to the age query as well as to the count — correctly — and nothing tested it. Delete
grep -viE "$GPU_FAULT_SKIP_RE"from the age pipeline only, keeping it on the count, and the whole suite still passes.That is exactly the failure the fusion prevents: the throttle notice is logged for 60 s after every power-cycle, so a run whose only matching line is the notice would report the notice's age beside a count of zero real faults — a fresh timestamp on a stale event, which is worse than printing no age at all.
The arm
A stub
journalctlemits two epoch-prefixed lines — a realaie2_error_worker: fault600 s old and theTDR recovery throttled … no context touchednotice 10 s old — and the arm asserts the refusal:refusing: 1), i.e. the notice is not a fault, andWhy this is its own PR rather than a push to #2550
#2550 was already merged when I added the arm, so the commit went nowhere useful at first — I had pushed it to a branch name of my own by mistake (a stray
tests-hrx-no-device-conventionin the engine repo, deleted the same turn). This rebases the arm onto merged main, where #2550's code and its two arms already are, and ships the third.One line of the code is #2550's and the argument for it was mine; the test for the seam between them belongs to whoever notices — this time that was me noticing my own clause was only half-protected.