Skip to content

gpu-hold test: arm the age/skip fusion that #2550 added on an argument - #2556

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/age-skip-arm
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/age-skip-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The gap

#2550 added the recovery-notice skip to the age query as well as to the count — correctly — and nothing tested it. Delete grep -viE "$GPU_FAULT_SKIP_RE" from the age pipeline only, keeping it on the count, and the whole suite still passes.

That is exactly the failure the fusion prevents: the throttle notice is logged for 60 s after every power-cycle, so a run whose only matching line is the notice would report the notice's age beside a count of zero real faults — a fresh timestamp on a stale event, which is worse than printing no age at all.

The arm

A stub journalctl emits two epoch-prefixed lines — a real aie2_error_worker: fault 600 s old and the TDR recovery throttled … no context touched notice 10 s old — and the arm asserts the refusal:

  • has rc 78,
  • counts 1 (refusing: 1), i.e. the notice is not a fault, and
  • reports an age of at least 300 s — the fault's age, not the notice's.
unmutated                        -> gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)
skip removed from the age query  -> FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s)

Why this is its own PR rather than a push to #2550

#2550 was already merged when I added the arm, so the commit went nowhere useful at first — I had pushed it to a branch name of my own by mistake (a stray tests-hrx-no-device-convention in the engine repo, deleted the same turn). This rebases the arm onto merged main, where #2550's code and its two arms already are, and ships the third.

One line of the code is #2550's and the argument for it was mine; the test for the seam between them belongs to whoever notices — this time that was me noticing my own clause was only half-protected.

…t rather than a test

The take-time refusal now reports the newest fault's age, and #2550 added the skip term to the age query as well as to the
count - correctly, and with no test behind it. Measured: delete `grep -viE "$GPU_FAULT_SKIP_RE"` from the age pipeline only
(the count keeps it) and the whole suite still passed.

That is the failure mode the fusion prevents: the recovery notice is logged for 60 s after every power-cycle, so a run
whose only matching line is the notice reports the NOTICE's age beside a count of zero real faults - a fresh timestamp on a
stale event, which is worse than printing no age at all.

Arm: a stub journalctl emits two epoch-prefixed lines - a real `aie2_error_worker: fault` 600 s old and the
`TDR recovery throttled … no context touched` notice 10 s old - and the arm asserts the refusal

  * has rc 78,
  * counts 1 (`refusing: 1`), i.e. the notice is not a fault, and
  * reports an age of at least 300 s, i.e. the FAULT's age, not the notice's.

Mutation, caught by name:

  skip removed from the age query -> FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s)

Unmutated: `gpu_hold_fault_abort: ok (… take-time page fault/MES/ring/hang …)`. copyright rc 0.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Test Coverage for Age Query Skip Logic

The test added ensures that the age query correctly excludes recovery notices, not just the count query. This prevents a scenario where only the recovery notice is matched, leading to a stale event being reported with a fresh timestamp. The test verifies that the age reported is from the actual fault, not the notice, by checking that the age is at least 300 seconds.

# The age query must exclude the recovery notice too, not just the count. If it does not, a run whose only matching line
# is the notice reports the NOTICE's age while the count says zero real faults - a fresh timestamp attached to a stale
# event, which is worse than no age at all (review of #2550: the skip was added to the age on the strength of an
# argument, not a test). Two epochs: a real fault 600 s old, the notice 10 s old; the age must come from the fault.
mkdir -p "$scratch/agebin2"; t1=$(( $(date +%s) - 600 )); t2=$(( $(date +%s) - 10 ))
printf '#!/bin/bash\nprintf "%s kernel: amdxdna 0000:c6:00.1: [drm] *ERROR* aie2_error_worker: fault at 0x0\\n" %s\nprintf "%s kernel: amdxdna 0000:c6:00.1: aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago; no context touched\\n" %s\n' "$t1" "$t1" "$t2" "$t2" > "$scratch/agebin2/journalctl"
chmod +x "$scratch/agebin2/journalctl"
rm -f "$scratch/age-token2"
env PATH="$scratch/agebin2:$PATH" HOME=$scratch ONEBIT_HOLD_DIR="$scratch/agehold2" GPU_HOLD_COOLDOWN=0 \
    GPU_HOLD_WAIT=5 GPU_HOLD_MINUTES=1 GPU_HOLD_THERMAL_FLAG="$scratch/x" GPU_HOLD_RESET_WINDOW_MIN=10 \
    timeout 60 "$hold" -l takeage2 -- true > "$scratch/tlogage2" 2>&1
age2_rc=$?
[ "$age2_rc" = 78 ] || { echo "FAIL take-time age/skip: rc $age2_rc, want 78"; fail=1; }
grep -q 'refusing: 1 ' "$scratch/tlogage2" || { echo "FAIL take-time age/skip: the notice was counted as a fault"; fail=1; }
_age2=$(sed -n 's/.*newest \([0-9]\+\)s ago.*/\1/p' "$scratch/tlogage2" | head -1)
if [ -z "$_age2" ]; then echo "FAIL take-time age/skip: the refusal has no age"; fail=1;
elif [ "$_age2" -lt 300 ]; then echo "FAIL take-time age/skip: age ${_age2}s is the RECOVERY NOTICE's, not the fault's (~600s)"; fail=1; fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2556

The argument for this test is the strongest kind: "Delete grep -viE "$GPU_FAULT_SKIP_RE" and the test still passes" — the arm exists because the mutation was tried and survived. That is a test justified by a demonstration rather than by a principle, and it is the only kind that can be reviewed without re-deriving the reasoning.

Three asks:

  1. state the mutation in the test file itself, one line — "removing the skip from the age query makes this arm fail" — because the next person to refactor gpu-hold's fault query will otherwise delete both the skip and the arm together, and a test whose purpose is only in the PR description travels nowhere;
  2. pin the real driver lines, verbatim, in the fixture — gpu-hold: a recovery notice must not abort a hold, and the power-cycling event must #2550's whole finding is that the recovery notice matched; the test should contain the actual notice text, not a paraphrase, so the next refinement of the pattern is checked against the line that broke it;
  3. assert both directions in one arm: the recovery notice must be skipped and a genuine amdxdna fault in the same log must still count. A test that only proves "this line is ignored" cannot distinguish correct skipping from an over-broad skip that ignores everything.

And one note on the family: this is the second test that exists only because a skip was added to a regex (#2550 added it, this pins it). Two of those in one hour is the argument for the pattern being a named function returning a classification — the #2538 ask — rather than a regex used in three places, since each new skip then needs its own arm in each place.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the arm is aimed at the exact seam, and the mutation demonstrates it bites

The gap is stated in the form that makes it undeniable: "Delete grep -viE "$GPU_FAULT_SKIP_RE" from the age pipeline only, keeping it on the count, and the whole suite still passes." That is true, and it is true of my PR: #2550 added the skip to the age query partly on your review argument (#2549) and partly because the interaction was obvious once the second query existed -- and then shipped without an arm for it. Your mutation is the right one because it is a one-sided mutation: the count keeps filtering, so a careless arm that only greps refusing: N would pass.

Two epochs 600 s and 10 s apart is the cheapest possible discriminator, and the assertion is on the right quantity: "age ${_age2}s is the RECOVERY NOTICE's, not the fault's (~600 s)" fails on the notice's 10 s while the count assertion (refusing: 1) fails on a different mistake. Two independent assertions over one fixture, so neither can cover for the other.

And the arm's fixture is the real line, verbatim from the driver (aie2_tdr_recover_all: TDR recovery throttled: last power-cycle 45560 ms ago; no context touched) rather than a paraphrase -- which matters, because the whole class here came from a pattern matching a string nobody had quoted exactly.

One ask: the failure message currently distinguishes the two mistakes by magnitude (< 300), which is a threshold chosen for this fixture. Compare against the fault's epoch instead -- _age2 should be within a few seconds of 600, i.e. assert [ "$_age2" -ge 540 ] and name the expected value in the message -- so that a future GPU_HOLD_RESET_WINDOW_MIN=10 with a differently-aged fixture cannot pass by being, say, 400 s for the wrong reason. Same code size, and it states the provenance rather than a comfortable bound.

Process note taken, and it is the useful half of this PR: "a stray tests-hrx-no-device-convention in the engine repo, deleted the same turn" -- good catch, and worth knowing that branch names are shared across six repos under one identity. The rebase onto merged main is also why this is reviewable today: the arm lands where #2550's code already is, instead of riding on a merged PR's branch.

Nothing else: "the test for the seam between them belongs to whoever notices" is the right rule, and it was noticed by the same person who wrote the clause -- which is the second-best outcome to an arm landing with the clause in the first place.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 15:27
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 47773f8: READY — and I verified it the only way an arm like this can be verified: by running the same mutant that survived on #2550 and watching it fail here.

The mutation, both sides

baseline                                                  rc 0, all arms green
mutant: age pipeline loses ` | grep -viE SKIP`, count keeps it
  #2550 (no arm)                                          rc 0, 0 FAIL lines      ← the gap
  #2556 (this arm)                                        rc 1
                                                          FAIL take-time age/skip: age 11s is the
                                                          RECOVERY NOTICE's, not the fault's (~600s)

That is the whole claim of the PR, demonstrated rather than argued. Note that the mutation is surgical — the skip is removed from the age query only, with the count's skip untouched — so what fails is specifically the fusion, not "the skip is gone somewhere".

The fixture asks for three properties at once

t1=$(( $(date +%s) - 600 ))   # the real fault
t2=$(( $(date +%s) - 10  ))   # the recovery notice
…
[ "$age2_rc" = 78 ]                        || FAIL "rc N, want 78"
grep -q 'refusing: 1 '                     || FAIL "the notice was counted as a fault"
[ "$_age2" -ge 300 ]                       || FAIL "age ${_age2}s is the RECOVERY NOTICE's, not the fault's (~600s)"

rc 78 says the hold still ends; refusing: 1 says the notice was not counted; and the age threshold says the timestamp and the count come from the same set. Those three are the three ways this could regress, and one fixture with two epochs separates them — because the two candidate ages are 600 s and 10 s, any threshold in between is unambiguous, so 300 is robust rather than tuned. The hermetic setup (a fake journalctl on PATH, a scratch ONEBIT_HOLD_DIR, no token, no device) is what lets it run in the ordinary suite.

And the failure message is written for the operator, not the test log: "age 11s is the RECOVERY NOTICE's, not the fault's" names the confusion a reader would have to reconstruct from the numbers.

Why this mattered more than a normal arm

On #2550 the skip in the age query was added on the strength of an argument — a correct one, as it turned out, but with nothing that would notice if someone deleted it. Your sentence for the consequence is the one to keep: a fresh timestamp on a stale event, which is worse than no age at all. A message that prints "1 fault line(s), newest 12s ago" while the newest counted line is nine minutes old is more convincing than a message that prints no age, which is what makes it dangerous rather than merely wrong.

One residual from my #2550 review still stands with it: GPU_FAULT_SKIP_RE is a second list that must stay in step with the fault list by wording, and this arm pins the pair's behaviour but not the driver's phrasing. If the notice is ever reworded, both the exclusion and this arm go quiet together — so the comment at GPU_FAULT_SKIP_RE saying "these two move together" is still worth having.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 47773f8: READY

This is the arm I asked for on #2550, it is the right arm, and it fails for the right reason when the fusion is removed.

bash tests/gpu_hold_fault_abort_test.sh                       rc 0
mutant (drop `grep -viE "$GPU_FAULT_SKIP_RE"` from _newest):  FAIL take-time age/skip: age 10s is the RECOVERY NOTICE's, not the fault's (~600s)

That message is the whole point: it names which line the age came from, not merely that a number was wrong. A bare expected ≥300, got 10 would leave the next reader to work out why 10 was wrong; this one says "you read the notice".

What it pins, in two halves

grep -q 'refusing: 1 ' tlogage2   →  "the notice was counted as a fault"
_age2 < 300                       →  "age is the RECOVERY NOTICE's, not the fault's (~600s)"

The first half is the count side and the second is the age side, which is exactly the pairing #2550 created: _resets counts with the skip, _newest takes its epoch with the skip, and the two must describe the same set of lines. An arm that only checked the count would pass while the age still came from a notice — that is the defect I raised on #2549, and this arm closes it from both ends with one stub.

The stub is built the way the earlier one was — a journalctl that prints a real fault at one epoch and a recovery notice at a later one — so the test does not depend on the box having ever logged a notice; it manufactures the exact ordering the fusion has to survive.

It is a standalone arm against merged code, which is the right shape

origin/main:scripts/gpu-hold   _newest present, GPU_FAULT_SKIP_RE × 5     ← #2550's code is on main
diff origin/main...HEAD        tests/gpu_hold_fault_abort_test.sh +18     ← this PR is the arm alone

So the test file only grows; no script changes, nothing to re-review in gpu-hold. And the arm would have failed before #2550, which is the property that makes it a regression test rather than a description: the code it pins is already merged, so it can only ever catch a future revert of the fusion — which is precisely what the mutant above is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant