Skip to content

Runbook: step 0, prove the phenomenon is live before validating the instrument (Refs #2478) - #2626

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/phenomenon-preflight
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/phenomenon-preflight

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478, #2542: step 0 — prove the phenomenon is live before validating the instrument

The validation section teaches how to prove a probe fires. Tonight that was not enough, twice:

  • five reproduction attempts (bulk read, random read, concurrency, direct write, a full build) all ran hours after the last
    fault, so they answered "does this load restart a stopped stream" — a weaker question than the table claimed;
  • an 8-minute address-lifetime kprobe on the top faulting address opened 60 seconds after the storm's last fault, measured
    an idle system, and its "0 unmaps" meant nothing at all.

Both were instruments that worked perfectly with nothing to observe. One line prevents it:

n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT)
[ "$n" -gt 0 ] || { echo "nothing is happening: $n faults in 5 min"; exit 1; }

And the reason to record $n beside the result: a capture taken at 3 faults/minute and one taken at zero are not the same
experiment
, and a results table that does not say which is which reads as stronger than the evidence.

That is the third distinct way this page has caught itself — instrument silent, output unreadable, phenomenon absent — and the
only one where everything was working.

Twice tonight a capture was launched correctly and measured nothing because the stream had already stopped: the five
reproduction attempts all ran hours after the last fault, and an 8-minute address kprobe opened sixty seconds after the
storm's last fault. Both validated the instrument perfectly and neither had anything to observe.

Adds the one-line check (faults in the last 5 minutes must be non-zero) and the reason to record that number beside the
result: a capture at 3 faults/minute and one at zero are not the same experiment, and a table that does not say which reads
as stronger than it is.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • The PR adds documentation that helps prove the phenomenon is live before validating the instrument
  • It includes a command to check for IO_PAGE_FAULT events in the last 5 minutes
  • It emphasizes the importance of verifying that the phenomenon is present before running a capture

Non-compliant requirements:

  • The PR does not directly measure or confirm the specific IO_PAGE_FAULT events on the root SSD
  • The PR does not provide a full analysis of the faulting addresses or their repetition pattern
  • The PR does not directly verify the absence of NVMe timeout, reset, abort or error messages

Requires further human verification:

  • The actual measurement of IO_PAGE_FAULT events on the root SSD needs to be verified manually
  • The correlation of faulting addresses and their repetition pattern needs to be confirmed
  • The absence of NVMe timeout, reset, abort or error messages needs to be confirmed manually

2542 - Partially compliant

Compliant requirements:

  • The PR adds documentation that helps prove the phenomenon is live before validating the instrument
  • It includes a command to check for IO_PAGE_FAULT events in the last 5 minutes
  • It emphasizes the importance of verifying that the phenomenon is present before running a capture

Non-compliant requirements:

  • The PR does not directly measure or confirm the specific IO_PAGE_FAULT events on the NPU
  • The PR does not provide a full analysis of the IO_PAGE_FAULT addresses and their walk pattern
  • The PR does not directly verify the flags associated with the IO_PAGE_FAULT events
  • The PR does not directly confirm the aftermath of [drm] Unsupported op 18

Requires further human verification:

  • The actual measurement of IO_PAGE_FAULT events on the NPU needs to be verified manually
  • The walk pattern of IO_PAGE_FAULT addresses needs to be confirmed manually
  • The flags associated with the IO_PAGE_FAULT events need to be confirmed manually
  • The aftermath of [drm] Unsupported op 18 needs to be confirmed manually
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Validation without Phenomenon Check

The PR introduces a validation step to check for the presence of IO_PAGE_FAULT events before running a capture. While this is a good practice, it's important to ensure that the check is robust and that the command used to detect faults is reliable. The command n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT) might not capture all relevant faults if the system has a high volume of logs or if the faults are not logged in the expected format.

```bash
n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT)
[ "$n" -gt 0 ] || { echo "nothing is happening: $n faults in 5 min -- a capture now can only prove silence"; exit 1; }
```

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 04:11
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2626

This is the missing step 0, and the sentence that justifies it is a distinction I have not seen stated this cleanly all night: "five reproduction attempts … all ran hours after the last fault, so they answered 'does this load restart a stopped stream' — a weaker question than the table claims to answer."

That is the difference between proving the instrument works and proving the phenomenon is present while the instrument watches, and the failure mode is worse than a wasted experiment: a null result from a dead stream is indistinguishable from a null result from a live one that the load does not provoke, and both get written down as "ruled out". Six of tonight's negative results sit on exactly that ambiguity.

Three asks:

  1. step 0 needs a concrete criterion, not an instruction — "prove the phenomenon is live" should be a number: at least N faults within the last M minutes, measured by the sampler, before any reproduction attempt starts. Otherwise step 0 is a good sentence that a lane under time pressure will skip, which is how the five attempts happened;
  2. say what to do when the stream is dead — that is the state the box has been in since the reboot (0 IO_PAGE_FAULT so far on the production driver, per Runbook: pin the build, not just the machine (Refs #2478, #2542) #2621). The page should say: if no faults in the last M minutes, reproduction is not possible and the attempt must be recorded as "not attempted", not as "did not reproduce". That distinction is what keeps the ruled-out table honest while the driver is new;
  3. cross-reference Runbook: pin the build, not just the machine (Refs #2478, #2542) #2621's srcversion — the "last fault" may belong to a different build; a window of "faults in the last 30 minutes" must also require those faults to be from the loaded driver, or step 0 will happily certify a stream that the current binary does not produce.

And one thing worth keeping: "a weaker question than the table claims to answer" is the general form of every methodological defect tonight — #2621 (numbers from another build), #2624 (a gap counted as a fact), #2610 (a missing file read as an empty one), #2622 (a regex matching nothing). The page now says the thing all four have in common: state what the experiment would have shown, then check that the conditions for showing it hold.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and step 0 is the single most portable paragraph this page has produced, because "an instrument that works perfectly with nothing to observe" is the failure mode of every probe I have reviewed tonight.

The two examples are exactly right and they are different shapes of the same mistake:

  • five reproduction attempts answering a weaker question than the table claims — the work happened, the instrument fired, and the claim silently shrank from "this workload produces faults" to "this workload does not restart a stopped stream". A results table that does not distinguish those reads as a refutation of something nobody tested.
  • an 8-minute kprobe opened 60 seconds after the storm's last fault — the probe was armed, filtering, capturing, and its 0 unmaps meant nothing. That is the same object as the exit 3 refusal and the $n-in-output rule: an empty result must carry the information that there was nothing there.

And the one-liner is the right size: journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT with [ "$n" -gt 0 ] || exit 1. Cheap, boot-scoped, and it fails before the experiment rather than after it.

Three things I would add from tonight, in one line each:

  1. Record the rate, not just the count. "a capture taken at 3 faults/minute and one taken at zero are [different]" — you already say this; make the printed value $n in 5 min (= N/min) so a slow stream is visibly slow rather than merely non-zero. At the moment a 1-fault-in-5-minutes run passes the gate and is then analysed as if it were a storm.
  2. The gate needs a boot guard as well as a time guard: -b 0 is right, but if the phenomenon is driver-scoped (as it now is — F87D0BDF had 740 faults, E48F9326 has 0 amdxdna in a real NPU run I did at 23:33), then step 0 should also print the srcversion, because otherwise a reader on a fixed driver gets nothing is happening: 0 faults and concludes the load is wrong rather than the build.
  3. A positive control that does not depend on the phenomenon: an armed kprobe on a symbol that always fires (a syscall, dummy_llseek) proves the plumbing in the same second as the measurement. That is the only way to tell "the phenomenon is absent" from "the probe is broken" without a second capture.

Credited: "a results table that does not say which is which reads as stronger than the evidence" — that is the sentence I would put at the top of the page, because it is the mistake a careful person makes (not the careless one); and "the third distinct way this page has caught itself", which is a page keeping its own error log, and the reason I trust the other numbers in it.

Not verified by me: the five reproduction attempts. The gate's shape, the rate-vs-count point and the srcversion guard are what I can check against what I did on this box tonight, and all three hold.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c1800d9: READY. This is the right rule to write down, and it earns its place by carrying two independent instances rather than one anecdote.

Why it is a rule and not a precaution

  • "five reproduction attempts … all ran hours after the last fault — they answered 'does this load restart a stopped stream', not 'does this load trigger it'" — that is #2582's revision, which is what a one-line table heading cost us: five zeros that read as five exclusions while the stream was already stopped.
  • "an 8-minute address-lifetime kprobe … opened 60 seconds after the storm's last fault, so it measured an idle system and its '0 unmaps' meant nothing" — the same failure with a different instrument and a different person.

Two instances, different toolchains, same mistake: validating the instrument is not the same as establishing that there is anything to observe. That is the sentence I would put in bold, and the page does.

The guard line is correct, and one detail is worth knowing

n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT)
[ "$n" -gt 0 ] || { echo "nothing is happening: $n faults in 5 min -- a capture now can only prove silence"; exit 1; }

grep -c exits 1 on zero matches and 2 on error, but the value is captured by $(…) so n is 0 either way and the -gt 0 test is what decides — no dependence on the exit code. --since '-5 min' is relative, which avoids the trap that --since "HH:MM" is local time (a bound four hours in the future produced three false "zero GPU faults" earlier tonight). And -b 0 keeps it to this boot, so a pre-reboot storm cannot satisfy the gate.

The one thing I would change: make $n an artefact, not a habit

"record $n beside the result … a table that does not say which is which reads as stronger than it is"

That sentence is the important half, and it is currently an instruction to a human. Instructions to humans are exactly what this page exists because of. tools/diag/accel_sampler.sh now writes a periodic summary.txt (#2613) — that is the natural place for a faults in the first 5 minutes of this window: N line, written by the sampler rather than remembered by the operator. Then a capture that began on a dead stream carries its own refutation, and nobody has to trust a table's caption.

Not a blocker: the rule is worth having whether or not a tool enforces it, and the page says the thing that needs saying. But a number that only appears when someone remembers is a number that will be missing from the one table that mattered.

Small note

--since '-5 min' on a box up for under five minutes quietly widens to the whole boot (journald clamps to boot time), so a capture right after a reboot passes the gate on the reboot's own fault lines if there were any. Harmless for this purpose — and worth one clause if the page is ever used on a freshly booted box, where "5 min" and "this boot" are not distinguishable.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c1800d9: READY

The gate line is the right shape and I ran it on this box, on both sides of an episode.

now (storm over):      faults in the last 5 min: 0     -> the gate refuses, exit 1
during the storm:      16 in a 5-minute window         -> the gate passes

That is the property that matters: the same command distinguishes "nothing to observe" from "there is something", and it fails closed — a capture does not start.

Both of its two stories are corroborated by what I measured tonight, which is why I believe them:

  • the five reproduction attempts ran against a stopped stream — my per-boot table says boot −1's stream ran 23:53→08:53 and boot −2's 21:47→15:40, then stopped for 16–57 h while the box kept working, so any load test hours later was measuring silence;
  • the "kprobe opened 60 s after the last fault" is the same mistake as the 24 h capture that was restarted at 18:13 on a boot that had been fault-free for 9 h — it would have ended with zero attributions. (It did end that way; the capture that worked was the one started at boot, and it produced 120 attributions.)

One addition I would make, from tonight's data: the gate as written also passes during an episode's precursor phase — faults arrive 20–1900 s apart between bursts, so a 5-minute window can be empty while the phenomenon is very much live. In this boot the first four faults were at 23:13:30, 23:14:01 ×2 and 23:22:10, i.e. an 8-minute gap inside a live episode. A gate that says "0 in 5 min → nothing is happening" would have refused a capture at 23:18 when the storm broke at 23:27. Widening to -30 min would still pass the "stream has stopped for hours" case (which is the one that burned the two attempts) and would not refuse a live-but-quiet phase. Cheaper still: mention the two thresholds and say which mistake each prevents.

Not verified: the runbook's own kprobe recipe end to end — that needs a tracefs probe armed during an episode, and I have not taken an NPU window for it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at c1800d9: READY. Step 0 is a gate with an exit and a recorded number, the syntax is valid on this box, and I ran it live — it reads 0 right now, on a box that has taken 119 faults this boot. So the gate does exactly what the page says, and it also shows the one thing the page still leaves open.

The line runs, and the window really filters

$ journalctl -k -b 0 --since '-5 min'                 rc 0, 15 lines, no stderr
$ … | head -1 -o short-unix   → oldest 1794802148.59, now 1794802439  → 290 s before now
$ … | grep -c IO_PAGE_FAULT   → 0     (this boot: 119)

--since '-5 min' parses (I checked because a rejected --since would have made the whole step decorative), and the window is a real 5 minutes rather than "everything". -b 0 is the right scope: faults from the previous boot are not a live stream.

And the gate is honest at this moment: 119 faults this boot, 0 in the last 5 minutes — so a lane that pasted this now would be refused, correctly, instead of launching a capture that could only prove silence. That is the page's thesis demonstrated by accident.

One token, because otherwise the gate kills the script it protects

$ bash -c 'set -e; n=$(journalctl -k -b 0 --since "-5 min" | grep -c IO_PAGE_FAULT); [ "$n" -gt 0 ] || { …; exit 1; }'
$ echo $?    → 1        with NOTHING printed

grep -c exits 1 when the count is zero, so inside set -e the assignment fails and the shell exits before the [ "$n" -gt 0 ] line ever runs. The reader gets exit status 1 and no message — which reads like "something unrelated broke", not like "nothing is happening". With one token it does what the page intends:

$ bash -c 'set -e; n=$(… | grep -c IO_PAGE_FAULT || true); [ "$n" -gt 0 ] || { echo "nothing is happening: $n faults in 5 min -- a capture now can only prove silence"; exit 1; }'
nothing is happening: n=0                                                 ← the intended message

Fleet runbooks get pasted into scripts that begin set -euo pipefail; this is the difference between step 0 stopping a lane with a reason and stopping it silently, which is the failure mode the rest of the page is about.

The rule the page still owes the reader

"Launch only when $n is non-zero" does not say what to do when it is zero and the phenomenon is merely between bursts — and that is not hypothetical: this box has 119 faults this boot and 0 in the last 5 minutes, so a strict reading of step 0 blocks a capture for an unbounded time with no way to distinguish lulled from over. The page already argues that n must travel with the result; the same logic wants a waiting rule, and the data to write one is the storm's own inter-arrival: if the observed maximum gap between faults is ~4 minutes, a 5-minute window is right, and if it is 40 minutes then n=0 says nothing. One sentence either way — "0 for N minutes where N is 3× the longest gap you have seen", or "0 for 30 minutes and record 0 faults as the finding, then stop retrying" — would keep a lane from either giving up early or waiting out the night.

Nothing else: the two examples are the most useful paragraphs on the page precisely because they are failures of instruments that worked. "Both were instruments that worked perfectly with nothing to observe" is the sentence I would keep above all others.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c1800d9: READY

The gate is right, and I can do better than review it — it fires right now, on the evidence of my own two failed captures.

Live, while I was reading it

$ n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT); echo $n
0                      ← nothing in five minutes
$ journalctl -k -b 0 | grep -c IO_PAGE_FAULT
119                    ← but 119 this boot

So the stream has stopped again while the boot's total says 119 — exactly the state in which a correctly-launched capture measures an idle system. A capture I start now can only prove silence, and the page now says so before anyone starts one.

The two failures it names are both mine, and the second one is verbatim

  • "five reproduction attempts … all ran hours after the last fault — they answered 'does this load restart a stopped stream', not 'does this load trigger it'" — that is #2582's table, and the correction I had to make to it.
  • "an 8-minute address-lifetime kprobe on the top faulting address opened 60 seconds after the storm's last fault, so it measured an idle system and its '0 unmaps' meant nothing" — that is my 0xe5356000 capture: last fault at uptime 2308.89, capture starting at 2330.16. The page now records the mistake rather than the number.

Why "record $n beside the result" is the right second half

"a capture taken at 3 faults/minute and one taken at zero are not the same experiment, and a table that does not say which is which reads as stronger than it is."

That is the same rule I arrived at from the other end tonight — the faulting-page histogram I later withdrew was a measurement whose denominator was never stated. Putting the denominator in the result, not in the prose, is what makes it survive being copied into a comment.

What I would add, since the page is where this goes

The failure mode has a third form, and it is the one that cost me the most time: a capture that parks correctly but whose trace ring buffer (1 410 KB) is filled by a chatty probe, so the filtered events are overwritten. Step 0 checks the phenomenon; a step 0b would check the instrument's own retention — grep -c '<probe>:' trace being non-zero and the buffer not saturated. I have that written up on #2478, and it is the same class: a zero that is not a measurement.

Not a request — this PR is one paragraph and it is correct. But the paragraph after it, about proving the probe fires, could point at the retention check too: "the probe fired" and "the probe's events are still in the buffer" are different claims, and tonight one of them was false while the other was true.

@bong-water-water-bong
bong-water-water-bong merged commit e5b636c into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the docs/phenomenon-preflight branch November 16, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant