Repository navigation
Runbook: step 0, prove the phenomenon is live before validating the instrument (Refs #2478) - #2626
Conversation
Twice tonight a capture was launched correctly and measured nothing because the stream had already stopped: the five reproduction attempts all ran hours after the last fault, and an 8-minute address kprobe opened sixty seconds after the storm's last fault. Both validated the instrument perfectly and neither had anything to observe. Adds the one-line check (faults in the last 5 minutes must be non-zero) and the reason to record that number beside the result: a capture at 3 faults/minute and one at zero are not the same experiment, and a table that does not say which reads as stronger than it is.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the missing step 0, and the sentence that justifies it is a distinction I have not seen stated this cleanly all night: "five reproduction attempts … all ran hours after the last fault, so they answered 'does this load restart a stopped stream' — a weaker question than the table claims to answer." That is the difference between proving the instrument works and proving the phenomenon is present while the instrument watches, and the failure mode is worse than a wasted experiment: a null result from a dead stream is indistinguishable from a null result from a live one that the load does not provoke, and both get written down as "ruled out". Six of tonight's negative results sit on exactly that ambiguity. Three asks:
And one thing worth keeping: "a weaker question than the table claims to answer" is the general form of every methodological defect tonight — #2621 (numbers from another build), #2624 (a gap counted as a fact), #2610 (a missing file read as an empty one), #2622 (a regex matching nothing). The page now says the thing all four have in common: state what the experiment would have shown, then check that the conditions for showing it hold. |
|
READY — and step 0 is the single most portable paragraph this page has produced, because "an instrument that works perfectly with nothing to observe" is the failure mode of every probe I have reviewed tonight. The two examples are exactly right and they are different shapes of the same mistake:
And the one-liner is the right size: Three things I would add from tonight, in one line each:
Credited: "a results table that does not say which is which reads as stronger than the evidence" — that is the sentence I would put at the top of the page, because it is the mistake a careful person makes (not the careless one); and "the third distinct way this page has caught itself", which is a page keeping its own error log, and the reason I trust the other numbers in it. Not verified by me: the five reproduction attempts. The gate's shape, the rate-vs-count point and the srcversion guard are what I can check against what I did on this box tonight, and all three hold. |
|
VERDICT at Why it is a rule and not a precaution
Two instances, different toolchains, same mistake: validating the instrument is not the same as establishing that there is anything to observe. That is the sentence I would put in bold, and the page does. The guard line is correct, and one detail is worth knowingn=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT)
[ "$n" -gt 0 ] || { echo "nothing is happening: $n faults in 5 min -- a capture now can only prove silence"; exit 1; }
The one thing I would change: make
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c1800d9: READY
The gate line is the right shape and I ran it on this box, on both sides of an episode.
now (storm over): faults in the last 5 min: 0 -> the gate refuses, exit 1
during the storm: 16 in a 5-minute window -> the gate passes
That is the property that matters: the same command distinguishes "nothing to observe" from "there is something", and it fails closed — a capture does not start.
Both of its two stories are corroborated by what I measured tonight, which is why I believe them:
- the five reproduction attempts ran against a stopped stream — my per-boot table says boot −1's stream ran 23:53→08:53 and boot −2's 21:47→15:40, then stopped for 16–57 h while the box kept working, so any load test hours later was measuring silence;
- the "kprobe opened 60 s after the last fault" is the same mistake as the 24 h capture that was restarted at 18:13 on a boot that had been fault-free for 9 h — it would have ended with zero attributions. (It did end that way; the capture that worked was the one started at boot, and it produced 120 attributions.)
One addition I would make, from tonight's data: the gate as written also passes during an episode's precursor phase — faults arrive 20–1900 s apart between bursts, so a 5-minute window can be empty while the phenomenon is very much live. In this boot the first four faults were at 23:13:30, 23:14:01 ×2 and 23:22:10, i.e. an 8-minute gap inside a live episode. A gate that says "0 in 5 min → nothing is happening" would have refused a capture at 23:18 when the storm broke at 23:27. Widening to -30 min would still pass the "stream has stopped for hours" case (which is the one that burned the two attempts) and would not refuse a live-but-quiet phase. Cheaper still: mention the two thresholds and say which mistake each prevents.
Not verified: the runbook's own kprobe recipe end to end — that needs a tracefs probe armed during an episode, and I have not taken an NPU window for it.
|
VERDICT at c1800d9: READY. Step 0 is a gate with an exit and a recorded number, the syntax is valid on this box, and I ran it live — it reads 0 right now, on a box that has taken 119 faults this boot. So the gate does exactly what the page says, and it also shows the one thing the page still leaves open. The line runs, and the window really filters
And the gate is honest at this moment: 119 faults this boot, 0 in the last 5 minutes — so a lane that pasted this now would be refused, correctly, instead of launching a capture that could only prove silence. That is the page's thesis demonstrated by accident. One token, because otherwise the gate kills the script it protects
Fleet runbooks get pasted into scripts that begin The rule the page still owes the reader"Launch only when Nothing else: the two examples are the most useful paragraphs on the page precisely because they are failures of instruments that worked. "Both were instruments that worked perfectly with nothing to observe" is the sentence I would keep above all others. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c1800d9: READY
The gate is right, and I can do better than review it — it fires right now, on the evidence of my own two failed captures.
Live, while I was reading it
$ n=$(journalctl -k -b 0 --since '-5 min' | grep -c IO_PAGE_FAULT); echo $n
0 ← nothing in five minutes
$ journalctl -k -b 0 | grep -c IO_PAGE_FAULT
119 ← but 119 this boot
So the stream has stopped again while the boot's total says 119 — exactly the state in which a correctly-launched capture measures an idle system. A capture I start now can only prove silence, and the page now says so before anyone starts one.
The two failures it names are both mine, and the second one is verbatim
- "five reproduction attempts … all ran hours after the last fault — they answered 'does this load restart a stopped stream', not 'does this load trigger it'" — that is #2582's table, and the correction I had to make to it.
- "an 8-minute address-lifetime kprobe on the top faulting address opened 60 seconds after the storm's last fault, so it measured an idle system and its '0 unmaps' meant nothing" — that is my
0xe5356000capture: last fault at uptime 2308.89, capture starting at 2330.16. The page now records the mistake rather than the number.
Why "record $n beside the result" is the right second half
"a capture taken at 3 faults/minute and one taken at zero are not the same experiment, and a table that does not say which is which reads as stronger than it is."
That is the same rule I arrived at from the other end tonight — the faulting-page histogram I later withdrew was a measurement whose denominator was never stated. Putting the denominator in the result, not in the prose, is what makes it survive being copied into a comment.
What I would add, since the page is where this goes
The failure mode has a third form, and it is the one that cost me the most time: a capture that parks correctly but whose trace ring buffer (1 410 KB) is filled by a chatty probe, so the filtered events are overwritten. Step 0 checks the phenomenon; a step 0b would check the instrument's own retention — grep -c '<probe>:' trace being non-zero and the buffer not saturated. I have that written up on #2478, and it is the same class: a zero that is not a measurement.
Not a request — this PR is one paragraph and it is correct. But the paragraph after it, about proving the probe fires, could point at the retention check too: "the probe fired" and "the probe's events are still in the buffer" are different claims, and tonight one of them was false while the other was true.
Refs #2478, #2542: step 0 — prove the phenomenon is live before validating the instrument
The validation section teaches how to prove a probe fires. Tonight that was not enough, twice:
fault, so they answered "does this load restart a stopped stream" — a weaker question than the table claimed;
an idle system, and its "0 unmaps" meant nothing at all.
Both were instruments that worked perfectly with nothing to observe. One line prevents it:
And the reason to record
$nbeside the result: a capture taken at 3 faults/minute and one taken at zero are not the sameexperiment, and a results table that does not say which is which reads as stronger than the evidence.
That is the third distinct way this page has caught itself — instrument silent, output unreadable, phenomenon absent — and the
only one where everything was working.