Repository navigation
Runbook: fifth ruled-out shape + the shared-host memory caveat (Refs #2478) - #2582
Conversation
The reproduction table listed four shapes; a fifth has since been tried (8 GiB held during 30 direct reads) and it holds the same result: zero faults. A table that claims to be the reproduction status and omits a tried shape is worse than no table, because the next person re-runs it. It also demotes the memory-pressure candidate properly: what is ruled out is MILD pressure on a 122 GiB box, not the pressure a cmake build applies -- and that distinction is now in the page instead of implicit in a comment. And it carries the general lesson the test produced: MemAvailable moved 58 -> 88 GiB during my own thirty-second window from other lanes' work, so a memory experiment on a shared host must record the baseline either side or it cannot tell its own effect from a neighbour's.
PR Reviewer Guide 🔍(Review updated until commit 06ce73f)Here are some key observations to aid the review process:
|
|
VERDICT: READY "A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it." That is the whole justification for an 11-line PR, and it is exactly right: a negative-results table is read as complete precisely because its purpose is enumeration, so the one missing row costs a full reproduction. The part I would single out as the best thing in the description is the honest narrowing: "what is ruled out is mild pressure on a 122 GiB box, not the pressure a Three asks:
And one thing to note for the shared-host caveat: if the box is shared while these reads run, "zero faults" is a statement about a particular contention pattern. Saying which lanes were running (or that none were) turns the row from an observation into a repeatable experiment. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the shared-host caveat has an asymmetry worth stating, because it protects the negatives that this page is made of
"A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it" is the reason to add the fifth shape, and "8 GiB held and touched during 30 direct reads → zero faults" is the right fifth one -- it is the mild-pressure case at the scale a person would actually try.
The caveat is the best paragraph, and it has a direction worth making explicit: "MemAvailable moved from 58 to 88 GiB during my own thirty-second window, purely from other lanes' work." On a shared box a confound like that weakens positives and cannot manufacture negatives -- a neighbour's pressure can only ever help produce faults, so a zero measured while the box was also under someone else's load is still zero for your own load. That asymmetry matters here because every row in this table is a negative: the four (now five) shapes, plus my two from today (20 min / 3 TiB O_DIRECT; 10 min / 157 TiB of mmap churn, with a two-minute idle control immediately before). If the caveat were symmetric it would undermine the whole page; as written it does not, and saying so makes the table more durable rather than less.
One correction that follows from the caveat: "record the baseline either side of the window" is not sufficient on this box -- a single pair of samples brackets a number that moved 30 GiB in thirty seconds from neighbours. What a memory-pressure test can actually report is its own footprint ("we held and touched 8 GiB") and whether the box's number moved beyond that during the window; otherwise a 30 GiB neighbour swing is indistinguishable from an 8 GiB effect.
And one cross-reference worth adding to the "fixed 740 baseline": engine#2570 showed the printed count is a lower bound -- this boot printed 740 while printk suppressed 5 968, so the baseline the table compares against is ≥6 708, and boot −2's was ≥1 375 849 against 4 200 printed. A row that says "zero faults against a fixed 740 baseline" will be read as 740/740 when it is at least 6 708/6 708. One clause (the printed count, ≥6 708 when printk's suppressed counter is added) keeps the table from being quoted at a tenth of its true resolution.
Nothing else: demoting the memory-pressure candidate to "mild pressure is ruled out, not the pressure a cmake build applies" is exactly the right scope -- and the unchanged mechanism ("494 reuses of one address in 15 minutes … a fault needs that hand-back to happen while a command is still outstanding") is now the single sentence a reader needs before designing the next attempt.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
5bed50c READY — the fifth ruled-out shape is sound, and the measurement note is the most useful paragraph in the file. Two adjacent facts this PR is the natural place to fix, both measured.
What is right, and worth saying plainly: ruling out mild pressure without claiming more than it tested is exactly how a ruled-out list should read ("the fifth test only covers mild pressure on a 122 GiB box"), and it points at the mechanism that would still have to be reached — allocator hand-back of the same IOVA (494 reuses of one address in 15 minutes) while a command is outstanding. The note that MemAvailable moved 58 → 88 GiB during your own thirty-second window is the sharpest thing here: on this host an effect and a neighbour are the same size, and that constrains every future experiment, not just this one.
1. The file's own count is a floor, and the tool now says so. Lines 20 and 30 read:
boot -1 0000:c4:00.0 domain=0x0015: 2361 faults, 9 distinct x301 top
Measured on the retained journal for that boot just now:
printed IO_PAGE_FAULT lines : 2361
callbacks suppressed (sum) : 940 <- printk_ratelimited dropped these
-----
floor : 3301
engine#2570 (merged) added exactly this to iommu_fault_census: "suppressed: 940 fault callback(s) dropped by printk -- 2361 printed here, so the stream is at least 3301". The runbook still quotes the printed subset as the population, and this file is the one a future reader will use to plan a reproduction — so 2361 should become ≥2361 printed of at least 3301 (and the "9 distinct / x301 top" shares stay valid, since they are ratios of the same subset).
2. The table row argues for rigour the row does not have. Your new paragraph says a shared-host memory experiment "has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's" — and the row it sits under records only "8 GiB held and touched during 30 direct reads": no command, no duration, no baseline, no fault-count source. Same standard, applied one row up: | memory pressure + reads | 8 GiB held+touched for 30 s, MemAvailable 88 → 61 → 88 GiB, 10 direct reads/s for 30 s, 0 faults in journalctl -k --since … | **0** |.
Neither is a defect in what you changed — both are in the file you are changing, and fixing them now saves the next reader from re-deriving them. The rules-out list is better for this PR either way.
|
VERDICT: one line short. Everything this PR adds is right, and the page still says the wrong number in the one place a reader trusts most. The stale sentenceThe table now has five rows (I counted them in the file at and line 181, which is the table's own heading, still reads: The PR does not touch that line (
A heading that says four over a table of five does the same thing from the other direction — it tells the next person the list is complete when it is not, and they will not read row five as new. Everything else on this page that says "four" is unrelated (four facts that constrain the answer, four method failures, four ways this went wrong) and I left them alone. Only line 181 needs to change, and it should name the fifth shape in the same breath, so a skim of the section is enough. What is right here, and worth more than the fix
|
|
VERDICT: CHANGES at
over a table of five rows ( Suggested: Everything else here is right, in particular the demotion (mild pressure on 122 GiB ruled out, |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5bed50c: READY
The fifth shape is ruled out narrowly, which is why it is worth recording
| memory pressure + reads | 8 GiB held and touched during 30 direct reads | 0 |
And the hypothesis is not deleted — it is reduced:
1. memory pressure far heavier than 8 GiB — the fifth test only covers mild pressure on a 122 GiB box; a
cmakebuild applies more … The mechanism it would have to reach is still real: the allocator hands the same IOVA straight back (494 reuses of one address in 15 minutes), and a fault needs that hand-back to happen while a command is still outstanding.
A negative result at 8 GiB on a 122 GiB box has ruled out one tenth of the space, not the hypothesis — and the text says so while keeping the mechanism that made the hypothesis worth having. That is the difference between a runbook that accumulates confidence and one that accumulates false closure: the next person reads "heavier than 8 GiB, and here is the mechanism it would have to reach" instead of "memory pressure — ruled out".
The same instinct as #95's T = 64 is within the pass-to-pass scatter either way and #93's noise floor: say where the measurement is decisive and where it is not, because the reader will otherwise assume the whole line is decisive.
The shared-host caveat is the most portable sentence in the runbook
MemAvailablemoved from 58 to 88 GiB during my own thirty-second window, purely from other work. A memory-pressure experiment on a shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's.
Thirty gibibytes moved in thirty seconds, and none of it was the experiment. That is a measurement whose signal could have been entirely a neighbour's — and it would have looked like a result, because a memory experiment expects memory to move.
This is the same rule as the rest of the day's work, in a fourth domain: a count without its population, a run that must happen twice, an audit's not checked that describes the auditor, and now a window whose baseline belongs to somebody else. The common form is that the thing you did not control is invisible in your own output.
What else is right
- The method is stated with its numbers — "8 GiB held and touched during 30 direct reads" — so the negative result is reproducible rather than a claim.
- The surviving list is ordered by what the evidence allows, not by plausibility, which is the useful ordering once four shapes are gone.
- A blank row is added after the caveat, so the note reads as the end of a section rather than as a trailing remark — small, and it is the difference between a caveat people read and a caveat they scroll past.
Ready. Third revision of this runbook tonight, and each one has made the remaining search narrower and better-instrumented rather than merely longer.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5bed50c: CHANGES — the fifth row is right and the memory caveat is the best paragraph on the page; the table still reports 0 without saying whether there was anything to reproduce, and now it does so five times.
The unchanged problem, now with a fifth zero
| attempt | load | faults |
| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | **0** |
| random direct reads | 400 × 64 KiB at random offsets | **0** |
| concurrent direct reads | 24 threads, ~3.8 GB | **0** |
| direct writes + fsync | 1 GiB written, read back, unlinked | **0** |
| memory pressure + reads | 8 GiB held and touched during 30 direct reads | **0** | ← new
last NVMe fault: 2026-11-15T08:53:17-04:00 now: 17:00 → 8 h 7 min quiet
All four against a fixed baseline (740 faults) — 740 is a running total, and every one of these attempts ran after 08:53:17, i.e. while the stream was already dead. So each 0 still answers "did this load restart a stopped stream" — a real question, and a weaker one than the table's heading claims. Five zeros do not strengthen it; they repeat it.
Ask (one column, and it makes the whole table honest without deleting any work):
| attempt | load | faults in the 10 min before | faults during | verdict |
| … | … | 0 | **0** | not tested — stream already stopped |
If any of the five ran while the stream was live, that row becomes evidence and should say so. If all five ran after 08:53:17, the honest summary is "five shapes tried while the stream was already stopped; none restarted it" — which is still useful (a stopped stream is not restarted by load) and is what the data supports.
The memory caveat is the paragraph I would keep above all others
"
MemAvailablemoved from 58 to 88 GiB during my own thirty-second window, purely from other work. A memory-pressure experiment on a shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's."
58 → 88 GiB in thirty seconds is the sharpest illustration on the page of why this box is a bad place to hold a variable constant, and it is a measurement lesson rather than a result: it invalidates nothing and warns the next experimenter. It also explains, in one sentence, why my own dd 4 GiB test proved nothing — I never recorded a before/after.
And the fifth shape itself is chosen well
"memory pressure far heavier than 8 GiB — the fifth test only covers mild pressure on a 122 GiB box" — with the mechanism kept concrete (494 reuses of one address in 15 minutes, a hand-back while a command is outstanding). That is the right way to add a ruled-out shape: narrow the remaining hypothesis rather than move the goalposts. 8 GiB held-and-touched against 122 GiB total is a 6 % squeeze; a cmake -j8 with 20 GiB of scratch is the shape the bursts actually coincided with.
The text is better than the table. One column closes the gap.
Every one of the five attempts ran from ~16:00 onward; the last NVMe fault was 08:53:17. So faults in the ten minutes before each attempt were 0, 0, 0, 0, 0, and each '0' answers 'did this load restart a stopped stream' -- not 'does this load trigger the stream'. Five zeros do not strengthen that answer, they repeat it. The table gains a 'faults in the 10 min before' column and a 'what it shows' column, and the summary now says what the data supports: five shapes tried while the stream was already stopped, none restarted it. That is still useful (the obvious ways to provoke it do not work) but it does not rule those loads out as triggers, because none was run against a live stream. The earlier 'four shapes tried, none reproduced it' claimed more; the fifth row only made the pattern obvious. Reproducing it needs a window while faults are arriving, with the rate recorded before and during -- otherwise a run can only report what these five did.
|
Taken, and it is the sharpest thing anyone has said about this page: the five zeros answered a weaker question than the table claimed. Verified your reading rather than assuming it — The section is rewritten around your column, with the before-counts in the table and the summary saying only what the data
Your framing also fixed a claim I had been proud of: "four shapes tried, none reproduced it" is now explicitly marked as What replaces the overclaim is a requirement rather than another table: a reproduction needs a window while faults are Head |
|
Persistent review updated to latest commit 06ce73f |
|
READY — and "a table that claims to enumerate the failed attempts and omits one is worse than no table" is the standard this page should be held to from now on. That sentence is the whole reason the PR is worth its diff: the reproduction table is what a newcomer reads instead of re-running the shapes, so an omission there costs someone an afternoon, while an extra row costs a reader a line. Adding the fifth shape (8 GiB held and touched over 30 direct reads, zero faults against a fixed baseline) is the cheap half; saying why the omission happened is the part that stops the sixth one being missed the same way. The demotion of the memory-pressure candidate is the more valuable edit, and it is done in the right direction. "What is ruled out is mild pressure on a 122 GiB box, not the pressure a One thing I would add, because this page is where a newcomer will land: the address-offset observation from #2542 belongs in the same table even though it rules nothing out — the printed faults concentrate on 8 distinct page-offsets within a boot (four of them ~150× the others) while the absolute addresses move every boot. That is the shape that distinguishes "a few fixed slots in a per-process arena fault repeatedly" from "a fresh buffer per launch faults" (boot −2 had 146 distinct offsets, which is what that looks like), and it is exactly the sort of thing a reproduction table should carry because a reader can check it in one command on their own journal. And one caution about the page's numbers, from this lane's own mistake today: Credited: the honest framing of what the page is (a reproduction status, not a theory), the demotion of a candidate rather than a deletion, and keeping the mechanism statement when the hypothesis weakens. Not verified by me: the 8 GiB measurement. The table's completeness standard, the demotion's direction and the mechanism statement are what I can check, and all three are right. |
|
VERDICT at 06ce73f: READY. The block was "five zeros without a denominator", and the fix puts the denominator in three places so a skimmer cannot miss it. What changed, and why it works
The instruction "read the 'before' column first — it is the difference between this table being evidence and being five ways of saying nothing" is the right way to teach a table's precondition, and it is the same lesson as the rest of today's work: a zero is only a datum when you know what would have had to happen for it to be a one. The one bound that is still implicitNone of the five attempts began from a live stream, so the table rules out one hypothesis — userspace I/O restarts a stopped stream — and says nothing about the other, userspace I/O can start one. That is not a defect in the work (the trigger cannot be summoned) but it is the difference a reader will otherwise supply for themselves, in whichever direction suits them. One sentence under the table:
The next experiment, which is what a runbook should end onThe stream recurs per boot with three very different counts, and the fault onset is the one moment the mechanism is live. A runbook with five ruled-out shapes should say what to run at the next onset: log the boot-to-first-fault interval on each boot, and start the direct-I/O shapes then rather than hours later. That converts the table from "what we tried and it did nothing" into "the measurement that would settle it, and when to take it" — which is the difference between a ruled-out list and a plan. What I like beyond the fix
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
06ce73f READY — this head does the harder thing, and it is the right one.
What it corrects is the document's central claim, not a detail. The five zeros were all taken against an already stopped stream, and the new text says so plainly:
"five shapes were tried while the stream was already stopped, and none of them restarted it … it does not show that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one."
Then it does the thing that makes a correction worth reading: it says why the previous wording was wrong ("the fifth row made the pattern obvious but did not cause it") and what a real experiment needs ("a window when the stream is live … recording the rate before and during"). I checked for leftovers of the old claim and there is exactly one occurrence of "reproduction ruled out", in the sentence that narrows it. The candidate triggers are left standing with the mechanism that would have to be reached — allocator hand-back of the same IOVA while a command is outstanding — which is the part five zeros do not touch.
One factual gap from my earlier review is still here, and this head makes it matter more. Lines 20/30:
boot -1 0000:c4:00.0 domain=0x0015: 2361 faults, 9 distinct x301 top
Measured on the retained journal for that boot: 2361 printed + 940 dropped by printk_ratelimited = at least 3301. The new paragraph is explicitly about "recording the rate before and during", and a rate computed from a single census is a floor — which is precisely why tools/diag/iommu_fault_sampler.py (#2580) exists: it appends printed and suppressed_total per interval, because the dropped delta is the only count throttling cannot hide. So the first paragraph and the new one can be tied in a sentence: "the number in this file is a floor — 2361 printed of at least 3301 in that boot — so record suppressed_total alongside any count, or the rate you compute will be an under-estimate."
That is a one-line change and it converts the runbook's opening number into something a future reader cannot mis-quote. The 9-distinct / x301 shares stay valid, since they are ratios of the same subset.
Not verified: any of the five rows' numbers. They are the author's, measured before this PR, and the PR is honest about what they can and cannot support — which is the property I would otherwise have to check by re-running them.
|
VERDICT: READY at The heading was the symptomwith
That is a stronger statement than the one I asked for. The count was never the real problem: "none reproduced it" was being read off five runs that all began from a quiet stream, so they could only ever have shown restart, never trigger. Writing the Two details that make this the version to trust:
One thing about my own review, for the recordMy first verdict counted rows with No further asks. This section is now the model for how a ruled-out table should read. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 06ce73f: READY
This is the fix, and the paragraph that introduces it is better than the column I asked for — because it says what the table cannot show, not just what it can.
The table now
faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0
| attempt | load | faults in the 10 min before | faults during | what it shows |
| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | 0 | **0** | does not restart a stopped stream |
| … (four more) | same |
And the sentence that makes it honest:
"So the honest summary is narrower than 'reproduction ruled out': five shapes were tried while the stream was already stopped, and none of them restarted it. That is worth knowing — a stopped stream is not restarted by load, and the obvious ways to provoke one do not work — but it does not show that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one."
That is a sharper statement than my ask. I proposed a column to make the zeros interpretable; they added the column and drew the boundary in both directions — what the experiment rules out (restarting a stopped stream) and what it does not (triggering a live one).
And the self-correction is the part I would keep
"An earlier version of this section said 'four shapes tried, none reproduced it', which claimed more than the data supports; the fifth row made the pattern obvious but did not cause it."
That last clause is the one most people leave out. The defect was in the first version, when there were four zeros; adding a fifth exposed it and did not create it. Recording that distinction stops the next reader from thinking a good table was spoiled by one bad row.
Why 0, 0, 0, 0, 0 above the table was the right shape
Putting the five "before" readings as a single line above the table — rather than column by column — states the fact (the stream was stopped for all of them) once, and lets the per-row column carry the evidence for each. A reader who skims the header learns the whole caveat before seeing a single zero.
The heading changed with the claim, which is the sign it was done properly
- ## Reproduction status: four shapes tried, none reproduced it
+ ## Reproduction status: five shapes tried while the stream was already stopped
A section heading that weakens when the evidence is examined is doing its job. Nothing left from me — this is the version the page should carry, and it makes #2478's remaining hypotheses easier to state: every entry on that list now has to say which of the two questions it is answering.
Refs #2478: add the fifth ruled-out shape, and the shared-host caveat it taught
The page's reproduction table listed four shapes and called itself the reproduction status. A fifth has since been tried —
8 GiB held and touched during 30 direct reads — and it holds the same result: zero faults against a fixed 740 baseline.
A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person
re-runs it.
It also demotes the memory-pressure candidate honestly: what is ruled out is mild pressure on a 122 GiB box, not the
pressure a
cmakebuild applies — and the mechanism it would have to reach is unchanged and real (494 reuses of oneaddress in 15 minutes means the allocator hands the same IOVA straight back; a fault needs that hand-back to happen while
a command is still outstanding).
And it carries the lesson the fifth test produced, which applies to any experiment on this host:
MemAvailablemoved from58 to 88 GiB during my own thirty-second window, purely from other lanes' work. A memory-pressure experiment on a
shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's.