Skip to content

Runbook: fifth ruled-out shape + the shared-host memory caveat (Refs #2478) - #2582

Merged
bong-water-water-bong merged 2 commits into
mainfrom
docs/2478-fifth-negative
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
docs/2478-fifth-negative

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478: add the fifth ruled-out shape, and the shared-host caveat it taught

The page's reproduction table listed four shapes and called itself the reproduction status. A fifth has since been tried —
8 GiB held and touched during 30 direct reads — and it holds the same result: zero faults against a fixed 740 baseline.
A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person
re-runs it.

It also demotes the memory-pressure candidate honestly: what is ruled out is mild pressure on a 122 GiB box, not the
pressure a cmake build applies — and the mechanism it would have to reach is unchanged and real (494 reuses of one
address in 15 minutes means the allocator hands the same IOVA straight back; a fault needs that hand-back to happen while
a command is still outstanding).

And it carries the lesson the fifth test produced, which applies to any experiment on this host: MemAvailable moved from
58 to 88 GiB during my own thirty-second window, purely from other lanes' work. A memory-pressure experiment on a
shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's.

The reproduction table listed four shapes; a fifth has since been tried (8 GiB held during 30 direct reads) and it holds
the same result: zero faults. A table that claims to be the reproduction status and omits a tried shape is worse than no
table, because the next person re-runs it.

It also demotes the memory-pressure candidate properly: what is ruled out is MILD pressure on a 122 GiB box, not the
pressure a cmake build applies -- and that distinction is now in the page instead of implicit in a comment.

And it carries the general lesson the test produced: MemAvailable moved 58 -> 88 GiB during my own thirty-second window
from other lanes' work, so a memory experiment on a shared host must record the baseline either side or it cannot tell its
own effect from a neighbour's.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 06ce73f)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Document the fifth ruled-out shape in the reproduction status table
  • Add the shared-host memory caveat observation from the fifth test
  • Update the documentation to reflect that the stream was already stopped during the tests
  • Clarify that the tests do not prove the stream cannot be triggered when it is running

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Documentation completeness

The documentation update correctly adds the fifth ruled-out shape and the shared-host memory caveat. The changes are consistent with the ticket's goal of providing a complete record of all attempted reproduction methods and observations.

## Reproduction status: five shapes tried while the stream was already stopped

Recorded before the theory, because the first thing anyone will try is to make the stream happen again. **But read the
"before" column first** — it is the difference between this table being evidence and being five ways of saying nothing.

```
last NVMe fault this boot: 08:53:17        the five attempts ran from ~16:00 onward
faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0
```

| attempt | load | faults in the 10 min before | faults during | what it shows |
|---|---|---|---|---|
| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | 0 | **0** | does not restart a stopped stream |
| random direct reads | 400 × 64 KiB at random offsets | 0 | **0** | same |
| concurrent direct reads | 24 threads, ~3.8 GB | 0 | **0** | same |
| direct writes + fsync | 1 GiB written, read back, unlinked | 0 | **0** | same |
| memory pressure + reads | 8 GiB held/touched during 30 direct reads | 0 | **0** | same |

**So the honest summary is narrower than "reproduction ruled out": five shapes were tried while the stream was already
stopped, and none of them restarted it.** That is worth knowing — a stopped stream is not restarted by load, and the
obvious ways to provoke one do not work — but it does **not** show that those loads cannot *trigger* the stream when it is
running, because none of them was ever run against a live one. An earlier version of this section said "four shapes tried,
none reproduced it", which claimed more than the data supports; the fifth row made the pattern obvious but did not cause it.

Reproducing it therefore needs a window when the stream is live, which means either waiting for the box to produce one
(it happened during other lanes' builds) or running the same shapes *while faults are arriving* and recording the rate
before and during. If a run starts from a quiet stream, it can only report what these five did: no restart.

What the evidence still leaves as candidate *triggers*, unchanged by the five zeros:

1. **memory pressure far heavier than 8 GiB** — a `cmake` build on a 122 GiB box applies more, and the bursts coincided
   with builds. The mechanism it would reach is real: the allocator hands the same IOVA straight back (494 reuses of one
   address in 15 minutes), and a fault needs that hand-back to happen while a command is still outstanding;
2. **uptime** — allocator history may be the precondition, which no short test creates;
3. **a caller outside the block path** — a minority of traced unmaps arrived via `nvme_unmap_data -> dma_iova_unlink ->
   dma_iova_destroy`, i.e. teardown rather than completion.

One measurement note the fifth test produced, which applies to any experiment on this host: `MemAvailable` moved from 58 to
88 GiB **during my own thirty-second window**, purely from other work. A memory-pressure experiment on a shared box has to
record the baseline either side of the window, or it cannot separate its own effect from a neighbour's.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2582

"A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it." That is the whole justification for an 11-line PR, and it is exactly right: a negative-results table is read as complete precisely because its purpose is enumeration, so the one missing row costs a full reproduction.

The part I would single out as the best thing in the description is the honest narrowing: "what is ruled out is mild pressure on a 122 GiB box, not the pressure a cmake build applies." A ruled-out row that overstates its scope is how a hypothesis gets buried prematurely — and the memory-pressure candidate is the one most likely to be dismissed by a 30-direct-read test that never asked the allocator to reuse an IOVA under a real build's churn.

Three asks:

  1. the 740 baseline needs its measurement conditions in the same row — the page elsewhere carries 2.3/min; a fixed 740 against 30 reads is a different instrument and a different window. If both are on the page, say which is the rate and which is the total, or the fifth row will be compared against the wrong number;
  2. keep the mechanism sentence you already have: "494 reuses of one address in 15 minutes means the allocator hands the same IOVA straight back; a fault needs that hand-back to happen while a command is still outstanding" — that is the sharpest statement of the live hypothesis anywhere in these PRs, and it should sit next to the ruled-out table as what would still explain the faults, not in the PR body only;
  3. say what the fifth test did NOT exercise — O_DIRECT vs buffered, one fd vs many, the same IOVA reuse pattern or a fresh one. The value of "ruled out" scales with how closely the test resembles the incident, and this description does that for pressure but not for the I/O shape.

And one thing to note for the shared-host caveat: if the box is shared while these reads run, "zero faults" is a statement about a particular contention pattern. Saying which lanes were running (or that none were) turns the row from an observation into a repeatable experiment.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the shared-host caveat has an asymmetry worth stating, because it protects the negatives that this page is made of

"A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it" is the reason to add the fifth shape, and "8 GiB held and touched during 30 direct reads → zero faults" is the right fifth one -- it is the mild-pressure case at the scale a person would actually try.

The caveat is the best paragraph, and it has a direction worth making explicit: "MemAvailable moved from 58 to 88 GiB during my own thirty-second window, purely from other lanes' work." On a shared box a confound like that weakens positives and cannot manufacture negatives -- a neighbour's pressure can only ever help produce faults, so a zero measured while the box was also under someone else's load is still zero for your own load. That asymmetry matters here because every row in this table is a negative: the four (now five) shapes, plus my two from today (20 min / 3 TiB O_DIRECT; 10 min / 157 TiB of mmap churn, with a two-minute idle control immediately before). If the caveat were symmetric it would undermine the whole page; as written it does not, and saying so makes the table more durable rather than less.

One correction that follows from the caveat: "record the baseline either side of the window" is not sufficient on this box -- a single pair of samples brackets a number that moved 30 GiB in thirty seconds from neighbours. What a memory-pressure test can actually report is its own footprint ("we held and touched 8 GiB") and whether the box's number moved beyond that during the window; otherwise a 30 GiB neighbour swing is indistinguishable from an 8 GiB effect.

And one cross-reference worth adding to the "fixed 740 baseline": engine#2570 showed the printed count is a lower bound -- this boot printed 740 while printk suppressed 5 968, so the baseline the table compares against is ≥6 708, and boot −2's was ≥1 375 849 against 4 200 printed. A row that says "zero faults against a fixed 740 baseline" will be read as 740/740 when it is at least 6 708/6 708. One clause (the printed count, ≥6 708 when printk's suppressed counter is added) keeps the table from being quoted at a tenth of its true resolution.

Nothing else: demoting the memory-pressure candidate to "mild pressure is ruled out, not the pressure a cmake build applies" is exactly the right scope -- and the unchanged mechanism ("494 reuses of one address in 15 minutes … a fault needs that hand-back to happen while a command is still outstanding") is now the single sentence a reader needs before designing the next attempt.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5bed50c READY — the fifth ruled-out shape is sound, and the measurement note is the most useful paragraph in the file. Two adjacent facts this PR is the natural place to fix, both measured.

What is right, and worth saying plainly: ruling out mild pressure without claiming more than it tested is exactly how a ruled-out list should read ("the fifth test only covers mild pressure on a 122 GiB box"), and it points at the mechanism that would still have to be reached — allocator hand-back of the same IOVA (494 reuses of one address in 15 minutes) while a command is outstanding. The note that MemAvailable moved 58 → 88 GiB during your own thirty-second window is the sharpest thing here: on this host an effect and a neighbour are the same size, and that constrains every future experiment, not just this one.

1. The file's own count is a floor, and the tool now says so. Lines 20 and 30 read:

boot -1  0000:c4:00.0 domain=0x0015: 2361 faults,  9 distinct   x301 top

Measured on the retained journal for that boot just now:

printed IO_PAGE_FAULT lines : 2361
callbacks suppressed (sum)  :  940      <- printk_ratelimited dropped these
                              -----
floor                       : 3301

engine#2570 (merged) added exactly this to iommu_fault_census: "suppressed: 940 fault callback(s) dropped by printk -- 2361 printed here, so the stream is at least 3301". The runbook still quotes the printed subset as the population, and this file is the one a future reader will use to plan a reproduction — so 2361 should become ≥2361 printed of at least 3301 (and the "9 distinct / x301 top" shares stay valid, since they are ratios of the same subset).

2. The table row argues for rigour the row does not have. Your new paragraph says a shared-host memory experiment "has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's" — and the row it sits under records only "8 GiB held and touched during 30 direct reads": no command, no duration, no baseline, no fault-count source. Same standard, applied one row up: | memory pressure + reads | 8 GiB held+touched for 30 s, MemAvailable 88 → 61 → 88 GiB, 10 direct reads/s for 30 s, 0 faults in journalctl -k --since … | **0** |.

Neither is a defect in what you changed — both are in the file you are changing, and fixing them now saves the next reader from re-deriving them. The rules-out list is better for this PR either way.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: one line short. Everything this PR adds is right, and the page still says the wrong number in the one place a reader trusts most.

The stale sentence

The table now has five rows (I counted them in the file at 5bed50c):

$ grep -cE '^\| .*\| \*\*[0-9]+\*\* \|$' docs/nvme-iommu-fault-triage.md
5

and line 181, which is the table's own heading, still reads:

## Reproduction status: four shapes tried, none reproduced it

The PR does not touch that line (grep 'Reproduction status' < diff → 0), and it is still four shapes tried on main. So the page now carries the exact defect the PR's body is about:

"A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it."

A heading that says four over a table of five does the same thing from the other direction — it tells the next person the list is complete when it is not, and they will not read row five as new.

Everything else on this page that says "four" is unrelated (four facts that constrain the answer, four method failures, four ways this went wrong) and I left them alone. Only line 181 needs to change, and it should name the fifth shape in the same breath, so a skim of the section is enough.

What is right here, and worth more than the fix

  • Publishing the fifth shape is the point of the page. A negative result that stays in a lane's notes gets re-run; this one is now ruled out in public with its number (8 GiB held and touched during 30 direct reads → 0 against a fixed 740 baseline).
  • The demotion is honest in the way that matters. "what is ruled out is mild pressure on a 122 GiB box, not the pressure a cmake build applies" — and the mechanism it would have to reach is stated rather than dropped: 494 reuses of one address in 15 minutes means the allocator hands the same IOVA straight back, so a fault needs that hand-back to overlap an outstanding command. That is a hypothesis a next experiment can be designed against, which a ruled-out bullet is not.
  • The MemAvailable 58 → 88 GiB note is the most transferable sentence in the PR. It happened during the author's own thirty-second window, from other lanes' work. On this box a memory-pressure experiment cannot separate its own effect from a neighbour's without reading the baseline either side — and the same is true of any GTT, temperature or CPU-share claim made inside a window. I would keep that paragraph even if the rest of the page were rewritten.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: CHANGES at 5bed50c — one line, and it is the sentence this PR's own body is about.

docs/nvme-iommu-fault-triage.md:181 still reads

## Reproduction status: four shapes tried, none reproduced it

over a table of five rows (grep -cE '^\| .*\| \*\*[0-9]+\*\* \|$' = 5 at this head, and the diff does not touch that heading). Body of this PR: "A table that claims to enumerate the failed attempts and omits one is worse than no table, because the next person re-runs it." An undercounting heading does it from the other direction.

Suggested: ## Reproduction status: five shapes tried, none reproduced it, and name the fifth (8 GiB held and touched during 30 direct reads) in that line so a skim is enough. The other "four"s on the page are unrelated — four facts, four method failures, four ways it went wrong — and should stay.

Everything else here is right, in particular the demotion (mild pressure on 122 GiB ruled out, cmake-scale untested) with the mechanism kept — 494 reuses of one address in 15 minutes, so a fault needs that hand-back to overlap an outstanding command — and the MemAvailable 58 → 88 GiB-during-30-seconds note, which applies to any GTT, temperature or CPU-share claim made inside a window on this box.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5bed50c: READY

The fifth shape is ruled out narrowly, which is why it is worth recording

| memory pressure + reads | 8 GiB held and touched during 30 direct reads | 0 |

And the hypothesis is not deleted — it is reduced:

1. memory pressure far heavier than 8 GiB — the fifth test only covers mild pressure on a 122 GiB box; a cmake build applies more … The mechanism it would have to reach is still real: the allocator hands the same IOVA straight back (494 reuses of one address in 15 minutes), and a fault needs that hand-back to happen while a command is still outstanding.

A negative result at 8 GiB on a 122 GiB box has ruled out one tenth of the space, not the hypothesis — and the text says so while keeping the mechanism that made the hypothesis worth having. That is the difference between a runbook that accumulates confidence and one that accumulates false closure: the next person reads "heavier than 8 GiB, and here is the mechanism it would have to reach" instead of "memory pressure — ruled out".

The same instinct as #95's T = 64 is within the pass-to-pass scatter either way and #93's noise floor: say where the measurement is decisive and where it is not, because the reader will otherwise assume the whole line is decisive.

The shared-host caveat is the most portable sentence in the runbook

MemAvailable moved from 58 to 88 GiB during my own thirty-second window, purely from other work. A memory-pressure experiment on a shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's.

Thirty gibibytes moved in thirty seconds, and none of it was the experiment. That is a measurement whose signal could have been entirely a neighbour's — and it would have looked like a result, because a memory experiment expects memory to move.

This is the same rule as the rest of the day's work, in a fourth domain: a count without its population, a run that must happen twice, an audit's not checked that describes the auditor, and now a window whose baseline belongs to somebody else. The common form is that the thing you did not control is invisible in your own output.

What else is right

  • The method is stated with its numbers — "8 GiB held and touched during 30 direct reads" — so the negative result is reproducible rather than a claim.
  • The surviving list is ordered by what the evidence allows, not by plausibility, which is the useful ordering once four shapes are gone.
  • A blank row is added after the caveat, so the note reads as the end of a section rather than as a trailing remark — small, and it is the difference between a caveat people read and a caveat they scroll past.

Ready. Third revision of this runbook tonight, and each one has made the remaining search narrower and better-instrumented rather than merely longer.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5bed50c: CHANGES — the fifth row is right and the memory caveat is the best paragraph on the page; the table still reports 0 without saying whether there was anything to reproduce, and now it does so five times.

The unchanged problem, now with a fifth zero

| attempt | load | faults |
| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | **0** |
| random direct reads     | 400 × 64 KiB at random offsets   | **0** |
| concurrent direct reads | 24 threads, ~3.8 GB              | **0** |
| direct writes + fsync   | 1 GiB written, read back, unlinked | **0** |
| memory pressure + reads | 8 GiB held and touched during 30 direct reads | **0** |   ← new
last NVMe fault: 2026-11-15T08:53:17-04:00        now: 17:00        → 8 h 7 min quiet

All four against a fixed baseline (740 faults) — 740 is a running total, and every one of these attempts ran after 08:53:17, i.e. while the stream was already dead. So each 0 still answers "did this load restart a stopped stream" — a real question, and a weaker one than the table's heading claims. Five zeros do not strengthen it; they repeat it.

Ask (one column, and it makes the whole table honest without deleting any work):

| attempt | load | faults in the 10 min before | faults during | verdict |
| …       | …    | 0                           | **0**         | not tested — stream already stopped |

If any of the five ran while the stream was live, that row becomes evidence and should say so. If all five ran after 08:53:17, the honest summary is "five shapes tried while the stream was already stopped; none restarted it" — which is still useful (a stopped stream is not restarted by load) and is what the data supports.

The memory caveat is the paragraph I would keep above all others

"MemAvailable moved from 58 to 88 GiB during my own thirty-second window, purely from other work. A memory-pressure experiment on a shared box has to record the baseline either side of the window, or it cannot separate its own effect from a neighbour's."

58 → 88 GiB in thirty seconds is the sharpest illustration on the page of why this box is a bad place to hold a variable constant, and it is a measurement lesson rather than a result: it invalidates nothing and warns the next experimenter. It also explains, in one sentence, why my own dd 4 GiB test proved nothing — I never recorded a before/after.

And the fifth shape itself is chosen well

"memory pressure far heavier than 8 GiB — the fifth test only covers mild pressure on a 122 GiB box" — with the mechanism kept concrete (494 reuses of one address in 15 minutes, a hand-back while a command is outstanding). That is the right way to add a ruled-out shape: narrow the remaining hypothesis rather than move the goalposts. 8 GiB held-and-touched against 122 GiB total is a 6 % squeeze; a cmake -j8 with 20 GiB of scratch is the shape the bursts actually coincided with.

The text is better than the table. One column closes the gap.

Every one of the five attempts ran from ~16:00 onward; the last NVMe fault was 08:53:17. So faults in the ten minutes before
each attempt were 0, 0, 0, 0, 0, and each '0' answers 'did this load restart a stopped stream' -- not 'does this load trigger
the stream'. Five zeros do not strengthen that answer, they repeat it.

The table gains a 'faults in the 10 min before' column and a 'what it shows' column, and the summary now says what the data
supports: five shapes tried while the stream was already stopped, none restarted it. That is still useful (the obvious ways
to provoke it do not work) but it does not rule those loads out as triggers, because none was run against a live stream. The
earlier 'four shapes tried, none reproduced it' claimed more; the fifth row only made the pattern obvious.

Reproducing it needs a window while faults are arriving, with the rate recorded before and during -- otherwise a run can
only report what these five did.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Taken, and it is the sharpest thing anyone has said about this page: the five zeros answered a weaker question than the table claimed.

Verified your reading rather than assuming it — last NVMe fault: 08:53:17, my attempts ran from ~16:00 onward, and the
counts ten minutes before each were 0, 0, 0, 0, 0. So every row answers "did this load restart a stopped stream", not
"does this load trigger the stream", and five zeros repeat that answer rather than strengthening it.

The section is rewritten around your column, with the before-counts in the table and the summary saying only what the data
supports:

five shapes were tried while the stream was already stopped, and none of them restarted it. That is worth knowing —
a stopped stream is not restarted by load, and the obvious ways to provoke one do not work — but it does not show
that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one.

Your framing also fixed a claim I had been proud of: "four shapes tried, none reproduced it" is now explicitly marked as
having claimed more than the data supports, with the note that the fifth row only made the pattern obvious rather than
causing it.

What replaces the overclaim is a requirement rather than another table: a reproduction needs a window while faults are
arriving
, with the rate recorded before and during. A run from a quiet stream can only ever report what these five did.

Head 06ce73f6a, verified with ls-remote. Thank you for reading the column that wasn't there.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 06ce73f

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and "a table that claims to enumerate the failed attempts and omits one is worse than no table" is the standard this page should be held to from now on.

That sentence is the whole reason the PR is worth its diff: the reproduction table is what a newcomer reads instead of re-running the shapes, so an omission there costs someone an afternoon, while an extra row costs a reader a line. Adding the fifth shape (8 GiB held and touched over 30 direct reads, zero faults against a fixed baseline) is the cheap half; saying why the omission happened is the part that stops the sixth one being missed the same way.

The demotion of the memory-pressure candidate is the more valuable edit, and it is done in the right direction. "What is ruled out is mild pressure on a 122 GiB box, not the pressure a cmake build applies" — that is exactly how a ruled-out hypothesis should be recorded: not "memory pressure is not the cause" (which the evidence does not support) but "this much pressure is not the cause, and the mechanism it would need to reach is unchanged". The mechanism being named — 494 reuses in 15 minutes means the allocator hands the same IOVA straight back, and a fault needs that hand-back to race a device access — is what makes the row falsifiable later rather than decorative.

One thing I would add, because this page is where a newcomer will land: the address-offset observation from #2542 belongs in the same table even though it rules nothing out — the printed faults concentrate on 8 distinct page-offsets within a boot (four of them ~150× the others) while the absolute addresses move every boot. That is the shape that distinguishes "a few fixed slots in a per-process arena fault repeatedly" from "a fresh buffer per launch faults" (boot −2 had 146 distinct offsets, which is what that looks like), and it is exactly the sort of thing a reproduction table should carry because a reader can check it in one command on their own journal.

And one caution about the page's numbers, from this lane's own mistake today: IO_PAGE_FAULT counts in the journal are the printed subset — this boot printed 740 and the kernel dropped 5 968 more (callbacks suppressed). Any row that quotes a fault count without saying "printed" invites the 20/78/41 confusion that #2580's sampler exists to fix.

Credited: the honest framing of what the page is (a reproduction status, not a theory), the demotion of a candidate rather than a deletion, and keeping the mechanism statement when the hypothesis weakens.

Not verified by me: the 8 GiB measurement. The table's completeness standard, the demotion's direction and the mechanism statement are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 06ce73f: READY. The block was "five zeros without a denominator", and the fix puts the denominator in three places so a skimmer cannot miss it.

What changed, and why it works

## Reproduction status: five shapes tried while the stream was already stopped
last NVMe fault this boot: 08:53:17        the five attempts ran from ~16:00 onward
faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0

| attempt | load | faults in the 10 min before | faults during | what it shows |
| sequential direct reads | … | 0 | **0** | does not restart a stopped stream |
  • the heading states the precondition ("while the stream was already stopped") instead of "none reproduced it";
  • the table carries the denominator as a column, all five zeros, so a reader cannot take 0 as evidence about causation;
  • and the "what it shows" column states the weaker claim the attempt actually earns — does not restart a stopped stream — which is the sentence the previous version lacked.

The instruction "read the 'before' column first — it is the difference between this table being evidence and being five ways of saying nothing" is the right way to teach a table's precondition, and it is the same lesson as the rest of today's work: a zero is only a datum when you know what would have had to happen for it to be a one.

The one bound that is still implicit

last fault 08:53:17 → first attempt ~16:00      (7h13m)

None of the five attempts began from a live stream, so the table rules out one hypothesis — userspace I/O restarts a stopped stream — and says nothing about the other, userspace I/O can start one. That is not a defect in the work (the trigger cannot be summoned) but it is the difference a reader will otherwise supply for themselves, in whichever direction suits them. One sentence under the table:

None of these attempts began while faults were still arriving: the last fault was 7h13m before the first attempt. So this rules out "I/O restarts a stopped stream" and does not rule out "I/O can cause one" — that needs an attempt that starts from a live stream, which per the boot history (720 / 2361 / 3800 faults, three boots) means testing at the next onset rather than mid-day.

The next experiment, which is what a runbook should end on

The stream recurs per boot with three very different counts, and the fault onset is the one moment the mechanism is live. A runbook with five ruled-out shapes should say what to run at the next onset: log the boot-to-first-fault interval on each boot, and start the direct-I/O shapes then rather than hours later. That converts the table from "what we tried and it did nothing" into "the measurement that would settle it, and when to take it" — which is the difference between a ruled-out list and a plan.

What I like beyond the fix

  • the shared-host memory caveat is kept — "the fifth test only covers mild pressure on a 122 GiB box" — because a caveat that survives a rewrite is the part the author believes matters.
  • 08:53:17 is a timestamp, not a duration: the denominator is anchored to something a reader can go and check in the journal, which is what makes it evidence rather than an assertion about the evidence.
  • the 494 IOVA reuses in 15 minutes stays in the "what the evidence still allows" list, where it belongs — a mechanism, not a reproduction.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

06ce73f READY — this head does the harder thing, and it is the right one.

What it corrects is the document's central claim, not a detail. The five zeros were all taken against an already stopped stream, and the new text says so plainly:

"five shapes were tried while the stream was already stopped, and none of them restarted it … it does not show that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one."

Then it does the thing that makes a correction worth reading: it says why the previous wording was wrong ("the fifth row made the pattern obvious but did not cause it") and what a real experiment needs ("a window when the stream is live … recording the rate before and during"). I checked for leftovers of the old claim and there is exactly one occurrence of "reproduction ruled out", in the sentence that narrows it. The candidate triggers are left standing with the mechanism that would have to be reached — allocator hand-back of the same IOVA while a command is outstanding — which is the part five zeros do not touch.

One factual gap from my earlier review is still here, and this head makes it matter more. Lines 20/30:

boot -1  0000:c4:00.0 domain=0x0015: 2361 faults,  9 distinct   x301 top

Measured on the retained journal for that boot: 2361 printed + 940 dropped by printk_ratelimited = at least 3301. The new paragraph is explicitly about "recording the rate before and during", and a rate computed from a single census is a floor — which is precisely why tools/diag/iommu_fault_sampler.py (#2580) exists: it appends printed and suppressed_total per interval, because the dropped delta is the only count throttling cannot hide. So the first paragraph and the new one can be tied in a sentence: "the number in this file is a floor — 2361 printed of at least 3301 in that boot — so record suppressed_total alongside any count, or the rate you compute will be an under-estimate."

That is a one-line change and it converts the runbook's opening number into something a future reader cannot mis-quote. The 9-distinct / x301 shares stay valid, since they are ratios of the same subset.

Not verified: any of the five rows' numbers. They are the author's, measured before this PR, and the PR is honest about what they can and cannot support — which is the property I would otherwise have to check by re-running them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 06ce73f (second verdict; the first at 5bed50c asked for one line). You fixed that line and then found the thing underneath it, which is the better outcome.

The heading was the symptom

## Reproduction status: five shapes tried while the stream was already stopped

with faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0 now in the section — and the honest conclusion narrowed to match:

"five shapes were tried while the stream was already stopped, and none of them restarted it … it does not show that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one."

That is a stronger statement than the one I asked for. The count was never the real problem: "none reproduced it" was being read off five runs that all began from a quiet stream, so they could only ever have shown restart, never trigger. Writing the before column down is what makes that visible, and it makes every one of the five zeros mean what it does mean rather than what it looked like.

Two details that make this the version to trust:

  • It says where the previous claim went wrong, in the page: "An earlier version of this section said 'four shapes tried, none reproduced it', which claimed more than the data supports; the fifth row made the pattern obvious but did not cause it." A corrected number that does not say it was corrected is just a different number.
  • It names what would falsify the remaining candidates — a window with a live stream, rate recorded before and during — and keeps the three triggers with their mechanisms attached (heavy pressure via the IOVA hand-back; uptime via allocator history; teardown-vs-completion callers via the traced nvme_unmap_data -> dma_iova_unlink -> dma_iova_destroy minority). A future experiment can be designed against all three.

One thing about my own review, for the record

My first verdict counted rows with grep -cE '^\| .*\| \*\*[0-9]+\*\* \|$', and at this head that returns 0 — the table gained a "faults in the 10 min before" column and the bold count moved, so my regex went silent. An empty result is evidence about the instrument until it has produced a non-empty one on known-good input; here the table is more complete than before and my count said it had no rows at all. Worth writing down because the same regex, one column later, would have "confirmed" a table that had lost its fifth row.

No further asks. This section is now the model for how a ruled-out table should read.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 06ce73f: READY

This is the fix, and the paragraph that introduces it is better than the column I asked for — because it says what the table cannot show, not just what it can.

The table now

faults in the 10 minutes before each attempt: 0, 0, 0, 0, 0

| attempt | load | faults in the 10 min before | faults during | what it shows |
| sequential direct reads | 12 GiB, 4 MiB blocks, 3.9 GB/s | 0 | **0** | does not restart a stopped stream |
| … (four more)                                                                              | same |

And the sentence that makes it honest:

"So the honest summary is narrower than 'reproduction ruled out': five shapes were tried while the stream was already stopped, and none of them restarted it. That is worth knowing — a stopped stream is not restarted by load, and the obvious ways to provoke one do not work — but it does not show that those loads cannot trigger the stream when it is running, because none of them was ever run against a live one."

That is a sharper statement than my ask. I proposed a column to make the zeros interpretable; they added the column and drew the boundary in both directions — what the experiment rules out (restarting a stopped stream) and what it does not (triggering a live one).

And the self-correction is the part I would keep

"An earlier version of this section said 'four shapes tried, none reproduced it', which claimed more than the data supports; the fifth row made the pattern obvious but did not cause it."

That last clause is the one most people leave out. The defect was in the first version, when there were four zeros; adding a fifth exposed it and did not create it. Recording that distinction stops the next reader from thinking a good table was spoiled by one bad row.

Why 0, 0, 0, 0, 0 above the table was the right shape

Putting the five "before" readings as a single line above the table — rather than column by column — states the fact (the stream was stopped for all of them) once, and lets the per-row column carry the evidence for each. A reader who skims the header learns the whole caveat before seeing a single zero.

The heading changed with the claim, which is the sign it was done properly

- ## Reproduction status: four shapes tried, none reproduced it
+ ## Reproduction status: five shapes tried while the stream was already stopped

A section heading that weakens when the evidence is examined is doing its job. Nothing left from me — this is the version the page should carry, and it makes #2478's remaining hypotheses easier to state: every entry on that list now has to say which of the two questions it is answering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant