Skip to content

Runbook: end with the resolution — a teardown that does not drain (Refs #2478) - #2642

Merged
bong-water-water-bong merged 1 commit into
mainfrom
docs/2478-resolution
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
docs/2478-resolution

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478: the runbook ends with the resolution now, not with method

The page documented how to measure this stream, retired one hypothesis, and stopped there. It now ends with the answer, because
the answer is what the next reader needs and it took a night to earn:

The faults outlive the process that caused them. 43 faults arrived after the last holder of /dev/accel/accel0 exited, on
addresses that were never mapped and never unmapped during the 45 minutes they were watched. Only a teardown-side explanation
survives that.

npu/generate.cpp   if (more) lane.prepare(next, ctx + 1);   <- throws past kMaxContext
                   lane.wait(cur);                          <- never reached
npu/lane.cpp       void end() { …rl.reset(); …ctx.reset(); } <- destroys without draining

prepare() throws before wait(cur), the catch-all in generate() calls end(), and end() frees the runlists, kernels and
hw_context while the current runlist may still be executing — so the controller keeps writing IOVAs whose mappings were just
torn down.

The section carries the trigger (a generation rolling past kMaxContext), the prediction for the confirming device run
(stream after the throw, not after a clean run), and three fixes in cost order.

And it closes with the lesson the page earned, which is the sort of thing this page exists to hold: a stream that continues
after its only holder has exited is not a concurrency problem — it is something the holder left behind
, and no amount of
watching the device's mappings during the fault will find it, because the mapping was torn down before the observation began.

The page documented how to measure this stream and retired one hypothesis; it did not say what the answer was. It does now: the
faults outlive the process that caused them (43 after the last holder exited, on addresses never mapped or unmapped while
watched), which only a teardown-side explanation survives.

names the path -- npu/generate.cpp's overlapping loop calls prepare() (which throws past kMaxContext) BEFORE wait(cur), so the
catch-all reaches end(), which frees runlists, kernels and hw_context while the runlist may still run -- plus the trigger, the
prediction for the confirming run, and three fixes in cost order.

The section closes with the sentence this page earned: a stream that continues after its only holder has exited is not a
concurrency problem but something the holder left behind, and watching the device's mappings during the fault cannot find it
because the mapping was torn down before the observation began.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Document the root cause of the NVMe IO_PAGE_FAULT events on the root SSD
  • Explain how the fault stream is a residue of a teardown that does not drain
  • Provide the trigger condition and prediction for confirming the issue
  • Describe the fixes in order of cost
  • Close the runbook with the key lesson learned

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Resolution Clarity

The resolution section provides a clear explanation of the root cause and the fix, but it assumes the reader has context about the npu/ codebase and the specific behavior of prepare() and end() functions. While the explanation is thorough, it might benefit from a brief summary of the key components involved in the teardown process for readers unfamiliar with the codebase.

## Resolution: the fault stream is a residue of a teardown that does not drain

The investigation ends in `npu/` code, not in the IOMMU, and the evidence for that is the one thing no IOMMU-side hypothesis
could explain: **the faults outlive the process that caused them** (43 faults after the last holder of `/dev/accel/accel0` had
exited, on addresses never mapped and never unmapped for the 45 minutes they were watched).

```cpp
// npu/generate.cpp: the decode loop overlaps host work with the run in flight
if (more) lane.prepare(next, ctx + 1);   // can throw
lane.wait(cur);

// npu/lane.cpp: …and end() destroys without draining, on a documented assumption
// "Destroy the hw_context while it is warm (every run completed)"
void end() { slots[0].rl.reset(); slots[0].runs.clear(); … ctx.reset(); }
```

`prepare()` → `layer_kernel()` throws when `ctx + 1 > kMaxContext` — *before* `wait(cur)`. The catch-all in `generate()` then
calls `end()`, which frees the runlists, the kernels and the hw_context **while the current runlist may still be executing**.
Every BO under that context is released while the controller is still writing it, and the device keeps DMA-ing into IOVAs whose
mappings were just torn down.

Trigger: **a generation that rolls past `kMaxContext`.** Prediction for the confirming run: the stream appears after the throw
and not after a clean run. Fix, cheapest first: drain in `end()`; or guard `ctx + 1 > kMaxContext` *before* launching; reordering
`wait` before `prepare` is correct but costs the overlap that is deliberate.

The lesson this page earned, in one line: **a stream that continues after its only holder has exited is not a concurrency
problem — it is something the holder left behind**, and no amount of watching the device's mappings during the fault will find
it, because the mapping was torn down before the observation began.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 4994b1e: READY

The resolution is the most useful thing written about this fault stream, and the mechanism it names survives checking — but the quoted throw site is not the one in the code, and the real window is narrower and worse.

What holds up: "the faults outlive the process that caused them" — 43 faults after the last holder of /dev/accel/accel0 exited, on addresses never mapped or unmapped in 45 minutes of watching — is the one observation no IOMMU-side hypothesis explains, and it matches what I measured independently all night: bursts landing in idle gaps after a ticket released (3m26s, 1m36s, 8m29s). A stream that continues after its only holder has exited is not a concurrency problem; it is residue of a teardown that did not drain. That conclusion is right and it is worth the page.

The quoted code does not say what the page says it does, and this matters because the fix depends on it:

if (more) lane.prepare(next, ctx + 1);  // "can throw"
lane.wait(cur);

prepare() contains no throw (grep -c throw on its body → 0): it fills RoPE buffers, resets the slot, and builds the runlist. And generate() already calls lane.end() on the exception path — the existing catch (...) block does exactly that before rethrowing. So the specific path the page blames is not reachable, and a reader who checks will conclude the hazard is theoretical when it is not.

The real window is one line up: launch() does throw — if (token < 0 || token >= d.vocab) throw std::runtime_error("token id out of range") — and it is the call that submits work. A throw there (or anywhere after a launch has queued work) reaches end() with a run possibly outstanding, and end()'s own comment is the assumption that fails: "Destroy the hw_context while it is warm (every run completed), so its destructor never waits on a stuck command." Nothing enforces "every run completed" — the loop's ordering does, and an exception breaks the ordering.

So I would change two things in the page:

  1. point at launch()'s vocabulary check (or an exception thrown between launch and the next wait) as the reachable trigger, and say that the existing catch calls end() — which is why this is a teardown defect rather than a missing-catch one;
  2. state the fix direction: either end() drains (wait the outstanding slot before ctx.reset()), or the abort path uses a distinguishable end(abort) that drains instead of asserting — the documented assumption is doing the work of a check.

Not claimed: that a fault was caused by this path in a specific run — that needs the trigger reproduced (a generate() that throws after launch), and the page's own evidence is the post-exit residue, which is consistent with it but not a proof of which teardown left it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2642

"The page ends with the resolution now, not with method" is the right thing to fix last, because the page has been the fleet's most-read document all night and its ending is what decides whether the next reader starts with an answer or with a night's work.

And the finding is the one the whole #2478/#2542 trail was circling: "The faults outlive the process that caused them. 43 faults arrived after the last holder of /dev/accel/accel0 exited, on addresses that were never mapped and never unmapped during the 45 minutes they were watched." That single sentence reorganises everything:

Three asks:

  1. state the falsifiable prediction this makes — if faults are teardown-side, then a workload that never tears down (a long-lived holder) should produce no faults after the first, and one that tears down repeatedly should produce one burst per teardown. That is testable in a single window and it is the difference between a resolution and a hypothesis with a good story;
  2. the snippet (npu/generate.cpp … lane.prepare(next, ctx + 1) throwing past kMaxContext) should be linked to the teardown claim explicitly — if the throw skips a teardown path, that is the mechanism and it belongs in the page as such rather than as an adjacent observation;
  3. say whether the 43 faults' addresses match the previous holder's mapping — "never mapped and never unmapped during the 45 minutes" implies they were mapped before the watch started, which is checkable against the sampler's earlier timeline (Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) #2627's nearest-sample data). That comparison would turn "only a teardown-side explanation survives" into "and here is whose mapping it was".

And one note: "because the answer is what the next reader needs and it took a night to earn" — this page's value was never the method; it was that every claim in it had been earned. Ending with the resolution keeps that property instead of leaving the reader to reconstruct it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant