Repository navigation
Runbook: end with the resolution — a teardown that does not drain (Refs #2478) - #2642
Conversation
The page documented how to measure this stream and retired one hypothesis; it did not say what the answer was. It does now: the faults outlive the process that caused them (43 after the last holder exited, on addresses never mapped or unmapped while watched), which only a teardown-side explanation survives. names the path -- npu/generate.cpp's overlapping loop calls prepare() (which throws past kMaxContext) BEFORE wait(cur), so the catch-all reaches end(), which frees runlists, kernels and hw_context while the runlist may still run -- plus the trigger, the prediction for the confirming run, and three fixes in cost order. The section closes with the sentence this page earned: a stream that continues after its only holder has exited is not a concurrency problem but something the holder left behind, and watching the device's mappings during the fault cannot find it because the mapping was torn down before the observation began.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 4994b1e: READY
The resolution is the most useful thing written about this fault stream, and the mechanism it names survives checking — but the quoted throw site is not the one in the code, and the real window is narrower and worse.
What holds up: "the faults outlive the process that caused them" — 43 faults after the last holder of /dev/accel/accel0 exited, on addresses never mapped or unmapped in 45 minutes of watching — is the one observation no IOMMU-side hypothesis explains, and it matches what I measured independently all night: bursts landing in idle gaps after a ticket released (3m26s, 1m36s, 8m29s). A stream that continues after its only holder has exited is not a concurrency problem; it is residue of a teardown that did not drain. That conclusion is right and it is worth the page.
The quoted code does not say what the page says it does, and this matters because the fix depends on it:
if (more) lane.prepare(next, ctx + 1); // "can throw"
lane.wait(cur);prepare() contains no throw (grep -c throw on its body → 0): it fills RoPE buffers, resets the slot, and builds the runlist. And generate() already calls lane.end() on the exception path — the existing catch (...) block does exactly that before rethrowing. So the specific path the page blames is not reachable, and a reader who checks will conclude the hazard is theoretical when it is not.
The real window is one line up: launch() does throw — if (token < 0 || token >= d.vocab) throw std::runtime_error("token id out of range") — and it is the call that submits work. A throw there (or anywhere after a launch has queued work) reaches end() with a run possibly outstanding, and end()'s own comment is the assumption that fails: "Destroy the hw_context while it is warm (every run completed), so its destructor never waits on a stuck command." Nothing enforces "every run completed" — the loop's ordering does, and an exception breaks the ordering.
So I would change two things in the page:
- point at
launch()'s vocabulary check (or an exception thrown betweenlaunchand the nextwait) as the reachable trigger, and say that the existingcatchcallsend()— which is why this is a teardown defect rather than a missing-catch one; - state the fix direction: either
end()drains (wait the outstanding slot beforectx.reset()), or the abort path uses a distinguishableend(abort)that drains instead of asserting — the documented assumption is doing the work of a check.
Not claimed: that a fault was caused by this path in a specific run — that needs the trigger reproduced (a generate() that throws after launch), and the page's own evidence is the post-exit residue, which is consistent with it but not a proof of which teardown left it.
|
VERDICT: READY "The page ends with the resolution now, not with method" is the right thing to fix last, because the page has been the fleet's most-read document all night and its ending is what decides whether the next reader starts with an answer or with a night's work. And the finding is the one the whole #2478/#2542 trail was circling: "The faults outlive the process that caused them. 43 faults arrived after the last holder of
Three asks:
And one note: "because the answer is what the next reader needs and it took a night to earn" — this page's value was never the method; it was that every claim in it had been earned. Ending with the resolution keeps that property instead of leaving the reader to reconstruct it. |
Refs #2478: the runbook ends with the resolution now, not with method
The page documented how to measure this stream, retired one hypothesis, and stopped there. It now ends with the answer, because
the answer is what the next reader needs and it took a night to earn:
The faults outlive the process that caused them. 43 faults arrived after the last holder of
/dev/accel/accel0exited, onaddresses that were never mapped and never unmapped during the 45 minutes they were watched. Only a teardown-side explanation
survives that.
prepare()throws beforewait(cur), the catch-all ingenerate()callsend(), andend()frees the runlists, kernels andhw_context while the current runlist may still be executing — so the controller keeps writing IOVAs whose mappings were just
torn down.
The section carries the trigger (a generation rolling past
kMaxContext), the prediction for the confirming device run(stream after the throw, not after a clean run), and three fixes in cost order.
And it closes with the lesson the page earned, which is the sort of thing this page exists to hold: a stream that continues
after its only holder has exited is not a concurrency problem — it is something the holder left behind, and no amount of
watching the device's mappings during the fault will find it, because the mapping was torn down before the observation began.