Skip to content

docs(streaming): correct load_hot_layer's "page cache" backing claim #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
agourakis82 wants to merge 2 commits into
base: main
Choose a base branch
from
+7 −3
Open

docs(streaming): correct load_hot_layer's "page cache" backing claim #118

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
58014c4
docs(streaming): correct load_hot_layer's "page cache" backing claim
agourakis82 Sep 22, 2026
538b669
docs(streaming): distinguish paging from file-backed eviction
agourakis82 Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions python/src/edge0/streaming/layer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -992,8 +992,12 @@ def load_hot_layer(self, n_hot: int = 128) -> None:
"""Load/refresh the layer's hot-expert weights.

Backing store = one NUMPY array per (proj, part) holding the
stacked rows of the top-n_hot experts (page cache, survives across
requests, counts ZERO toward MLX active). The mx window
stacked rows of the top-n_hot experts. ``np.concatenate`` copies,
so this is anonymous process memory, not file-backed page cache.
The layer retains it across requests; these host allocations are
outside MLX active memory. The OS may page them out, but cannot
discard and reload them from the checkpoint like clean mapped
file pages. The mx window
(materialize_hot) exposes a few layers at a time so peak MLX
memory stays at window_size * n_hot * ~1.2MB instead of
40 * n_hot * 1.2MB.
Expand Down Expand Up @@ -1034,7 +1038,7 @@ def load_hot_layer(self, n_hot: int = 128) -> None:
raw = self._shard_for(name).raw(name)
per = raw.size // self.num_experts
rows = [raw[e * per:(e + 1) * per] for e in hot]
# concatenated numpy backing (page cache, not GPU), plus
# copied numpy backing (anonymous process memory), plus
# one all-zero row: the prefill path sends misses to row
# n_hot ("overflow row") and needs it to contribute zero.
# Without it that gather reads past the end of the stack.
Expand Down
Loading