edge0-8b: ~7-13s to first token on M1 Pro 16 GB, unchanged by warm_willneed / full_layer_prefill

[Hamaad616]

Environment

• MacBook Pro (MacBookPro18,1), Apple M1 Pro, 16 GB
• macOS 26 (build 26A428)
• Python 3.14.7 (venv from pip install -e '.[dev,fetch]'; not tested on 3.12)
• edge0 0.1.0, Edge0/Edge0-8B-A1B-preview weights (LoRA and prerouter files present; logs show [lora] applied=153 not_found=0 and prerouter installed: 16 heads, start=7, K=8)
• Model on the internal SSD, no other heavy apps running, one edge0 process only

What I see
Decode speed is reasonable, but every request has a long delay before the first token, even for a tiny prompt.

Server (edge0 serve, warm process), streaming request with a 27-token prompt. Timestamps on the SSE stream show the first chunk arriving about 10.8s after the server stamped the request, then chunks every ~0.1-0.3s.

Non-streaming timings against the same server:

Tokens generated | Total time -- | -- 12 | 10.5s 19 | 14.7s 200 | 25.7s

The engine reports built in 0.4-0.5s, so it isn't model load time.

*I'm not fully certain the edit applied on this run, since timings are nearly identical to the previous config.

What I checked

• edge0 models shows prefill_full=0 hot=0 for edge0-8b. As I read the docs, prefill_full_layers=0 means "every layer", and prod_k8() sets full_layer_prefill=True.
• warm_willneed defaults to False and is only enabled in staged_k4 (35B), so I tried it on the 8B preset. No meaningful change.
• GPU sampling during a request (powermetrics): GPU active residency 42-67%, ~0.8-1.4 W, i.e. mostly waiting rather than computing.
• No competing processes. The edge0 server's RSS was ~1.9 GB.

Questions

1. Is a fixed delay of this size per request expected on 16 GB machines, given the README benchmarks are from a 24 GB M4 Pro?
2. What is happening in that window? From the docs I suspected whole-layer prefill through the page cache, but toggling the options above made no visible difference.
3. Is there a supported way to keep more of the model resident between requests, or a setting I should be using on 16 GB?

Happy to run any diagnostics you suggest.

---

Some notes on the draft before you post it:

• Check the last table row. I flagged it because the timings looked identical to the previous run. Re-run with full_layer_prefill=False after confirming line 54, and update those numbers (or drop that row) so the report doesn't contain something you can't stand behind.
• Python 3.14. It's worth trying 3.12 first, since the README recommends it and maintainers may ask. If it makes a difference, that changes the report entirely.
• Tone check on point 2 of "What I checked". I hedged on what prefill_full_layers=0 means because I'm reading a docstring, not the code. If a maintainer corrects it, that's useful information rather than an error.
• Consider adding your sysctl output and the exact commands. The curl loop with timestamps from earlier is a good reproducible test.

[linyubupa]

Thanks for the detailed report — the powermetrics sample, and flagging that you couldn't stand behind the last table row, are what made this tractable. Sorry it took us this long to see it.

Two things were going on:

1. edge0-8b's preset prefills with prefill_full_layers=0, which means every layer — so each request loads all 128 experts of all 23 layers: 4.06 GiB, independent of prompt length. On 16 GB that read comes from disk every time, at roughly 0.7 GB/s, which is your ≈10.8 s. Your reading of the docstring was correct.

2. Your toggle did nothing because it was a no-op: the engines installed the prefill hook regardless of full_layer_prefill, and the hook fell through to load_full_layer() whenever prefill_full_layers=0. Counted on edge0-8b: 23 whole-layer loads with the flag True, and 23 with it False. So your measurements were right and the flag was broken. That one is on us.

Both are fixed on main now:

• fix(prefill): honor full_layer_prefill=False on the streaming engines #111 makes full_layer_prefill=False do what it says;
• feat(cli): --prefill-ondemand for demo / chat / serve #112 adds edge0 serve <model> --prefill-ondemand (also on demo / chat), which prefills through per-expert loads only. Same 27-token prompt, cold cache, M4 Pro: 4.06 GiB / 5.3 s → 0.4–0.8 GiB / ≈1.1 s, peak memory unchanged (878 MB).

warm_willneed only applies to the decode path, so no change there was expected — sorry for the experiment that went nowhere.

Could you pull the latest main and re-run your test? We'd expect your ≈10.8 s first token to drop to roughly 1–2 s on your M1 Pro:

git pull
edge0 serve /path/to/edge0-8b --prefill-ondemand

[agourakis82]

I could reproduce this and traced it to a concrete mechanism — not on a 16 GB machine (mine is an M5 Max, 36 GB), so treat the absolute numbers as illustrative, but the shape of the result should transfer directly to a tighter-RAM box.

Root cause: load_full_layer() re-reads the whole model's expert weights every request, cold-cache-dependent

edge0-8b's production profile (LayerOptions.prod_k8()) sets full_layer_prefill=True, prefill_full_layers=0. Per make_prefill_before_layer (src/edge0/engine/hooks.py), prefill_full_layers=0 means every layer gets load_full_layer() called during prefill — not just a leading subset. load_full_layer() (src/edge0/streaming/layer.py) does a direct mmap-backed raw() read + bitcast per (proj, part) tensor per layer; its own comment claims "~9ms/layer" — true only when the pages are already in the OS page cache.

Instrumented load_full_layer() timing across 3 back-to-back requests on the real checkpoint (max_new_tokens=16, greedy, no server restart between runs):

run 0 (cold):  total=5.27s  first_token=4.72s  load_full_layer sum=4.223s (23 calls)
run 1 (warm):  total=0.85s  first_token=0.38s  load_full_layer sum=0.241s (23 calls)
run 2 (warm):  total=0.83s  first_token=0.29s  load_full_layer sum=0.158s (23 calls)

load_full_layer() alone accounts for ~90% of first-token latency on the cold run, and drops ~17x once the pages are resident. On a 16 GB machine, if anything (other apps, the OS itself, or just enough concurrent memory pressure from the server process's own other allocations) evicts those pages between requests, every request pays something close to the cold number instead of the warm one — which matches "every request has a long delay" in your report far better than a one-time cold start would.

Why warm_willneed did nothing for you

It's not a red herring on your part — it's dead code for this profile. warm_willneed is only read in StreamingSwitchGLU.prefetch() and stage_experts() (src/edge0/streaming/layer.py), and stage_experts() early-returns immediately when self._staged_mode is False. prod_k8() sets staged=False and history_prefetch=False (the deployment profile note even says staged decode was found to degrade ling output and is deliberately off) — so neither call site that reads warm_willneed ever executes for edge0-8b in its default configuration. Toggling it can't do anything.

What actually helps today

EDGE0_PREWARM=1 (or LING_PREWARM=1) triggers Ling8BEngine._prewarm() at server startup: madvise(WILLNEED) + a full sequential read over every shard, plus a dummy prefill+step to pre-JIT the decode kernels. That's a real, working mitigation for the first request after server start — but it only runs once, so it won't help if pages get evicted again between later requests under memory pressure, which is the scenario a 16 GB box is most likely to hit repeatedly rather than just once.

Answering your three questions directly

1. Not a fixed cost independent of RAM — it's page-cache residency dependent, and 16 GB vs 24 GB changes how often you fall off that cliff, not whether the mechanism itself exists.
2. Confirmed: it's exactly the whole-layer prefill you suspected, not something else. The load_full_layer() cost is real disk I/O when cold, not compute.
3. Two things worth trying on 16 GB, neither requires a code change: (a) EDGE0_PREWARM=1 before the first request of a session; (b) if you have programmatic access to LayerOptions, set prefill_full_layers to a small N (e.g. 6-8) instead of 0 — make_prefill_before_layer already supports partial full-layer prefill (only the first N layers get the full-tensor path; the rest fall back to on-demand/hot-stack), which trades some steady-state decode throughput for a much smaller resident footprint that's more likely to survive between requests on a tight-memory machine. I haven't benchmarked (2) myself since I don't have 16 GB hardware to validate it against — flagging it as a hypothesis from reading the code, not a verified fix.

Happy to open a PR for a README/docs/streaming.md clarification (the warm_willneed-is-inert-under-prod_k8 gotcha in particular seems worth documenting explicitly) if that's useful, or to try to reproduce further with a memory-pressure simulation if a maintainer wants that.

[agourakis82]

Correction/addition to my comment above: I missed that this already has a supported, documented fix — edge0 demo|chat|serve --prefill-ondemand (src/edge0/models/base.py::apply_prefill_switches, forces full_layer_prefill=False). docs/streaming.md already states the exact tradeoff this issue is hitting:

The whole-layer prefill is the fastest path when the checkpoint stays in the page cache (warm 27-token prefill: 0.24s vs 0.37s on-demand on an M4 Pro), and the slowest one when it does not (cold: 5.3s/4.06 GiB read vs 0.6-1.1s/0.4-0.8 GiB; ... --prefill-ondemand selects the on-demand path for machines in the second group.)

That's essentially your exact numbers (my cold full-layer run: 4.72s) mapped to an existing, named cause with a maintainer-recommended fix for your machine class. I tried forcing --prefill-ondemand-equivalent behavior myself and did not cleanly reproduce the doc's 0.6-1.1s cold figure (I saw 5.63s) — but I'm running on a dev machine with a lot of concurrent activity right now (many other processes competing for page-cache residency), so I don't trust that number as a clean baseline and won't claim it disproves the doc. The strikeout-worthy part of my previous comment is the speculative prefill_full_layers=N tuning suggestion — try --prefill-ondemand first, it's the tested, documented path for exactly this symptom, before reaching for anything untested.

Would still be worth you confirming --prefill-ondemand actually resolves it on your 16 GB box, since the doc's numbers are from a 24 GB M4 Pro, not your tier.