llama : add --lazy-experts for MoE models larger than RAM - #26003
Conversation
Routed MoE expert tensors are the bulk of a large MoE checkpoint but only a few are touched per token. --lazy-experts maps the model without populating them: only the non-expert regions are prefetched (posix_madvise WILLNEED) and experts fault in from the page cache the first time they are routed to. This lets a model whose experts do not fit in RAM run off the page cache, which the OS then manages as an LRU of the hot expert set. On a 213 GB GLM-5.2 IQ1_M with 123 GB of RAM this is the difference between not running and running: ~198 GB of experts are left on demand and ~14.5 GB of non-expert weight is prefetched. Readahead is deliberately left enabled on the expert regions -- when the working set exceeds RAM the load is disk-bound, and a fault that pulls a larger contiguous chunk of the expert row is a win. LLAMA_LAZY_EXPERT_RANDOM forces MADV_RANDOM instead, which avoids dragging in neighbours and can be preferable when the model does fit. Two fixes fall out of this, both independent of the new flag: * llama_mmap::advise_range() aligns its range before calling (posix_)madvise. These require a page-aligned start, and tensor offsets are not aligned, so an unaligned call fails with EINVAL and the hint is silently dropped. Advisory hints round outward so the whole range is still covered; DONTNEED rounds inward so a neighbouring tensor's pages are never dropped. The error is also now reported correctly -- posix_madvise() returns the error number and does not set errno, so strerror(errno) printed an unrelated message. * When a tensor is uploaded from the mmap into a device buffer its mapped pages are dead weight. On a model larger than RAM they evict pages that are still needed and thrash the load, so they are dropped once the upload completes. Deliberately not included, having measured no benefit on this workload: pinning hot experts (neutral below ~24 GB of budget, and progressively worse above it, because pinned pages are not reclaimable and displace the page cache the rest of the model depends on); an explicit LRU/eviction cache with on-demand pread() for non-mmap loads (ties mmap warm, and its budget caps the cache below what mmap gets for free); and speculative prefetch of routed experts (on a working set larger than the cache it roughly doubles bytes read per token). The page cache is already the right policy here; the useful change is only to stop fighting it. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
|
Your proposed new argument explains, "let each expert fault in on demand the first time it is routed to." By default, models are mapped into memory, which allows the system to load only the necessary parts of the model as needed. This means --mmap is a default value. Do not use --no-mmap or --direct-io. |
|
@Volunteer-1 Yeah, so this doesn't change the A good question is whether this actually gives a speed benefit compared to the naive |
| // them in the page cache evicts pages that are still needed and thrashes the load. | ||
| // Drop them now -- advisory and page-aligned inward, so a clean re-access simply | ||
| // re-faults from the file and a neighbour's pages are never touched. | ||
| mapping->advise_range(weight->offs, n_size, llama_mmap::ADVICE_DONTNEED); |
|
it seems like repacked expert weight do not have a backing file and can not be evicted |
Did you check? |
|
I tried this on Linux with a RTX 3090, host RAM ring-fenced with cgroup to 56GB, and size in host RAM (reported by llamacpp) of 69GB; in other words available RAM is about 80% of what's necessary to stop disk I/O. The flag is giving me zero benefit compared to plain mmap (measured with llama-benchy running wikitext). hf = unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ2_XXS
ctx-size = 262144
jinja = true
flash-attn = on
parallel = 1
kv-unified = true
ngl = 99
n-cpu-moe = 37 # out of 43
no-warmup = 1
; lazy-experts = 1memory report: without lazy-experts: with lazy-experts: |
…gml-org#25294 ggml-org#26003 ggml-org#26414 ggml-org#27402 ggml-org#20596 ggml-org#22671) and the CPU-kernel cherry-picks Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
since similar logic has been added in #27794 . can changes for advise ADVICE_DONTNEED repacked weight split to a separate fixup pr? so that it can land before this pr merge |
|
This has basically proven to not be worth it, so I'm closing the PR. |
5 participants
Overview
Add an option to only prefetch (via WILLNEED) actually routed tensors instead of all tensors in a MoE.
This allows loading models that are much bigger than the available RAM by streaming the tensors from disk and using the system page cache to manage the active tensors. Benchmarked this to be more efficient than #25932 - system page cache turns out to be more efficient at managing the experts if you give it all the free memory (also the reason why
--n-cpu-moeover--cpu-moeactually hurts rather than helps decode in this approach).Two extra fixes: mark an uploaded mmap tensor as DONTNEED, fix
posix_madviseranges because they require a page-aligned start.Additional information
Ran a UD-IQ1_M quant of GLM-5.2 (220 GB) on a Strix Halo (128 GB unified RAM), achieving 1.4 t/s generation.
Requirements