30 tok/s for 20B MoE on 8 GB VRAM. Flat throughput to 32K context. Native MXFP4 + GGUF Q4_K/Q5_K/Q6_K via ggml CUDA kernels — zero dequant. Expert offloading for models that don't fit in GPU memory.
Adaptive hot/cold MoE expert cache for llama.cpp: keep the hottest experts in VRAM to speed up decode when offloading to RAM on a consumer GPU. Measured +26% (Qwen3-Coder-Next 80B) and +31.6% (gpt-oss-120B) on one RTX 5090.
Build your own AI companion. GPU-native avatar generation, Substance-Painter cognitive layers, holographic progressive memory, iDream 3D world engine, three-model cognitive hierarchy. Runs entirely on consumer GPUs. No cloud.
Run a 397B MoE model on a MacBook with C/Metal inference, 4.4+ tok/s, and tool calling
Add a description, image, and links to the expert-offloading topic page so that developers can more easily learn about it.
To associate your repository with the expert-offloading topic, visit your repo's landing page and select "manage topics."