Just me and my 3090s against the world. Local inference on consumer hardware.
I run large MoE models on a box that shouldn't be able to hold them — 2x RTX 3090 (48GB VRAM) and 192GB of DDR4, with most of the expert layers offloaded to host memory. I post what worked, what didn't, and the numbers behind both.
Qwen3.8-Flash-Next on 2x3090 + DDR4 — 17 → 41 t/s decode through expert-cache offload, UD-Q4_K_XL, and MTP speculative decoding. Still tuning; prefill is the open problem. Writeups: Part 1 · Part 2
llama.cpp #28223 (open) — keeps an explicit -ot host buffer override from being silently downgraded to a plain CPU buffer under mmap. Prefill went from 166 → 379 t/s on my setup with 40 expert layers on the host.
kv-sparsity-profiler · Python · MIT
Can query-selected KV paging fit a 262k context on a single 24GB GPU? Measured on Qwen3.6-27B: no. Even a perfect oracle selector falls far short of the attention mass needed. Harness plus the negative result, so nobody else has to run it.
bitrebuttal · JavaScript
Downloads that finish. A supervisor around aria2c for huge model files — survives stalls, expired links, and reboots without losing progress. SHA256-verified. Moved ~200GB with it.
lifeos · Python · AGPL-3.0
Self-hosted personal organiser you talk to. Whisper → local LLM → SQLite → FastAPI → React PWA. Nothing leaves your machine. Runs end to end on 12GB of VRAM.
Dual Xeon E5-2696 v4 · 2x RTX 3090 · 192GB DDR4-2133 ECC · Ubuntu Server 24.04
Serving via llama.cpp and vLLM.