I make vLLM start faster. I built thaw to snapshot and fork live LLM sessions.
vLLM contributor and Inferact open-source fellow, working with Simon Mo on cold start and model hot-swap. Based in San Jose.
I maintain the cold-start roadmap. Recent merged work:
- Initialized-engine snapshots: experimental same-host Linux x86-64 TP1 commands to create, inspect and restore a prepared engine. An earlier controlled benchmark reduced activation from 54.7 s to 20.4 s (Qwen3-32B BF16, H100 PCIe, warm model and generated caches; medians, n=3 per arm). Separate H200 validation covered create, restore, container restart, a second restore, captured-output checks and cleanup.
- Zstd container images: use zstd for CI images and publish a separate zstd Docker Hub variant while retaining the existing gzip images.
- Pluggable sleep-mode backends, compile-cache invalidation fixes, and memory-profile reuse across boots.
In progress:
- Precompiled Python bytecode: move compilation into the image build so fresh containers do less work before serving.
- Shared CLI declarations: share Python and Pydantic argument declarations across the CLI and runtime.
Snapshot a running LLM session, fork independent continuations from its KV state, and inspect or diff saved sessions on a laptop without a GPU. Rust + CUDA, with vLLM and SGLang integrations.
This is the project that led me into upstream vLLM.
Re-feeding Is Not Replaying · sole-author preprint.
Replaying a transcript can change which tokens a credit-estimation method identifies as important. I measured that against exact KV-state resume and a replica noise floor. Batch-invariant kernels eliminated the discrepancy in the tested configurations.
- Recall: a Rust/Tauri League of Legends overlay that recommends your next purchase from live game state.
- ProjectGorgon: speculative decoding with custom Triton/CUDA kernels. Where I learned GPU programming.
- sentinel: a Go log-streaming engine with LSM storage, a write-ahead log, and replication.
- Madison Bus ETA: arrival predictions with XGBoost and conformal uncertainty intervals. Retired; source available.
M.S. CS at Northeastern, Silicon Valley. B.S. Data Science, UW–Madison.