Servekit is a kit of optimizations for serving large language models (LLMs) with minimal cold-start latency 🥶. It wraps sglang calls and implements some optimizations.
Currently, it supports fast weight loading for lustre storages like CSCS's capstor and iopsstor ⚡.
Install directly from GitHub:
pip install git+https://github.com/eth-easl/servekit.gitOr, for local development (editable install, from this directory):
pip install -e .servekit launch -- python -m sglang.launch_server --model-path <model> ...Prepend servekit launch -- to an engine command to enable servekit's optimizations. Currently, we offer:
- Fast weight loading: loads the weights in a multiprocessing fashion adapted to network storage like Lustre drives (e.g. CSCS
capstorandiopsstor) to RAM (/dev/shm) and then to GPU memory.
servekit profile -- python -m sglang.launch_server --model-path <model> ...
servekit profile -- vllm serve <model> --tensor-parallel-size 4 ...Parses the engine's own log output for phase timings (no engine changes needed) and prints a per-phase duration table once the server is ready.
servekit bench --url http://127.0.0.1:8080 --out bench.json --wait-ready 300Runs a correctness check (greedy completions on fixed prompts) and a fixed
concurrent throughput workload against POST /v1/completions.
- Support Multi-Node fast weight loading
- Support Pipeline parallelism ()
- Support vllm fast weight loading
PYTHONPATH=src python -m pytest tests -q