A local AI stack manager for developers: one CLI to download GGUF models, run llama.cpp, and integrate with your favorite AI editor.
yllama models add unsloth/Qwen3.6-35B-A3B-GGUF
yllama models download qwen3.6-35b-a3b
yllama serve -ngl 35 -c 8192
| Workflow | Before yllama | With yllama |
|---|---|---|
| Get a model | Search HuggingFace → copy URL → download manually | yllama models add <url> |
| Start inference | Spawn server, manage PID, health-check | yllama serve |
| Editor integration | Manually sync host/port in every config | yllama vibe (auto-starts + syncs) |
| LiteLLM proxy | Write YAML by hand | yllama litellm |
yllama delegates inference to llama.cpp. Editor integrations require additional tools.
macOS
brew install llama-cppLinux
# Debian / Ubuntu
sudo apt install llama.cpp
# Fedora
sudo dnf install llama.cppVerify: llama-server --help should display usage info.
npm install -g @anthropic-ai/claude-codeLinux: If npm installs to a custom prefix, add it to your PATH:
export PATH="$(npm config get prefix)/bin:$PATH"Verify: claude --version should show the installed version.
Install Vibe from the official source. After installation, ensure the binary is on your PATH (create a symlink if needed):
ln -s "/Applications/Vibe.app/Contents/MacOS/Vibe" ~/.local/bin/vibe# Register, download, and go
yllama models add <hf-url>
yllama models download <name>
yllama serve # auto-selects your modelOr skip the two-step add/download with --download:
yllama models add <hf-url> --download
yllama serveyllama models list # show cached models
yllama models add <url> # register a GGUF model (HF repo URLs, owner/repo, or search)
yllama models add <url> --download # register and download in one step
yllama models add <path.gguf> # register a GGUF already on disk (symlinked)
yllama models add <path.gguf> --copy # ...copied into ~/.yllama/models instead
yllama models download <name> # download with progress bar
yllama models delete <name> # remove from disk and registry
Point models add at a .gguf you already have and it is registered without
any download:
yllama models add ~/models/Qwen3-30B-Q4_K_M.gguf # --link (default)
yllama models add ~/models/Qwen3-30B-Q4_K_M.gguf --copy # duplicate it| Flag | What lands in ~/.yllama/models |
Notes |
|---|---|---|
--link (default) |
a symlink | no extra disk; breaks if you move the original |
--copy |
a real copy | self-contained; instant and free on APFS/Btrfs (copy-on-write) |
Either way the original file is never moved, and models delete only removes
what's inside ~/.yllama/models. GGUF metadata is scanned locally, so built-in
MTP heads are still detected and enabled at serve time. models list shows
local: <path> as the source, and flags an entry as missing if the file (or
a symlink's target) is gone.
yllama start [model] [-- <flags>] # foreground (debug-friendly)
yllama serve [model] [-- <flags>] # background daemon
yllama stop # kill the running server
yllama attach # tail the server log
Extra flags are forwarded verbatim to llama-server (-ngl, -c, --temp, etc.).
yllama vibe [folder] [-- <args>] # launch Vibe (auto-starts server, syncs config)
yllama claude [folder] [-- <args>] # launch Claude Code with local llama.cpp
yllama sync # sync running server into ~/.vibe/config.toml
yllama litellm [--output <path>] # generate LiteLLM proxy YAML
yllama install [--dir PATH] # install the binary to PATH
- 0 models — error with a helpful message
- 1 model — used automatically
- default set — used automatically (no picker)
- 2+ models, no default — interactive TUI picker
| File | Contents |
|---|---|
~/.yllama/config.toml |
server host, port, binary path |
~/.yllama/models/manifest.json |
registered models, download status, extra flags |
cargo install --path .
yllama install --dir ~/.local/bin # or any PATH directoryMIT