Skip to content
BranchesTags

Latest commit

 

History

9,991 Commits

Folders and files

NameName
Last commit message
Last commit date
.devops
.devops
 
 
.gemini
.gemini
 
 
.github
.github
 
 
.pi/gg
.pi/gg
 
 
app
app
 
 
benches
benches
 
 
ci
ci
 
 
cmake
cmake
 
 
common
common
 
 
conversion
conversion
 
 
docs
docs
 
 
examples
examples
 
 
ggml
ggml
 
 
gguf-py
gguf-py
 
 
grammars
grammars
 
 
include
include
 
 
licenses
licenses
 
 
media
media
 
 
models
models
 
 
pocs
pocs
 
 
requirements
requirements
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
tools
tools
 
 
vendor
vendor
 
 
.clang-format
.clang-format
 
 
.clang-tidy
.clang-tidy
 
 
.dockerignore
.dockerignore
 
 
.ecrc
.ecrc
 
 
.editorconfig
.editorconfig
 
 
.flake8
.flake8
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
.pre-commit-config.yaml
.pre-commit-config.yaml
 
 
AGENTS.md
AGENTS.md
 
 
AUTHORS
AUTHORS
 
 
CLAUDE.md
CLAUDE.md
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CMakePresets.json
CMakePresets.json
 
 
CODEOWNERS
CODEOWNERS
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 
build-xcframework.sh
build-xcframework.sh
 
 
convert_hf_to_gguf.py
convert_hf_to_gguf.py
 
 
convert_hf_to_gguf_update.py
convert_hf_to_gguf_update.py
 
 
convert_llama_ggml_to_gguf.py
convert_llama_ggml_to_gguf.py
 
 
convert_lora_to_gguf.py
convert_lora_to_gguf.py
 
 
flake.nix
flake.nix
 
 
mypy.ini
mypy.ini
 
 
pyproject.toml
pyproject.toml
 
 
pyrightconfig.json
pyrightconfig.json
 
 
requirements.txt
requirements.txt
 
 
ty.toml
ty.toml
 
 

Repository files navigation

llama.cpp

This is a private fork (llama.cpp-kintsugi). See Why This Fork Exists below.

llama

License: MIT Release Server Docker Winget

Why This Fork Exists

Note on creation: This fork was developed with substantial AI assistance. While the upstream project's stance on AI-generated contributions is fully respected, this fork exists because the maintainer needed a fix on a timeline that didn't permit the upstream contribution process. The fork is private and not submitted as a PR. Every change was reviewed, tested, and understood before deployment.

Problem

Hybrid SSM models using the qwen35moe architecture — notably Ornith-1.0-35B and Qwen3.6-35B-A3B — crash at high context on AMD Strix Halo APUs (gfx1151, Vulkan backend). The crash is ggml_abort in common_context_seq_rm: the recurrent memory subsystem cannot handle partial sequence removal at 88%+ context utilization (184K+ of 262K tokens) because it lacks snapshot planes for token rollback.

The failure manifests during branching agentic conversations, where cache compaction triggers seq_rm to remove partial cells from the recurrent state. Without sufficient n_rs_seq snapshot planes, the removal fails and the server aborts. Upstream issue #22450 describes the identical symptom — slot hang after multi-turn cache invalidation on Qwen3.6-35B-A3B MoE.

Fix

Two classes of changes across 3 files (84 insertions, 23 deletions):

Critical fix — common/common.cpp:1595: Set n_rs_seq = 4 when speculative decoding is not active. This allocates 4 snapshot planes for the recurrent state, enabling seq_rm to handle partial range removal at high context. Non-recurrent architectures are clamped to 0 downstream (llama-context.cpp:55-58), so this has zero impact on dense models.

Checkpoint correctness fixes — server-context.cpp + common/common.h: Eight changes to the checkpoint subsystem for hybrid models:

  • Save a checkpoint at the end of every generation (captures recurrent state S_N)
  • Hybrid-aware checkpoint selection (position-independent search, not SWA-based)
  • llama_synchronize() after state restore (prevents GPU L1/L2 cache staleness on Vulkan)
  • Preserve checkpoints after forced resets (prevents erasure cascade)
  • Enforce minimum --ctx-checkpoints for hybrid models

Testing

Methodology:

  1. Context fill: 611K chars of synthetic Python code loaded as a single user message, reaching ~230K prompt tokens (88% of 262K) in ~900 seconds
  2. Linear follow-up: Send a follow-up question continuing the conversation — verifies cache reuse at high context
  3. Branching follow-up: Send a different follow-up (not a continuation) — triggers cache compaction and exercises the seq_rm partial removal path
  4. Second branch: Another branching question at higher context pressure

Results on Ornith-1.0-35B Q8_0 (36.9 GB), Strix Halo Vulkan:

Test Upstream Kintsugi
Fill to 230K Crashes at 184K (ggml_abort) ✅ Completes
Linear turn at 230K — ✅ 8s, cache reused, 28 t/s
Branch turn 1 — ✅ 8s, no crash
Branch turn 2 — ✅ 8s, no crash

Affected Models

Model Architecture Fixed?
Ornith-1.0-35B qwen35moe ✅
Qwen3.6-35B-A3B qwen35moe ✅
Qwen3.6-35B-A3B-MTP qwen35moe ✅
Gemma 4 (all variants) gemma4 Not affected
GPT-OSS-120B gpt-oss Not affected
All other models — Zero impact

Deployment

# Build (Strix Halo / Vulkan)
cmake -B build-vulkan -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON
make -j16 llama-server

# Run
llama-server \
  --model ornith-1.0-35b-Q8_0.gguf \
  --ctx-size 262144 --n-gpu-layers 99 --parallel 1 \
  --flash-attn on --jinja --ctx-checkpoints 16

--ctx-checkpoints 16 is required for hybrid models (the fork enforces a minimum of 4 if omitted). Generation speed drops from ~52 t/s to ~28 t/s at high context due to attention scaling on 215 GB/s bandwidth — this is a hardware limitation, not a software bug.

Cross-references

  • Fork documentation: ~/Documents/llama-enhance/kintsugi-documentation.md
  • Full investigation: ~/Documents/llama-enhance/KINTSUGI_INVESTIGATION.md

Backporting to Upstream

The fix can be upstreamed as a series of small, independently reviewable PRs:

PR 1: common/common.cpp — n_rs_seq default for hybrid architectures

The simplest and highest-impact change. Sets n_rs_seq = 4 when speculative decoding is not active. The downstream clamp in llama-context.cpp:55-58 already handles architectures that don't support rs_rollback, so this is safe for all models. Justification: the recurrent memory already has the rs_rollback mechanism; it's simply never enabled outside of speculative decoding.

PR 2: common/common.h — is_generation_checkpoint field

Adds a boolean to common_prompt_checkpoint distinguishing generation-end checkpoints from prompt-processing checkpoints. Enables downstream code to prefer generation checkpoints for cross-turn restore.

PR 3: server-context.cpp — checkpoint save at generation end

Adds checkpoint creation at the end of every generation for hybrid models. This ensures a checkpoint exists for the next turn regardless of how many batches the prompt required.

PR 4: server-context.cpp — hybrid-aware checkpoint selection

When the SWA-based checkpoint search triggers, use position-independent selection for hybrid models (prefer generation checkpoints by flag, fall back to any valid checkpoint). Does not change the gate condition (pos_min >= pos_min_thold); only changes the search logic inside it.

PR 5: server-context.cpp — Vulkan pipeline barrier + erasure fix

Two small changes: llama_synchronize() after state restore (prevents GPU stale-cache reads on Vulkan's separate transfer/compute queues), and preserving checkpoints after forced resets for hybrid models.

Each PR is small (~10-30 lines), affects a single file, and can be reviewed independently. Together they eliminate the crash at 88%+ context for hybrid SSM models on bandwidth-constrained hardware.

Manifesto / ggml / ops

LLM inference in C/C++

Recent API changes

Hot topics

Quick start

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine:

Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more.

Example command:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

  • Plain C/C++ implementation without any dependencies
  • Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
  • AVX, AVX2, AVX512 and AMX support for x86 architectures
  • RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
  • 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
  • Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Models

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

Multimodal

Bindings
UIs

(to have a project listed here, it should clearly state that it depends on llama.cpp)

Tools
  • akx/ggify – download PyTorch models from Hugging Face Hub and convert them to GGML
  • akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp
  • crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption
  • gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage
  • Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example)
  • unslothai/unsloth – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0)
Infrastructure
  • Paddler - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure
  • GPUStack - Manage GPU clusters for running LLMs
  • llama_cpp_canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly
  • llama-swap - transparent proxy that adds automatic model switching with llama-server
  • Kalavai - Crowdsource end to end LLM deployment at any scale
  • llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes.
  • LLMKube - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal support"
Games
  • Lucy's Labyrinth - A simple maze game where agents controlled by an AI model will try to trick you.

Supported backends

Backend Target devices
Metal Apple Silicon
BLAS All
BLIS All
SYCL Intel GPU
OpenVINO [In Progress] Intel CPUs, GPUs, and NPUs
MUSA Moore Threads GPU
CUDA Nvidia GPU
HIP AMD GPU
ZenDNN AMD CPU
Vulkan GPU
CANN Ascend NPU
OpenCL Adreno GPU
IBM zDNN IBM Z & LinuxONE
WebGPU All
RPC All
Hexagon [In Progress] Snapdragon
VirtGPU VirtGPU APIR

Obtaining and quantizing models

The Hugging Face platform hosts a number of LLMs compatible with llama.cpp:

You can either manually download the GGUF file or directly use any llama.cpp-compatible models from Hugging Face or other model hosting sites, by using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

By default, the CLI would download from Hugging Face, you can switch to other options with the environment variable MODEL_ENDPOINT. The MODEL_ENDPOINT must point to a Hugging Face compatible API endpoint.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo.

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

To learn more about model quantization, read this documentation

llama-cli

A CLI tool for accessing and experimenting with most of llama.cpp's functionality.

  • Run in conversation mode

    Models with a built-in chat template will automatically activate conversation mode. If this doesn't occur, you can manually enable it by adding -cnv and specifying a suitable chat template with --chat-template NAME

    llama-cli -m model.gguf

# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
  • Run in conversation mode with custom chat template 
    # use the "chatml" template (use -h to see the list of supported templates)
llama-cli -m model.gguf -cnv --chat-template chatml

# use a custom template
llama-cli -m model.gguf -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
  • Constrain the output with a custom grammar 
    llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'

# {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}

    The grammars/ folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.

    For authoring more complex JSON grammars, check out https://grammar.intrinsiclabs.ai/

llama-server

A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

  • Start a local HTTP server with default configuration on port 8080 
    llama-server -m model.gguf --port 8080

# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
  • Support multiple-users and parallel decoding 
    # up to 4 concurrent requests, each with 4096 max context
llama-server -m model.gguf -c 16384 -np 4
  • Enable speculative decoding 
    # the draft.gguf model should be a small variant of the target model.gguf
llama-server -m model.gguf -md draft.gguf
  • Serve an embedding model 
    # use the /embedding endpoint
llama-server -m model.gguf --embedding --pooling cls -ub 8192
  • Serve a reranking model 
    # use the /reranking endpoint
llama-server -m model.gguf --reranking
  • Constrain all outputs with a grammar 
    # custom grammar
llama-server -m model.gguf --grammar-file grammar.gbnf

# JSON
llama-server -m model.gguf --grammar-file grammars/json.gbnf

llama-perplexity

A tool for measuring the perplexity 1 (and other quality metrics) of a model over a given text.

  • Measure the perplexity over a text file 
    llama-perplexity -m model.gguf -f file.txt

# [1]15.2701,[2]5.4007,[3]5.3073,[4]6.2965,[5]5.8940,[6]5.6096,[7]5.7942,[8]4.9297, ...
# Final estimate: PPL = 5.4007 +/- 0.67339
  • Measure KL divergence 
    # TODO

llama-bench

Benchmark the performance of the inference for various parameters.

  • Run default benchmark 
    llama-bench -m model.gguf

# Output:
# | model               |       size |     params | backend    | threads |          test |                  t/s |
# | ------------------- | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         pp512 |      5765.41 ± 20.55 |
# | qwen2 1.5B Q4_0     | 885.97 MiB |     1.54 B | Metal,BLAS |      16 |         tg128 |        197.71 ± 0.81 |
#
# build: 3e0ba0e60 (4229)

llama-simple

A minimal example for implementing apps with llama.cpp. Useful for developers.

  • Basic text completion 
    llama-simple -m model.gguf

# Hello my name is Kaitlyn and I am a 16 year old girl. I am a junior in high school and I am currently taking a class called "The Art of

Contributing

  • Contributors can open PRs
  • Collaborators will be invited based on contributions
  • Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
  • Any help with managing issues, PRs and projects is very appreciated!
  • See good first issues for tasks suitable for first contributions
  • Read the CONTRIBUTING.md for more information
  • Make sure to read this: Inference at the edge
  • A bit of backstory for those who are interested: Changelog podcast

Other documentation

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

XCFramework

The XCFramework is a precompiled version of the library for iOS, visionOS, tvOS, and macOS. It can be used in Swift projects without the need to compile the library from source. For example:

// swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "MyLlamaPackage",
    targets: [
        .executableTarget(
            name: "MyLlamaPackage",
            dependencies: [
                "LlamaFramework"
            ]),
        .binaryTarget(
            name: "LlamaFramework",
            url: "https://github.com/ggml-org/llama.cpp/releases/download/b5046/llama-b5046-xcframework.zip",
            checksum: "c19be78b5f00d8d29a25da41042cb7afa094cbf6280a225abe614b03b20029ab"
        )
    ]
)

The above example is using an intermediate build b5046 of the library. This can be modified to use a different version by changing the URL and checksum.

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • subprocess.h - Single-header process launching solution for C and C++ - Public domain

Footnotes

  1. https://huggingface.co/docs/transformers/perplexity ↩

About

AI generated fixes for Hybrid SSM models using the qwen35moe architecture(Ornith-1.0-35B / Qwen3.6-35B-A3B) on AMD Strix Halo APUs (gfx1151, Vulkan backend).

Resources

Readme
MIT license

Contributing

Contributing

Security policy

Security policy
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages