Working Locally with Open Weight LLMs

In this course, we will concentrate on decoder models, also called GPT-like models. These models can be used for completing text, or answering questions (in the style of ChatGPT). As this use case is much more popular, we will focus on this.

Presentation

The slides contain additional background and theroretical information.

Notebooks

You can either try to run the notebooks directly or try to follow how I run them and use it as a documentation (or run it later).

Transformers

We will start with the transformers library as this is straightforward. You will see later, that this is not the optimal choice, though.

• 11-qwen3.5-9b.ipynb: Running an small Qwen3.5 model
• 12-gemma-4-12b.ipynb: Running Google's gemma-4-12B

vLLM

A better choice is often to use vllm as an execution engine and put this into a separate process. The access is then accomplished via the Open AI API.

• 21-vllm-gemma-4-12b.ipynb: gemma-4-12B runs in vllm
• 22-vllm-gemma-4-12b-qat-w4a16-ct.ipynb: a quantized version of gemma-4-12B runs in vllm

SGlang

sglang is another server which also allows running LLMs in a separate process. You will see that sglang can be quite a bite faster than vllm.

• 31-sglang-gemma-4-12b.ipynb: gemma-4-12B runs in sglang
• 32-sglang-gemma-4-12B-it-qat-w4a16-ct.ipynb: quantization speeds things up in sglang much faster
• 33-sglang-gemma-4-12B-it-qat-w4a16-ct-mtp.ipynb: multi-token prediction adds even more speed

tabbyAPI

tabbyapi is yet another execution environment which offers to run models in the very efficient exl3 quantization. Take a look at the speeds and the efficient memory usage!

• 41-tabbyapi-gemma-4-12B-it-exl3.ipynb: gemma-4-12B in exl3 4-bit quantization
• 42-qwen3.6-27B-exl3.ipynb: a much larger qwen3.6-27B in exl3 4-bit quantization

llama.cpp

llama.cpp was originally conceived as a CPU-only enviroment for LLMs. However, it has gained very broad hardware support and is the speed king for single users.

• 51-llama.cpp-gemma-4-12B-it-q4.ipynb: gemma-4-12b in 4-bit quantization with MTP
• 52-llama.cpp-qwen3.6-27B-q4.ipynb: same for qwen3.6-27B
• 53-llama.cpp-qwen3.6-35B-A3B-q4.ipynb: even more speed with MoE qwen3.6-35B-A3B

Running LLMs on the CPU

For this, a good starting point is LM Studio which is available for all major platforms. However, it is not Open Source Software.

An alternative (but also not completely open) is ollama.

If you want to run free software, llama.cpp is recommended. It is a very active project with multiple releases per day and supports many current models. llama.cpp can also run as server and is compatible to the OpenAI API.

Creating your own GGUFs

The easiest way is to download GGUFs, which are widely available on Hugging Face.

If a GGUF is not available, you can also create it there.

Using llama.cpp, you can convert Hugging Face repositories (download first via huggingface-cli download --local-dir Qwen3.5-9B Qwen/Qwen3.5-9B) to GGUF:

python convert_hf_to_gguf.py --outfile Qwen3.5-9B.gguf Qwen3.5-9B

Afterwards, you can quantize them:

$ build/bin/llama-quantize Qwen3.5-9B.gguf Qwen3.5-9B-Q4_K_M.gguf q4_k_m

Finally, run a llama-server to access the frontend:

$ build/bin/llama-server -m Qwen3.5-9B-Q4_K_M.gguf --port 8080

Trying models without local hardware

On Hugging Face, some models are available for inference.

You can also try OpenRouter, some of the models are free, for some you have to pay (very little). Many new models are available there.