Thinference

Thinference runs a small quantized language model locally from the terminal on modest hardware. It uses a CPU-only llama.cpp build and one quantized GGUF model already present on disk. No API key, cloud inference, server, Docker, or web UI is required

Status

Pre-release. Four commands are implemented:

• thinference doctor - validates a local setup
• thinference ask - runs exactly one non-interactive prompt through it
• thinference chat - starts one interactive conversation in it
• thinference benchmark - runs one small fixed CPU-only benchmark on it

Transcript persistence and file input are not implemented

Feasibility and the first measured local run are documented in docs/FEASIBILITY.md and docs/FIRST_RUN.md

Verified target

Thinference has been exercised on one configuration. Support for other machines and operating systems has not been verified

Item	Verified value
OS	macOS 12.7.6 (Monterey)
CPU	Intel Core i7-6820HQ, x86_64, 4 physical / 8 logical cores
RAM	16 GiB
Runtime	llama.cpp built from source, CPU-only, Accelerate BLAS, no Metal
Runtime commit	a7a6d0d
Model	Qwen3-4B-Instruct-2507-Q4_K_M.gguf (2,497,281,120 bytes)
Python	3.8.15 (standard library only)

Official prebuilt llama.cpp macOS x64 binaries require macOS 13.3 and do not run on this machine, so the runtime is built from source and is never installed system-wide

Usage

Run directly from a checkout, without installing anything

ask, chat, and benchmark run the doctor checks as a mandatory preflight first. If any mandatory check fails, no model is loaded and no runtime is started; only the failing checks are printed to standard error

doctor - validate the setup

PYTHONPATH=src python3 -m thinference doctor --config thinference.local.json

Example output:

Thinference doctor

[OK] Config: thinference.local.json
[OK] llama-cli: executable
[OK] Runtime version: 1 (a7a6d0d)
[OK] Runtime commit: a7a6d0d
[OK] llama-bench: executable
[OK] Model file: present
[OK] Model size: 2497281120 bytes
[OK] GGUF header: valid

Status: HEALTHY

Mandatory checks - any failure makes the result UNHEALTHY:

1. the configuration file exists, is valid JSON, and has the required fields and types
2. the configured llama_cli exists, is a regular file, and is executable
3. the runtime version banner is readable and contains the configured expected_commit
4. the model file exists, is a regular file, and is readable
5. the model's size matches expected_bytes exactly; a smaller file is reported as likely partial, and a larger one as an unexpected artifact
6. the model begins with the GGUF magic bytes

Optional check - reported as a warning and never as a failure:

• whether the configured llama_bench exists and is executable

doctor deliberately does not load the model, run inference, run a benchmark, download anything, or search PATH for a runtime. The only subprocess it starts is llama-cli --version, with a bounded timeout

ask - run one prompt

The prompt is given either as a positional argument or on standard input. Exactly one of the two must supply it

PYTHONPATH=src python3 -m thinference ask --config thinference.local.json \
  "Explain in one sentence what the thinference doctor command checks."

echo "В одном предложении объясни, что проверяет команда thinference doctor." \
  | PYTHONPATH=src python3 -m thinference ask --config thinference.local.json

ask resolves the prompt, runs the shared preflight, and starts the configured llama-cli once without a shell. The prompt is passed as one argument, preserving quoting and Unicode. Runtime output is streamed directly, and the child exit status is returned. There is no conversation history or interactive prompt

How standard input is treated:

• With a positional prompt, ask inspects only input that is already available and never waits for end-of-file
• Without a positional prompt, standard input is the requested source and is read to end-of-file
• A positional prompt plus already available non-whitespace standard input is rejected as ambiguous
• With no positional or piped prompt, the command fails immediately instead of reading from the terminal

The runtime is invoked with -ngl 0, so generation stays on the CPU, and is given at most inference.timeout_seconds. On timeout the child is terminated, then killed if it does not stop, and the command exits 124. Ctrl-C stops the child the same way and exits 130

chat - one interactive conversation

PYTHONPATH=src python3 -m thinference chat --config thinference.local.json

chat requires a real interactive terminal on both standard input and standard output. If either is redirected, the command prints a usage error, exits 2, and does not load the model. Use thinference ask for piped, redirected, or scripted input

After the preflight, chat starts the configured llama-cli once in native conversation mode (-cnv). It supplies no prompt, system prompt, or custom template, so the GGUF's embedded chat template is used. Standard input, output, and error remain attached to the terminal. Thinference does not manage conversation history, parse the exchange, or write a transcript to disk

Ctrl-C stops the session: the runtime is terminated, reaped, and the command exits 130

benchmark - one fixed measurement

PYTHONPATH=src python3 -m thinference benchmark --config thinference.local.json

The workload is fixed and deliberately not configurable:

• pp128 - prompt processing of 128 tokens
• tg32 - generation of 32 tokens
• one repetition
• inference.threads and inference.batch from the configuration
• CPU only (--n-gpu-layers 0), one model, no sweep and no automatic tuning

benchmark additionally requires the configured runtime.llama_bench to exist and be executable. When it is missing, the command prints one error, does not load the model, and exits 1. For doctor that same check stays a warning

The tool is started once, as an argument list with no shell and no PATH lookup, with one value per parameter so exactly two tests are produced. The native llama-bench result table is the result; it is not reparsed, stored, or charted. Its output is captured, the configured absolute paths are replaced with plain file names, and only then is it printed

Example output on the verified target:

Thinference benchmark
Model: Qwen3-4B-Instruct-2507-Q4_K_M.gguf
Workload: pp128, tg32
Threads: 4
Batch: 512
Repetitions: 1

| model                  |     size |   params | backend | threads | n_batch |  test |          t/s |
| ---------------------- | -------: | -------: | ------- | ------: | ------: | ----: | -----------: |
| qwen3 4B Q4_K - Medium | 2.32 GiB |   4.02 B | BLAS    |       4 |     512 | pp128 | 33.08 ± 0.00 |
| qwen3 4B Q4_K - Medium | 2.32 GiB |   4.02 B | BLAS    |       4 |     512 |  tg32 |  6.69 ± 0.00 |

build: a7a6d0d (1)

± 0.00 reflects the single repetition; it is not a precision claim. One sample on one machine describes that machine at that moment and nothing else

The run is bounded by inference.timeout_seconds; on timeout the child is terminated, then killed if it does not stop, and the command exits 1. Nothing is written to disk, and no benchmark history is kept

Exit codes

Shared by all four commands:

Code	Meaning
0	success - all mandatory checks passed, and the run completed
1	checks failed, preflight failed, the benchmark tool was unusable, the benchmark timed out, or the runtime exited non-zero
2	usage error, missing configuration, malformed JSON, invalid schema, a missing/empty/ambiguous prompt, or chat without a terminal
124	the runtime exceeded inference.timeout_seconds and was terminated (ask only)
130	ask, chat, or benchmark was interrupted with Ctrl-C

Configuration

Copy thinference.example.json to thinference.local.json and adjust it. The local file is git-ignored to keep machine-specific paths out of normal commits

{
  "runtime": {
    "llama_cli": ".local/llama.cpp/build/bin/llama-cli",
    "llama_bench": ".local/llama.cpp/build/bin/llama-bench",
    "expected_commit": "a7a6d0d"
  },
  "model": {
    "path": "models/Qwen3-4B-Instruct-2507-Q4_K_M.gguf",
    "expected_bytes": 2497281120
  },
  "inference": {
    "context": 4096,
    "threads": 4,
    "batch": 512,
    "n_predict": 128,
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "timeout_seconds": 300
  }
}

• Paths may be relative; they are resolved against the directory containing the configuration file
• runtime.llama_bench is optional for doctor, ask, and chat, and required by benchmark
• expected_commit must be a 7- to 40-character hexadecimal Git SHA prefix; matching is case-insensitive
• The inference section is optional for doctor and required for ask, chat, and benchmark. When present it is fully validated by every command. chat reuses the same settings, except timeout_seconds, which is not applied to an interactive session. benchmark uses only threads, batch, and timeout_seconds; its token counts are fixed
• The sampling and sizing values above are the ones measured in docs/FIRST_RUN.md on the verified target
• JSON is used instead of TOML so that no parser dependency is needed on Python versions older than 3.11

Verified results

A Qwen3 4B Q4_K_M model was used successfully through Thinference on an Intel MacBook with 16 GiB of RAM, CPU-only. The real ask, chat, and benchmark command paths were each exercised against the actual runtime and model

The fixed Thinference-managed benchmark measured:

Test	Result
pp128	33.08 t/s
tg32	6.69 t/s

Sanitized evidence, including the exact prompts, verbatim responses, exit codes, and limitations, is in results/intel-macbook-2016/. That package also retains an earlier direct llama.cpp baseline, kept separate, to show the run-to-run variability that was actually observed

This evidence covers one tested environment. It is not a general support or performance claim

Development and CI

Native development

Everything runs from the checkout with the standard library only. The Makefile targets are wrappers around these commands, not a separate build system

Task	Command	Make target
Run the tests	PYTHONPATH=src python3 -m unittest discover -s tests -v	make test
Install the package	python3 -m pip install .	make install
Install for editing	python3 -m pip install -e .	make install-editable
Build a wheel	python3 -m pip wheel . --no-deps --wheel-dir dist	make package
Validate the setup	PYTHONPATH=src python3 -m thinference doctor --config …	make doctor
Run one prompt	PYTHONPATH=src python3 -m thinference ask --config … "…"	make ask PROMPT='…'
Start a conversation	PYTHONPATH=src python3 -m thinference chat --config …	make chat
Run the benchmark	PYTHONPATH=src python3 -m thinference benchmark --config …	make benchmark

make with no target prints the same list. The interpreter is PYTHON (python3 by default) and the configuration file is CONFIG, which defaults to thinference.local.json:

make doctor CONFIG=/path/to/another.json

doctor, ask, chat, and benchmark still need a native llama.cpp build and a real GGUF model on the host. The Make targets do not provide either

Docker

The image is a development and CI convenience, nothing more:

make docker-build
make docker-test

It installs the package on a clean Debian-based Python image and runs the same unit suite, which uses fake runtimes and tiny fake GGUF files. It contains no llama.cpp build, no model, and no local configuration, so it cannot run inference, and containerized inference is neither verified nor supported. The host remains the only tested way to run the four commands

Continuous integration

.github/workflows/ci.yml runs on every push and pull request and checks only what a machine without a model can check:

• the unit suite on Python 3.8 and on Python 3.11
• installing the package with pip
• that the installed thinference command answers --help
• that the development image builds and that its unit suite passes

Python 3.8 is tested on the newest runner image for which actions/setup-python still publishes it. Nothing is published, and no real model, inference, or performance result is validated in CI

Limitations

• Only one machine, one runtime build, and one model have been tested
• doctor checks the GGUF magic bytes only; it does not parse GGUF metadata, so it cannot confirm the architecture or quantization of the file
• A correct byte size does not prove the file's contents are uncorrupted; no checksum is verified yet
• Available RAM, CPU features, and process memory use are not inspected
• The runtime and model are located by explicit configuration only; there is no discovery, no download, and no build support
• ask handles one prompt per process and keeps no history between runs
• chat keeps a conversation only for as long as the runtime process lives. Nothing is saved, so there is no transcript persistence, no way to resume a past session, and no file or repository input
• chat has been exercised against the real model on the verified target once, with a single short Russian turn (see docs/FIRST_RUN.md). Longer sessions, context exhaustion, and multi-turn behaviour are still unmeasured. inference.n_predict caps every reply, so a long answer can be cut off mid-sentence
• The runtime's own output is passed through unchanged; Thinference does not parse, reformat, or post-process the generated text
• benchmark measures one fixed workload with one repetition on one model. It records one machine state at one moment, not a controlled comparison across runs, models, thread counts, context sizes, quantizations, or machines, and it does not predict the speed of a real conversation. The result is printed and then forgotten; there is no stored history, result file, or chart
• Generation on the verified target is slow: the fixed benchmark measured 6.69 t/s for tg32, and the one real ask run reported 6.3 t/s, so a long n_predict takes minutes
• The package is not installed during development; the documented invocation uses PYTHONPATH=src

License

Thinference is licensed under the Apache License, Version 2.0. The full text is in LICENSE.