🌐 EN · SR · HR · BS · MK · SL · SQ · CNR · BG · EL · TR · RO · HU
An open benchmark for understanding the languages of the Balkans & Southeast Europe — not speed.
Most LLM benchmarks measure efficiency and English-centric skills. BalkanBench measures something else: whether a model truly understands these languages — ambiguity, homonyms, script, dialect, culture — and whether it is honest (admits what it doesn't know) instead of inventing facts.
Flagship example: „Горе горе горе горе него доле." — four times gore, four meanings: up · forests/mountains · [they] burn · worse. → "Up above, the forests burn worse than below." A model that resolves this has understood the language, not just translated it.
It runs 13 axes × 12 languages = 156 deterministic tests per model. No API keys, no LLM-judge — scoring is script/keyword/number based, so anyone can reproduce it.
Serbian (sr), Croatian (hr), Bosnian (bs), Macedonian (mk), Slovenian (sl), Albanian (sq), Montenegrin (cnr), Bulgarian (bg), Greek (el), Turkish (tr), Romanian (ro), Hungarian (hu).
Script awareness is built in: Azbuka (Cyrillic ≠ "just Cyrillic"), Latinica, and Greek are checked per answer — a model answering Macedonian in Latin script loses the script point.
| Axis | What it measures |
|---|---|
| FACT | Real knowledge of Balkan history, culture, geography |
| HALLU | Invented person/work/event → does it admit "I don't know" instead of fabricating? |
| DETAIL | Real entity + a fabricated detail → does it flag the uncertainty? |
| GRADED | Graded honesty: show partial knowledge (the real entity) and decline the invented detail |
| TEACH | Teaching/explaining a topic in the target language |
| REASON | Everyday reasoning and inference |
| LOGIC | Short math/logic — is the final result correct? |
| LOGIC2 | A second logic item (robustness) |
| ANALYSIS | Longer analysis — correct result and a visible step-by-step path |
| INSTRUCT | Following formatting / instruction constraints |
| LONGFORM | Coherent long-form text, in-language |
| SEARCH | Tool use: when unsure, does it call web_search? |
| TOOLBASE | Tool discrimination: for basic knowledge, answer directly (do not call a tool) |
Every answer is also checked for in-language / correct-script output — thinking and answering in the target language, not pivoting through English.
Each axis uses a fixed check — no subjective judging:
name/num— the correct fact/number appears in the answeridk— an honesty marker ("I don't know / can't confirm", per language) appearsgraded— the real entity and an honesty marker both appearscript/lang— output is in the expected script and languagecalc/calc_long— the final number matches (long form also requires intermediate steps)search— the model triggered theweb_searchtool;notool— it answered without one
Total: /156. Fully reproducible.
BalkanBench talks to any model served by Ollama.
# 1) Serve your model(s) with Ollama
ollama pull olivilo/zora # or your own model, e.g. ollama pull gemma2:9b
# 2) Run the benchmark (Ollama endpoint defaults to localhost:11434)
python3 matrix_ollama.py --models "olivilo/zora,gemma2:9b,qwen2.5:7b"Output: a per-axis score table for each model, /156, plus per-answer detail.
Compare any model — your fine-tune, a base model, a competitor — on the same 156 tasks.
===== olivilo/zora =====
FACT 7/12 HALLU 10/12 DETAIL 8/12 GRADED 6/12 TEACH 12/12 ...
>>> olivilo/zora: 84/156
No GPU required for the harness itself — Ollama runs the model, the benchmark just scores.
Zora leads the field, beating models 3–4× its size:
| Model | Size | Score | Note |
|---|---|---|---|
| Zora | 8B | 84 | CPT + SFT on 12 Balkan languages |
| Gemma-4-31B | 31B | 77 | Google's multilingual model |
| Mistral-24B | 24B | 73 | |
| Qwen3.6-30B | 30B | 73 | |
| Salamandra | 7B | 66 | |
| EuroLLM | 9B | 65 | EU-funded multilingual |
| Aya | 8B | 61 | Cohere's multilingual |
| BgGPT | 7B | 56 | Bulgarian-focused |
| YugoGPT | 7B | 35 | Serbian-focused |
Model: huggingface.co/sovasoft/zora
A model for the Balkans must understand these languages from the inside — their double meanings, dialects, scripts and culture — and be honest about its limits. Speed is secondary. Understanding counts.
Open, reproducible, natively cross-checked. Contributions welcome. By Sovasoft (ai.in.rs) with Akademija Ljiljana.
35 cases (was 18) — added 17 language-focused cases for deeper testing:
| New Category | Cases Added |
|---|---|
| Advanced Homonyms | gore-int (3 meanings), vodopad-int (compound words), vis-int (island vs elevation), konj-int (derivatives), rad-int, cena-int |
| Dialect Deep-Dive | pet-int (kajkavski/čakavski/štokavski), makedon-int (MK vs BG), ro-int, hu-int |
| Language Separation | lat-int (script differences), jezik-int (false friends), pisati-int, bel-int (ekavica/ijekavica), brat-int (all Balkan languages) |
| Multilingual Culture | gr-int (Greek), turk-int (Turkish) |
The v1.13 cases focus on language comprehension depth rather than the broader axis-based testing. Both sets are complementary: v1.1 cases test breadth, v1.13 cases test depth.
- FACT: Better factual knowledge via CPT with encyclopedia data
- HALLU: Maintained 6/6 (no fabrication of fake entities)
- LOGIC: Improved cat/mouse puzzle (classic LLM trap)
- ANALYSIS: Step-by-step reasoning with visible calculation path
- Homonyms: Deeper polysemy understanding (gore ×4, kosa ×3)
- Sprachtreue: 12/12 language fidelity (was 8/12)
# Serve your model with Ollama
ollama pull olivilo/zora
# Run the 35-case benchmark
python3 matrix_ollama.py --models "olivilo/zora"
# Run the 156-axis benchmark (13 axes × 12 languages)
python3 matrix_ollama.py --models "olivilo/zora,gemma2:9b,qwen2.5:7b"