Documentation: https://alinawazcode.github.io/sindhi-nlp/
Sentiment classification for Sindhi text. Given a Sindhi sentence, it returns negative, neutral, or positive with class probabilities.
The project contains a reproducible training pipeline, a fine-tuned XLM-RoBERTa model, a lightweight TF-IDF baseline, a Python SDK, a command-line tool, and an HTTP API.
Read this before trusting the numbers. The project began with a 150,000-sentence synthetic dataset on which every model scores 0.9998. That figure says nothing about real Sindhi: the same baseline scores 0.60 on human-written sentences. All headline results below come from a separate human-labeled dataset. See Data and Limitations.
Evaluated on 4,420 human-labeled Sindhi sentences (3,094 train / 442 validation / 884 test). The test split was scored once per model, after all choices were fixed.
| Model | Trained on | Validation macro-F1 | Test macro-F1 | Test accuracy |
|---|---|---|---|---|
| TF-IDF + LinearSVC | synthetic data only | 0.6002 | not run | not run |
TF-IDF + LinearSVC (baseline_natural) |
human-labeled sentences | 0.8665 | 0.8695 | 0.8688 |
| XLM-RoBERTa base | human-labeled sentences | 0.8663 | not run | not run |
XLM-RoBERTa base (xlmr_mixed) |
up to 30,000 synthetic + human-labeled (repeated 5x) | 0.8872 | 0.8906 | 0.8903 |
XLM-R scores higher, but the difference from the baseline is not statistically significant on this test set (exact McNemar test, p = 0.116; the baseline alone was wrong on 75 sentences, XLM-R alone on 56). Each accuracy carries a 95% interval of roughly ±2 points (baseline 0.845 to 0.889, XLM-R 0.868 to 0.909). The baseline trains in under a second on a CPU, so it is a reasonable choice when resources are limited.
Per-class scores, confusion matrices, and the paired comparison are written to artifacts/metrics/ when you run the evaluation commands below.
Requires Python 3.11 or newer.
pip install "sindhi-nlp[transformers]" # SDK, CLI, and the XLM-R model
pip install "sindhi-nlp[api]" # add the HTTP API
pip install "sindhi-nlp[plots]" # add confusion-matrix and comparison chartsThe plots extra, sindhi-nlp --version and the evaluate tool below need version 0.1.2 or newer (pip install --upgrade sindhi-nlp).
The first time you use the published model it downloads about 1.1 GB, which takes a few minutes on a slow connection. It is cached afterwards.
To work on the code, clone the repository and install it in editable mode with the test tools:
git clone https://github.com/alinawazcode/sindhi-nlp.git
cd sindhi-nlp
pip install -e ".[transformers,api,dev]"Model weights are not stored in Git. The trained XLM-R model is published on Hugging Face as alinawazmahar/sindhi-sentiment-xlmr and is downloaded and cached automatically the first time you use that name. You can also point at a local model folder (for example artifacts/models/xlmr_mixed) or train your own with the steps in Reproduce the results.
Python
from sindhi_nlp import SentimentPredictor
# a Hugging Face name (downloaded once, then cached) or a local folder
predictor = SentimentPredictor.from_pretrained("alinawazmahar/sindhi-sentiment-xlmr")
result = predictor.predict("اڄ جو ڏينهن تمام سٺو آهي")
print(result.label) # positive
print(result.probabilities) # {'negative': ..., 'neutral': ..., 'positive': ...}
results = predictor.predict_batch(["...", "..."]) # keeps the input orderInput is cleaned the same way as during training. Empty or non-text input raises InvalidInputError (pass errors="none" to predict_batch to get None for bad items instead), and inputs beyond the model's token limit are truncated.
Command line
sindhi-nlp predict "اڄ جو ڏينهن تمام سٺو آهي" --model-dir alinawazmahar/sindhi-sentiment-xlmr
sindhi-nlp batch --input texts.csv --output results.csv --model-dir artifacts/models/xlmr_mixed
sindhi-nlp info --model-dir artifacts/models/xlmr_mixed
sindhi-nlp --versionMeasure it on your own labeled data
python -m sindhi_nlp.evaluation.evaluate --model-dir alinawazmahar/sindhi-sentiment-xlmr \
--csv my_labeled.csv --text-column Text --label-column Label --show-errors 10HTTP API
uvicorn sindhi_nlp.api.main:app --host 127.0.0.1 --port 8000The model named in configs/inference.yaml is loaded at startup. Open http://127.0.0.1:8000/docs for interactive documentation, or see docs/api.md for endpoints, error formats, and examples.
About
confidence. XLM-R's probabilities are overconfident: wrong predictions can also come back near 1.0. Treat the value as a ranking score, not a calibrated probability.
Full guides live on the documentation site:
- Installation and Quickstart
- Complete examples: runnable scripts for text, CSV files, bad input, the API and evaluation (also in the
examples/folder) - Python SDK, Command line, HTTP API
- Tutorials: classify a CSV, evaluate on your own data, train on your own data
- Results, model card, FAQ, roadmap and known gaps
Two datasets are used. Details are in docs/dataset.md.
- Human-labeled dataset (4,420 sentences). Collected from Awaz and other articles and newspapers and labeled by one annotator (the author). Used for all reported results. It is published separately on Kaggle under CC BY-SA 4.0: https://www.kaggle.com/datasets/alinawaz06/sindhi-sentiment-analysis-dataset.
- Synthetic dataset (150,000 sentences). Machine-generated from sentence templates with a vocabulary of about 2,400 words. It is useful for testing the pipeline but is not representative of real Sindhi. It is not included in this repository.
With the data in place (data/raw/ for the synthetic CSV, data/external/natural_sindhi.csv for the human-labeled one):
# 1. Build the splits
python -m sindhi_nlp.data.split_data --config configs/base.yaml
python -m sindhi_nlp.data.prepare_external
# 2. Baselines (seconds on a CPU)
python -m sindhi_nlp.training.baseline
python -m sindhi_nlp.training.baseline --data-dir data/external/processed --model-name baseline_natural
# 3. XLM-RoBERTa (needs a GPU; a free Kaggle T4 works)
python -m sindhi_nlp.training.train --config configs/xlmr_mixed.yaml --smoke # quick check of the whole pipeline
python -m sindhi_nlp.training.train --config configs/xlmr_mixed.yaml # the published model's settings (3 epochs)
# 4. Evaluate. Validation can be repeated; the test split is a one-time evaluation.
python -m sindhi_nlp.evaluation.evaluate_external --model-dir artifacts/models/xlmr_mixed --split validation
python -m sindhi_nlp.evaluation.evaluate_external --model-dir artifacts/models/xlmr_mixed --split test --final
python -m sindhi_nlp.evaluation.compare_models --a baseline_natural --b xlmr_mixed --split testEvery run is seeded (42) and writes its configuration, metrics, and data hashes next to the model, so results can be checked later. See docs/architecture.md for how the pieces fit together.
configs/ YAML settings for data, training, evaluation, and inference
src/sindhi_nlp/
data/ loading, cleaning, validation, splitting
training/ TF-IDF baseline, XLM-R fine-tuning, metrics
evaluation/ scoring saved models, paired comparison
inference/ SentimentPredictor, preprocessing, result types
api/ FastAPI service
cli/ the sindhi-nlp command
utils/ config, logging, seeding
tests/ pytest suite
docs/ documentation site (MkDocs): guides, tutorials, reference, results
- One small evaluation set. 884 test sentences from a single source, labeled by one person. Inter-annotator agreement was not measured, and "neutral" has only a working definition (normal, everyday conversation), not a written guideline.
- Short, news-style sentences. Performance on social media, long documents, code-mixed text, other Sindhi scripts, dialects, or sarcasm is untested.
- The synthetic data adds little. Adding it to training raised validation macro-F1 from 0.866 to 0.887, a gap too small to treat as proven.
- Overconfident probabilities for the XLM-R model, as noted above.
- Not for decisions about people (employment, credit, moderation, legal outcomes) without human review.
See docs/model_card.md for the full model card.
pytest # tests
ruff check . # lint
ruff format --check .
mypy # type check
mkdocs serve # preview the documentation (pip install -e ".[docs]")See CONTRIBUTING.md before opening a pull request.
The same checks run on every push through GitHub Actions.
The code and the trained model weights are released under the MIT License (see LICENSE). The human-labeled dataset is distributed separately under CC BY-SA 4.0.
@software{nawaz2026sindhinlp,
author = {Nawaz, Ali},
title = {sindhi-nlp: Sentiment classification for Sindhi text},
year = {2026},
url = {https://github.com/alinawazcode/sindhi-nlp},
version = {0.1.2},
doi = {10.5281/zenodo.23832590}
}Ali Nawaz, alinawaz.code@gmail.com