CIFAR-10H — Modeling Human Annotator Disagreement

A ResNet-18 trained to predict the distribution of human annotations on CIFAR-10 images, rather than a single hard label. Uses the CIFAR-10H dataset which provides per-image soft labels derived from crowdsourced annotations.

---

What this project does

Standard image classifiers output a single predicted class. This project trains a model to output a full probability distribution over all ten CIFAR-10 classes that matches how human annotators actually responded to each image — including their disagreements. Three loss functions are compared: KL divergence, cross-entropy with soft labels, and a custom entropy-weighted KL loss.

---

Project structure

DNN/
├── scripts/
│   ├── prepare_data.py    # download and process CIFAR-10H into .npy arrays
│   ├── analyze_data.py    # EDA plots (entropy histogram, class entropy, confusion matrix)
│   ├── train_full.py      # pre-train on CIFAR-10, then fine-tune on CIFAR-10H (all 3 losses)
│   ├── eval.py            # evaluate checkpoints on the held-out test set
│   ├── robustness.py      # evaluate under noise and blur corruptions
│   ├── gradcam.py         # Grad-CAM visualisations
│   └── convert.py         # utility to convert checkpoint formats if needed
├── data/
│   ├── raw/               # CIFAR-10 files + cifar10h-probs.npy go here
│   └── processed/         # generated by prepare_data.py
├── checkpoints/           # saved model weights
├── outputs/               # plots and JSON results
├── logs/                  # training logs
├── report.tex             # LaTeX report
└── report.md              # original markdown notes

---

Requirements

Python 3.9+ and a CUDA-capable GPU are recommended. CPU training works but Stage 1 pre-training will be slow.

Install dependencies:

pip install torch torchvision numpy scipy matplotlib seaborn tqdm opencv-python

Tested with PyTorch 2.x. No specific version pinning needed beyond that.

---

Setup

Process the data

cd d:\Coding\MU\DNN
python scripts/prepare_data.py

This produces final_images.npy, final_probs.npy, and final_entropy.npy in data/processed/. It is safe to re-run — progress is checkpointed in data/processed/prepare_state.json so completed steps are skipped.

---

Running

All scripts are run from the project root (d:\Coding\MU\DNN), not from inside scripts/.

EDA plots

python scripts/analyze_data.py

Saves entropy histogram, per-class entropy bar chart, human confusion matrix, and low/high entropy image grids to outputs/.

Training

python scripts/train_full.py

This runs in two stages:

1. Pre-trains on CIFAR-10 for 30 epochs (skipped if checkpoints/pretrained.pt already exists)
2. Fine-tunes three separate models — one per loss function (kl, ce, kl_entropy) — for up to 50 epochs each with early stopping

Checkpoints are saved to checkpoints/ as kl.pt, ce.pt, and kl_entropy.pt. Logs go to logs/train_all.log.

Evaluation

python scripts/eval.py

Evaluates all three checkpoints on the held-out test set (last 2,000 images). Saves per-model JSON results to outputs/ and a combined outputs/summary.json.

Robustness

python scripts/robustness.py

Runs the KL+Entropy model on the test set under Gaussian noise ($\sigma = 0.1$ and $0.2$) and Gaussian blur. Results saved to outputs/robustness.json.

Grad-CAM

python scripts/gradcam.py

Saves activation overlay images to outputs/gradcam/ for five selected test images.

---

Expected results

After training and evaluation, outputs/summary.json should contain something close to:

Model	Accuracy	Mean KL	Mean JS	Cosine
KL	0.933	0.230	0.048	0.945
CE	0.917	0.264	0.053	0.936
KL+Entropy	0.924	0.241	0.051	0.940

Exact numbers may vary slightly depending on GPU and PyTorch version due to non-determinism in CUDA operations. The random seed is set to 42 in train_full.py but this only controls CPU-side randomness.

---

Notes

• The data split is 6000/2000/2000 (train/val/test). The val split is used only for early stopping during fine-tuning. The test split ([8000:]) is never seen during training.
• The model architecture is ResNet-18 with the first conv layer changed to $3\times3$/stride-1 and the max-pool removed. This is necessary because CIFAR-10 images are $32\times32$ and the standard ResNet-18 design would over-downsample them.
• On Windows, prepare_data.py uses a multiprocessing pool for entropy computation but falls back to single-threaded if that fails. The if __name__ == '__main__': guard is required for this to work correctly.
• Training logs are appended, not overwritten. Delete logs/train_all.log if you want a clean run.