MNIST MLP Hyperparameter Study

PyTorch MLP study of 243 historical MNIST configurations, with transparent methodology limitations and a held-out test evaluation of the selected model.

What is in this repository

The project compares learning rate, batch size, network depth, layer width, and nominal loss labels for a multilayer perceptron trained on MNIST. The 243 experiment rows in results/summary.txt are preserved from a 2025 university coursework sweep; they were not rerun during this publication audit.

Experiment 104 has the highest recorded historical validation accuracy:

• learning rate: 0.01
• batch size: 1
• hidden layers: [128, 128]
• nominal loss label: MSE
• actual loss used by the historical code: CrossEntropyLoss
• recorded training accuracy: 98.33%
• recorded validation accuracy: 97.66%

The 0.01-point difference from Experiment 97 is not meaningful evidence of superiority because each configuration was run once and the historical comparison did not use one common validation split.

Three distinct evidence layers

1. Historical 2025 sweep: the 243 preserved rows. The surviving code called the data loader nine times inside learning-rate/batch-size loops, and each call created an unseeded random 80/20 partition. Consequently, different groups were compared on different validation examples. The rows remain historical records, not results from the corrected method.
2. Documented 2026 retraining transcript: RESULTS.md contains console text reporting 97.75% test accuracy after retraining Experiment 104 on all 60,000 MNIST training images. No original JSON, raw log, checkpoint, or other machine-generated evaluation artifact was committed, so the transcript is the only surviving evidence for 97.75%.
3. Future corrected evaluations: evaluate_best_model.py performs a newly timestamped run and writes structured JSON with runtime versions, seed, device, deterministic settings, split sizes, dataset resources, downloaded-file hashes, measured accuracy, and Git commit. No new structured evaluation was generated during this audit.

Corrected method for future comparisons

mlp_mnist.py now downloads the MNIST training dataset once, creates one deterministic seed-42 80/20 split, stores its indices in the local run output, and reuses exactly those indices for every configuration. Loader shuffling is reset reproducibly for each configuration. The official MNIST test set is neither downloaded nor referenced by the sweep, so it cannot influence configuration selection.

This correction applies only to future runs. It does not retroactively change or validate the 2025 rows, and the repository intentionally does not rerun all 243 experiments merely to replace the historical record.

Installation

Supported Python version: 3.11, declared in .python-version and used by CI.

python3.11 -m venv .venv
# Windows: .venv\Scripts\activate
# Linux/macOS: source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

requirements.txt pins the repository's existing versions and includes PyTorch's official CPU-wheel index as an additional index. PyPI remains available for NumPy, Matplotlib, and transitive packages. Dependency resolution was validated for CPython 3.11; a complete local 3.11 installation was not run because this machine has no Python 3.11 interpreter.

Commands

Run fast tests without downloading MNIST or training:

python -m unittest discover -s tests -v

Run the corrected future sweep (243 configurations and potentially hours of CPU time):

python mlp_mnist.py

Local sweep output is written to the ignored experiment_results/ directory, including split_indices.json. It does not overwrite the historical results/summary.txt.

Perform a new selected-model retraining and held-out evaluation:

python evaluate_best_model.py \
  --summary results/summary.txt \
  --output results/selected_model_evaluation.json \
  --seed 42

This command downloads MNIST, trains for five epochs with batch size 1, and may take substantial time on CPU. A JSON file is written only after evaluation completes; its accuracy and timestamp belong to that new run and must not be labeled as the historical 97.75% transcript.

Regenerate the aggregate historical chart:

python plot_summary.py

Known historical limitations

• The nine learning-rate/batch-size groups used changing random validation splits.
• Each configuration was run once.
• Only five epochs were used.
• Rows labeled MSE or MAE actually used CrossEntropy because the code substituted it for incompatible class-index targets. The nominal loss dimension therefore does not compare three losses.
• The exact historical package versions, split indices, seeds after RNG consumption, and machine-generated test-evaluation artifact are unavailable.

Repository structure

README.md                    Project overview and corrected workflow
RESULTS.md                   Historical analysis and 2026 transcript provenance
mlp_mnist.py                 Future fixed-split sweep
data_utils.py                Reusable split and loader utilities
historical_results.py        Strict parser and best-row selection
evaluate_best_model.py       Selected-model retraining and JSON evaluation
evaluation_schema.py         Structured evaluation validation/writing
plot_summary.py              Aggregate historical visualization
results/summary.txt          243 preserved historical rows plus later comments
tests/                       Fast tests; no training or downloads

License

Repository code and documentation are available under the MIT License. MNIST is downloaded separately by torchvision and is not included in this repository.