A deep-learning image classifier for skin-lesion diagnosis, built in PyTorch on the DermaMNIST (HAM10000) dataset. The project trains models to classify dermatoscopic images into 7 lesion types, with a focus on the central challenge of the dataset: severe class imbalance, where the naive "high accuracy" is dangerously misleading.
DermaMNIST contains 7 classes of skin lesion, but they are wildly imbalanced: melanocytic nevi make up 67% of the data, while dermatofibroma is just 1.1%. This creates a trap: a model that ignores the images and always predicts the majority class scores 67% accuracy while learning nothing useful.
This matters clinically, not just statistically. The rare classes include melanoma, the most dangerous lesion. A model that plays the odds by ignoring rare classes will systematically miss the cases that matter most. So the real goal is not headline accuracy, but per-class recall, especially on the dangerous minority classes.
I ran a controlled 2x2 experiment, varying two factors independently:
- Architecture: a small from-scratch fully-connected network (
TinyNet) vs a pretrained ResNet-18 (transfer learning, frozen body + fine-tuned head). - Loss: standard cross-entropy vs inverse-frequency weighted cross-entropy (making rare-class errors cost more).
This isolates the effect of each factor and tests whether better features (transfer learning) and imbalance handling (weighting) are the same thing or separate, complementary tools.
Per-class recall (%), validation set:
| Class | TinyNet | TinyNet + weighted | ResNet | ResNet + weighted |
|---|---|---|---|---|
| actinic keratoses | 15.2 | 42.4 | 66.7 | 69.7 |
| basal cell carcinoma | 55.8 | 57.7 | 17.3 | 57.7 |
| benign keratosis | 26.4 | 41.8 | 26.4 | 25.5 |
| dermatofibroma | 0.0 | 25.0 | 33.3 | 41.7 |
| melanoma | 18.9 | 37.8 | 42.3 | 55.9 |
| nevi (majority) | 94.6 | 50.4 | 95.1 | 57.1 |
| vascular | 21.4 | 71.4 | 21.4 | 78.6 |
| overall accuracy | 72.0 | 48.2 | 75.0 | 54.0 |
Key findings:
-
Accuracy is misleading here. The plain TinyNet's 72% comes almost entirely from the majority class (94.6% recall on nevi) while it misses 81% of melanomas. High accuracy, poor screening tool.
-
Feature quality and imbalance handling are separate problems. Transfer learning raised overall accuracy and melanoma recall, but the plain ResNet still leaned on the majority class (95.1% nevi). Better features alone did not solve imbalance.
-
The two techniques stack. ResNet + weighted loss catches more melanomas than either technique alone, while staying balanced across rare classes. Its overall accuracy is lower than plain ResNet, but it is the more useful model for screening, where a missed melanoma is far costlier than a false alarm.
Beyond recall, this model was evaluated with a confusion matrix and full precision/recall/F1 analysis:
The model trades precision for recall on dangerous classes: it catches 56% of melanomas but with many false alarms (29% precision). The confusion matrix shows its errors are mostly "safe" ones, confusing melanoma with other malignant classes rather than benign ones, and over-flagging benign moles as suspicious. For a screening context, erring toward caution is the defensible bias.
The failure gallery bears this out. Almost every mistake is a benign nevus called something worse, and plenty of those moles do look dark and irregular. The source images are 28x28 upscaled to 224, so the fine texture that separates lesions is blurred out before the model ever sees it.
Macro-average F1 is 0.376 against a weighted-average of 0.596. That spread is the model still doing better on common classes than on rare ones.
The evaluation flagged two things holding the model back. The images were upscaled from 28x28, so the texture that separates lesion types was gone before the model saw them. And the ResNet body was frozen solid, so its features never adjusted from ImageNet to skin. I tried both fixes.
Native resolution. MedMNIST also publishes the images resized straight from the original HAM10000 photos to 224px, instead of the 28px thumbnails stretched up. The model was already resizing everything to 224px anyway, so the compute is identical; the only difference is whether the pixels are real or interpolated.
Partial fine-tuning. A full unfreeze is 11.2M parameters and too slow on CPU, so I unfroze just the last residual block (layer4) and the head, about 8.4M parameters. That lets the deepest features adapt while the early edge and texture filters stay as ImageNet left them. One catch: the still-frozen blocks have to run in eval() mode, because requires_grad = False stops their weights updating but not their BatchNorm running stats. The learning rate also drops 10x to 1e-4 so the pretrained weights survive the first few steps.
| Stage | Overall accuracy | Melanoma recall | Melanoma precision | Macro F1 |
|---|---|---|---|---|
| ResNet + weighted (upscaled, frozen) | 54.0 | 55.9 | 29.1 | 0.376 |
| + native 224px resolution (frozen) | 68.6 | 64.9 | n/a | n/a |
| + partial fine-tuning (layer4 unfrozen) | 77.0 | 73.9 | 43.2 | 0.680 |
(The frozen-native run only recorded per-class recall. Precision and macro F1 are from the fine-tuned model's classification report.)
Resolution alone added 14.6 points of accuracy at no extra cost. Fine-tuning added another 8.4 on top. Melanoma is the clearest case: recall went from 56% to 74% and precision from 29% to 43% together, so the model catches more real melanomas while raising fewer false alarms per flag. Usually one of those comes at the expense of the other. The gap between macro and weighted F1, which had been the clearest sign of a model coasting on the majority class, closed from 22 points to 10.6. Every class now sits above 58% recall; in the upscaled model the weakest was 25%.
Fine-tuning drove train loss down hard, 1.11 to 0.20, which on its own would read as overfitting. But validation recall went up on all seven classes, not just the easy ones, so this was the model learning better features rather than memorising the training set.
- Unfreeze the whole ResNet body, not just layer4. The remaining frozen layers would probably give a little more, but a full fine-tune needs a GPU to be practical.
- Augment or oversample the rare classes to pull melanoma precision up further without giving back recall.
- Tune the class-weight strength. It over-flags right now; there is likely a setting that trades a little recall for much less noise.
- Build a confusion matrix and failure gallery for the fine-tuned model. The first evaluation showed melanoma being confused with other malignant classes rather than benign ones, and it is worth checking whether that still holds.
SkinSight/
├── notebooks/ # step-by-step build, from PyTorch fundamentals to evaluation
│ ├── 00_tensors.ipynb ... 02_training_loop.ipynb # fundamentals (built from scratch)
│ ├── 03_data.ipynb # data loading + imbalance analysis
│ ├── 04_baseline_and_imbalance.ipynb # weighted-loss experiment
│ ├── 05_transfer_learning.ipynb # ResNet transfer learning
│ ├── 06_evaluation.ipynb # confusion matrix, F1, failure gallery
│ └── 07_higher_resolution.ipynb # native resolution + partial fine-tuning
├── src/ # reusable, importable pipeline
│ ├── model.py # TinyNet + ResNet builder
│ ├── data.py # transforms, loaders, class weights
│ └── train.py # train / evaluate / seed functions
└── requirements.txt
pip install -r requirements.txtThe src/ module runs the whole pipeline in a few lines. The frozen ResNet + weighted-loss baseline from the 2x2 experiment:
from src.model import build_resnet
from src.data import get_loaders, compute_class_weights
from src.train import set_seed, train, evaluate
set_seed(42)
train_loader, val_loader, train_data = get_loaders()
weights = compute_class_weights(train_data)
model = build_resnet(num_classes=7, freeze_body=True)
model = train(model, train_loader, class_weights=weights, epochs=3)
accuracy, per_class_recall = evaluate(model, val_loader)The Phase 2 model (native resolution, layer4 fine-tuned) is the same three calls with different arguments:
set_seed(42)
train_loader, val_loader, train_data = get_loaders(native=True)
weights = compute_class_weights(train_data)
model = build_resnet(num_classes=7, freeze_body=True, unfreeze_layer4=True)
model = train(model, train_loader, class_weights=weights, epochs=3, lr=1e-4, freeze_bn=True)
accuracy, per_class_recall = evaluate(model, val_loader)Or work through the notebooks in order to see the full build, from tensor basics to the final evaluation.
PyTorch, torchvision, scikit-learn, medmnist, matplotlib, NumPy.
This is an educational project on a public benchmark dataset. It is not a medical device and must not be used for actual diagnosis.
This project is licensed under the MIT License - see the LICENSE file for details.
The DermaMNIST dataset is distributed under CC BY-NC 4.0 (non-commercial). The underlying HAM10000 images are credited to their original authors.


