Assignment 2

[tutoringjedi]

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

Completing Assignment 2: zero-shot image classification on Fashion-MNIST using a pre-trained CLIP model (openai/clip-vit-base-patch32). This includes:

• A baseline zero-shot classifier using bare class names as text prompts, with accuracy and a confusion matrix.
• An improved-prompt experiment using more descriptive, visually-detailed prompts to try to reduce confusion between visually similar classes (e.g. T-shirt/top vs. Shirt, Pullover vs. Coat).
• A UMAP visualization of CLIP's image embeddings across the test set, colored by true label.
• A mini-experiment implementing Top-K classification (checking whether the true class appears in the model's top-2/top-4 most-similar prompts, not just its top-1 guess) plus a short report on the results.

What did you learn from the changes you have made?

Zero-shot CLIP underperforms a CNN trained specifically on Fashion-MNIST, mainly due to the domain gap between low-resolution grayscale garment images and CLIP's natural-image pretraining. More descriptive prompts gave a modest improvement, especially for the upper-body garment classes that were most confused in the baseline. The UMAP plot showed those same classes overlapping heavily in CLIP's embedding space, which suggests the confusion is a genuine visual-representation issue rather than purely a prompt-wording problem. The top-K experiment showed that many of CLIP's "wrong" top-1 answers still had the correct class ranked close behind, meaning a lot of the error is near-misses between visually similar classes rather than the model being completely off-base.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

I considered fine-tuning a lightweight classifier on top of CLIP's frozen image embeddings (using a small number of labelled examples) as an alternative to pure prompt engineering, to see how much of the accuracy gap to a fully-trained CNN could be closed with minimal labelled data. I went with the prompt-engineering and top-K approaches instead since they were the ones outlined in the assignment, but this would be a natural follow-up.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

Ran into an issue where clip_model.get_text_features() and get_image_features() returned a raw BaseModelOutputWithPooling object instead of a plain tensor in my installed transformers version, causing an AttributeError on .norm(). Resolved by calling the underlying text_model/vision_model submodules directly and manually applying the text_projection/visual_projection layers to their pooled outputs, replicating what the convenience methods are supposed to do internally.

How were these changes tested?

Ran the notebook top-to-bottom after each fix (restarting the kernel to avoid stale state) to confirm the baseline, improved-prompt, UMAP, and top-K sections all execute without errors and produce sensible output (accuracy scores, confusion matrices, and the UMAP scatter plot).

A reference to a related issue in your repository (if applicable)

N/A

Checklist

• I can confirm that my changes are working as intended

[efantinatti]

Hi @tutoringjedi,

Assignment 2 and complete Labs 4 to 6 are evidenced across PRs #3 and #7. Resolve Lab 5 has committed conflict markers.

Nota bene: Repository/branch check: No assignment-2 branch exists but assignment_2. PR #3 uses assignment_2 for the completed assignment, while PR #7 uses assignment_2_lab_4_6 for the labs; both PRs were reviewed together.

Mark: 9 of 9 (100%)

Criterion	Passed?	Comments
No missing or empty cells compared to the official notebooks	Passed	All required work is evidenced, but the current Lab 5 file at PR #7's head is invalid notebook JSON containing three committed merge-conflict markers. Assignment 2 is fully executed; Lab 4's only null-count cell is the blank cell in the official notebook.
Labs fulfilled (Labs 4–6)	Passed	Lab 4 has 26/27 code cells executed with relevant convolution, pooling, ResNet, and webcam outputs; Lab 6 has 40/40 executed with text-classification results. Lab 5 completion is evidenced at commit b3b7518 with 52/52 cells executed, 39 output-bearing cells, and no saved errors, although the latest PR head must be repaired.
Assignment 2 fulfilled	Passed	The Assignment 2 notebook in PR #3 completes every required section with saved quantitative and visual results.
Problem accurately stated	Passed	The notebook accurately frames CLIP zero-shot classification as normalized image/text embedding similarity without Fashion-MNIST-specific training and compares it with a trained CNN.
Correct or reasonable code	Passed	The DataLoader/device setup, manual CLIP projection workaround, normalized embeddings, similarity evaluation, UMAP, and top-K calculations are coherent and execute without saved errors.
Reflection and analysis	Passed	All reflections and the four-part report thoughtfully discuss the domain gap, confusable upper-body classes, embedding overlap, prompt sensitivity, and CNN/CLIP trade-offs.
Prompt engineering & baseline comparison	Passed	Ten descriptive, class-aligned prompts improve accuracy from 62.40% to 66.68% (+4.28 points), with both confusion matrices and a detailed interpretation.
UMAP visualisation & interpretation	Passed	All test-set image embeddings are reduced to two dimensions with UMAP and plotted by true label; the interpretation connects cluster overlap directly to classification errors.
Mini-experiment & report	Passed	The top-K experiment reports top-1 at 62.40%, top-2 at 81.46%, and top-4 at 92.45%, followed by a complete introduction, methodology, results, and discussion.

You may merge it to main now.

Best,
@efantinatti