Which components to adapt with LoRA in CLIP for continual learning?

[LouisDo2108]

Dear authors,

Thank you for maintaining such valuable resources and codebases for the continual learning community.
I have a question regarding the default components to adapt when using LoRA (or any parameter-efficient fine-tuning method) with CLIP.

I have noticed varying strategies across the literature:

• BOFA tunes only the text/vision projectors.
• MG-CLIP tunes only Wk and Wv in the attention layers for both encoders.
• HypCL tunes Wq and Wv in the vision encoder only.
• In standard Class-Incremental Learning (CIL) benchmarks (e.g., CIFAR-100, ImageNet-R, CUB, etc.), the text encoder is typically completely frozen.

Given this, I am looking for a general rule of thumb: in a general multimodal scenario, to maximize plasticity while strictly preserving CLIP's pre-trained knowledge (i.e., zero-shot), which components should ideally be tuned?

Specifically, I would greatly appreciate your thoughts on the following:

• Attention Layers: Is it generally better to adapt only Q and V, or K and V?
• MLPs and Embeddings: I think they are generally frozen.
• Projectors: Should the final vision/text projectors be fully fine-tuned or using a high LoRA rank (e.g., r=64 like in BOFA)?
• Norms: Is it recommended to update LayerNorm parameters?

Any insights or "best practices" your group follows when designing these architectures would be incredibly helpful.