MERGETUNE: Continued Fine-Tuning of Vision-Language Models

Wenqing Wang, Da Li, Xiatian Zhu +1

Fine-tuning vision-language models (VLMs) such as CLIP often leads to catastrophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adapt…