ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models

Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh +1

Fine-tuning vision-language models such as CLIP typically requires backpropagation (BP) through the full model, which is infeasible when only forward-pass access is available, as i…