Selective LoRA for Visual Tokens and Attention Heads

Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim +2

Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, but its standard all-token, all-head design ignores the heterogeneous structure of vision language mo…