SARA: Singular-Value Based Adaptive Low-Rank Adaption

Jihao Gu, Shuai Chen, Zelin Wang +2

With the increasing number of parameters in large pre-trained models, LoRA as a parameter-efficient fine-tuning(PEFT) method is widely used for not adding inference overhead. The L…