Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation

Yize Wu, Ke Gao, Ling Li +1

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as $W=W_0+sBA$, where $W_0$ is the ori…