A Single Linear Layer Yields Task-Adapted Low-Rank Matrices

Hwichan Kim, Shota Sasaki, Sho Hoshino +1

Low-Rank Adaptation (LoRA) is a widely used Parameter-Efficient Fine-Tuning (PEFT) method that updates an initial weight matrix $W_0$ with a delta matrix $\Delta W$ consisted by two low-…