RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoS…