LoRA Learns Less and Forgets Less

Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz +9

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected w…