ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws

Hai Huang, Randall Balestriero

Low-Rank Adaptation (LoRA) is the bread and butter of Large Language Model (LLM) finetuning. LoRA learns an additive low-rank perturbation, $AB$, of a pretrained matrix parameter $…