ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning

Fanyao Li, Yuxi Liu, Xiaohao Liu +3

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all…