Low-Rank Interconnected Adaptation across Layers

Yibo Zhong, Jinman Zhao, Yao Zhou

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates $\Delta W=AB$ for pretrained weights $W$ through low-rank adapters…