Efficient Modular Learning through Naive LoRA Summation: Leveraging Orthogonality in High-Dimensional Models

Zhanhao Cao, Clement Truong, Andrew Lizarraga

Recent advances in large language models are driven by scale, while parameter-efficient fine-tuning (PEFT) enables updating only a small fraction of parameters. Low-Rank Adaptation…