The Hidden Power of Scaling Factor in LoRA Optimization

Zicheng Zhang, Haoran Li, Jiaxing Wang +10

In Low-Rank Adaptation (LoRA), the scaling factor $\alpha$ is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this pa…