SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

Junghun Oh, Sungyong Baik, Kyoung Mu Lee

Low-Rank Adaptation (LoRA) enables efficient adaptation of large pretrained models to downstream tasks by parameterizing weight updates with low-rank matrices. In this paper, we in…