SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information

Kaiye Zhou, Shucheng Wang, Jun Xu

In the training of large language models, parameter-efficient techniques such as LoRA optimize memory usage and reduce communication overhead and memory usage during the fine-tunin…