Two Is Better Than One: Rotations Scale LoRAs

Hongcan Guo, Guoshun Nan, Yuan Yang +9

Scaling Low-Rank Adaptation (LoRA)-based Mixture-of-Experts (MoE) facilitates large language models (LLMs) to efficiently adapt to diverse tasks. However, traditional gating mechan…