Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning

Ziyu Zhao, Yixiao Zhou, Zhi Zhang +10

Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with…