OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning

Jinyuan Feng, Zhiqiang Pu, Tianyi Hu +3

Building mixture-of-experts (MoE) architecture for Low-rank adaptation (LoRA) is emerging as a potential direction in parameter-efficient fine-tuning (PEFT) for its modular design…