DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models

Guanzhi Deng, Bo Li, Ronghao Chen +5

Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning methods, such as LoRA, are widely adopted to adap…