ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Ahin Lee, Sehyun Yun, Joonha Park +1

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation…