Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging

Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang +3

Mixture-of-Experts (MoE) shines brightly in large language models (LLMs) and demonstrates outstanding performance in plentiful natural language processing tasks. However, existing…