Mixture of Experts in Large Language Models

Danyang Zhang, Junhao Song, Ziqian Bi +5

This paper presents a comprehensive review of the Mixture-of-Experts (MoE) architecture in large language models, highlighting its ability to significantly enhance model performanc…