Mixture of Heterogeneous Grouped Experts for Language Modeling

Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5

Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enf…