Advancing Expert Specialization for Better MoE

Hongcan Guo, Haolang Lu, Guoshun Nan +8

Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. However, we observe that the commonly use…