Turn Waste into Worth: Rectifying Top-$k$ Router of MoE

Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei +7

Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-$k$ routing mechanism su…