XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection

Yuanhang Yang, Shiyi Qi, Wenchao Gu +3

Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational…