Towards a Statistical Understanding of Mixture-of-Experts

Siyuan He, Bokai Yang, Jie Hu +2

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small s…