From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing

Rana Shahout, Colin Cai, Yilun Du +2

Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts through a learned gate function. While conditional routing reduces trainin…