Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

Yury Nahshan, Nati Daniel, Jacob Goldberger +1

Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing ter…