Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models

Yuan Sun

For pre-training of MoE (Mixture-of-Experts) models, one of the main issues is unbalanced expert loads, which may cause routing collapse or increased computational overhead. Existi…