$\ddot{I}•$-Balancing for Mixture-of-Experts Training

Lizhang Chen, Jonathan Li, Qi Wang +5

Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate o…