Fine-grained MoE Load Balancing with Linear Programming

Chenqi Zhao, Wenfei Wu, Linhai Song +2

Mixture-of-Experts (MoE) has emerged as a promising approach to scale up deep learning models due to its significant reduction in computational resources. However, the dynamic natu…