Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

Wei Wang, Zhiquan Lai, Shengwei Li +5

The size of deep learning models has been increasing to enhance model quality. The linear increase in training computation budget with model size means that training an extremely l…