Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

Xinglin Pan, Wenxiang Lin, Shaohuai Shi +3

Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in…