LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training

Xinyi Liu, Yujie Wang, Fangcheng Fu +4

Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different inp…