Accelerating MoE Training through Expert Placement and Replica Coordination

Zhiqiang Tan, Maoxin Wang, Wenxiang Lin +4

Distributed training of sparsely activated mixture-of-experts (MoE) models is often bottlenecked by workload imbalance and all-to-all (A2A) communication induced by dynamic token r…