RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication

Chutian Wang, Wenhao He, Jingmin Zhu +3

Irregular All-to-All communication is a major bottleneck in expert-parallel Mixture-of-Experts (MoE) models. Even with fixed expert routing and placement, uneven utilization of par…