FAST: An Efficient Scheduler for All-to-All GPU Communication

Yiran Lei, Dongjoo Lee, Liangyu Zhao +9

All-to-All(v) communication is a critical primitive in modern machine learning workloads, particularly mixture-of-experts (MoE) models. Unfortunately, efficient scheduling is chall…