Symphony: Taming Step Misalignments in the Network for Ring-based Collective Operations

Yuze Jin, Xin Zhe Khooi, Ruyi Yao +1

Ring-based collective operations are widely used in distributed AI training due to their efficient bandwidth utilization. While ring communication excels at pipelining, its perform…