From promise to practice: realizing high-performance decentralized training

Zesen Wang, Jiaojiao Zhang, Xuyang Wu +1

Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability over synchronous data-parallel methods like All-Reduce…