Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

Yunze Wei, Tianshuo Hu, Cong Liang +1

The past few years have witnessed the flourishing of large-scale deep neural network models with ever-growing parameter numbers. Training such large-scale models typically requires…