Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression

Lin Meng, Yuzhong Sun, Weimin Li

Existing Data Parallel (DP) trainings for deep neural networks (DNNs) often experience limited scalability in speedup due to substantial communication overheads. While Overlapping…