Empowering Distributed Training with Sparsity-driven Data Synchronization

Zhuang Wang, Zhaozhuo Xu, Jingyi Xi +3

Distributed training is the de facto standard to scale up the training of deep learning models with multiple GPUs. Its performance bottleneck lies in communications for gradient sy…