SSD-SSD: Communication sparsification for distributed deep learning training

Yemao Xu, Dezun Dong, Yawei Zhao +2

Intensive communication and synchronization cost for gradients and parameters is the well-known bottleneck of distributed deep learning training. Based on the observations that Syn…