CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

Enda Yu, Dezun Dong, Yemao Xu +2

Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel c…