$\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression

Abstract

Communication is a key bottleneck in distributed training. Recently, an \emph{error-compensated} compression technology was particularly designed for the \emph{centralized} learning and receives huge successes, by showing significant advantages over state-of-the-art compression based methods in saving the communication cost. Since the \emph{decentralized} training has been witnessed to be superior to the traditional \emph{centralized} training in the communication restricted scenario, therefore a natural question to ask is "how to apply the error-compensated technology to the decentralized learning to further reduce the communication cost." However, a trivial extension of compression based centralized training algorithms does not exist for the decentralized scenario. key difference between centralized and decentralized training makes this extension extremely non-trivial. In this paper, we propose an elegant algorithmic design to employ error-compensated stochastic gradient descent for the decentralized scenario, named $\text{DeepSqueeze}$. Both the theoretical analysis and the empirical study are provided to show the proposed $\text{DeepSqueeze}$ algorithm outperforms the existing compression based decentralized learning algorithms. To the best of our knowledge, this is the first time to apply the error-compensated compression to the decentralized learning.

References in corpus (9)

• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparsified SGD with Memory
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• Distributed Delayed Stochastic Optimization
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

Cited by in corpus (10)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Decentralized Deep Learning with Arbitrary Communication Compression
• Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks
• Linear Convergent Decentralized Optimization with Compression
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
• Trends and Advancements in Deep Neural Network Communication
• Decentralized Composite Optimization with Compression
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning