PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning

Abstract

Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models. However, algorithms for decentralized training with compressed communication over arbitrary connected networks have been more complicated, requiring additional memory and hyperparameters. We introduce a simple algorithm that directly compresses the model differences between neighboring workers using low-rank linear compressors applied on model differences. Inspired by the PowerSGD algorithm for centralized deep learning, this algorithm uses power iteration steps to maximize the information transferred per bit. We prove that our method requires no additional hyperparameters, converges faster than prior methods, and is asymptotically independent of both the network and the compression. Out of the box, these compressors perform on par with state-of-the-art tuned compression algorithms in a series of deep learning benchmarks.

arXiv commentTo appear in NeurIPS 2020

References in corpus (17)

• Language Models are Few-Shot Learners
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparsified SGD with Memory
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Stochastic Gradient Push for Distributed Deep Learning
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• The Early Phase of Neural Network Training
• Moniqua: Modulo Quantized Communication in Decentralized SGD

Cited by in corpus (7)

• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• Balanced Hodge Laplacians Optimize Consensus Dynamics over Simplicial Complexes
• Consensus Control for Decentralized Deep Learning
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• ErrorCompensatedX: error compensation for variance reduced algorithms