PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

Abstract

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on power iteration that can i) compress gradients rapidly, ii) efficiently aggregate the compressed gradients using all-reduce, and iii) achieve test performance on par with SGD. The proposed algorithm is the only method evaluated that achieves consistent wall-clock speedups when benchmarked against regular SGD with an optimized communication backend. We demonstrate reduced training times for convolutional networks as well as LSTMs on common datasets. Our code is available at https://github.com/epfml/powersgd.

Presented at NeurIPS 2019

References in corpus (15)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparsified SGD with Memory
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Measuring the Effects of Data Parallelism on Neural Network Training
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Sketchy Decisions: Convex Low-Rank Matrix Optimization with Optimal Storage
• Detecting Memorization in ReLU Networks

Cited by in corpus (45)

• Zero-Shot Text-to-Image Generation
• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Optimal Client Sampling for Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Communication optimization strategies for distributed deep neural network training: A survey
• On Biased Compression for Distributed Learning
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Optimal Gradient Compression for Distributed and Federated Learning
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• Reducing Communication for Split Learning by Randomized Top-k Sparsification
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• On the Utility of Gradient Compression in Distributed Training Systems
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• Sparse Communication for Training Deep Networks
• Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• DRIVE: One-bit Distributed Mean Estimation
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• Faster Neural Network Training with Approximate Tensor Operations
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Activations and Gradients Compression for Model-Parallel Training
• OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning
• Accelerating Distributed ML Training via Selective Synchronization
• Communication-Efficient Distributed SGD with Error-Feedback, Revisited
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• New Bounds For Distributed Mean Estimation and Variance Reduction
• Trends and Advancements in Deep Neural Network Communication
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• CSER: Communication-efficient SGD with Error Reset
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• Towards Tight Communication Lower Bounds for Distributed Optimisation
• MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training
• Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
• Improving Differentially Private SGD via Randomly Sparsified Gradients
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Invexifying Regularization of Non-Linear Least-Squares Problems