On the Utility of Gradient Compression in Distributed Training Systems

Abstract

A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent work proposes gradient and model compression methods. In this work, we evaluate the efficacy of gradient compression methods and compare their scalability with optimized implementations of synchronous data-parallel SGD across more than 200 different setups. Surprisingly, we observe that only in 6 cases out of more than 200, gradient compression methods provide speedup over optimized synchronous data-parallel training in the typical data-center setting. We conduct an extensive investigation to identify the root causes of this phenomenon, and offer a performance model that can be used to identify the benefits of gradient compression for a variety of system setups. Based on our analysis, we propose a list of desirable properties that gradient compression methods should satisfy, in order for them to provide a meaningful end-to-end speedup.

References in corpus (41)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Zero-Shot Text-to-Image Generation
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• Towards Federated Learning at Scale: System Design
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Don't Decay the Learning Rate, Increase the Batch Size
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• Sparsified SGD with Memory
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• MLPerf Training Benchmark
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Distributed Mean Estimation with Limited Communication
• Beyond Data and Model Parallelism for Deep Neural Networks
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• Scaling Distributed Machine Learning with In-Network Aggregation
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• 8-Bit Approximations for Parallelism in Deep Learning
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Understanding Top-k Sparsification in Distributed Deep Learning
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Natural Compression for Distributed Deep Learning
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Large batch size training of neural networks with adversarial training and second-order information
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Distributed Learning with Sublinear Communication
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training

Cited by in corpus (6)

• A Field Guide to Federated Optimization
• Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
• Accelerating Distributed ML Training via Selective Synchronization
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research