An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems

Abstract

The recent many-fold increase in the size of deep neural networks makes efficient distributed training challenging. Many proposals exploit the compressibility of the gradients and propose lossy compression techniques to speed up the communication stage of distributed training. Nevertheless, compression comes at the cost of reduced model quality and extra computation overhead. In this work, we design an efficient compressor with minimal overhead. Noting the sparsity of the gradients, we propose to model the gradients as random variables distributed according to some sparsity-inducing distributions (SIDs). We empirically validate our assumption by studying the statistical characteristics of the evolution of gradient vectors over the training process. We then propose Sparsity-Inducing Distribution-based Compression (SIDCo), a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) while being faster by imposing lower compression overhead. Our extensive evaluation of popular machine learning benchmarks involving both recurrent neural network (RNN) and convolution neural network (CNN) models shows that SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.

Accepted at the 2021 Machine Learning and Systems (MLSys) Conference

References in corpus (16)

• TensorFlow: A system for large-scale machine learning
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• Sparsified SGD with Memory
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Understanding Top-k Sparsification in Distributed Deep Learning
• SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• Limits on Sparse Data Acquisition: RIC Analysis of Finite Gaussian Matrices

Cited by in corpus (6)

• AI-based Fog and Edge Computing: A Systematic Review, Taxonomy and Future Directions
• On the Utility of Gradient Compression in Distributed Training Systems
• DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification
• Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks