GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
arXiv:2305.12201 · doi:10.1109/CLOUD60044.2023.00045
Abstract
Distributed data-parallel (DDP) training improves overall application throughput as multiple devices train on a subset of data and aggregate updates to produce a globally shared model. The periodic synchronization at each iteration incurs considerable overhead, exacerbated by the increasing size and complexity of state-of-the-art neural networks. Although many gradient compression techniques propose to reduce communication cost, the ideal compression factor that leads to maximum speedup or minimum data exchange remains an open-ended problem since it varies with the quality of compression, model size and structure, hardware, network topology and bandwidth. We propose GraVAC, a framework to dynamically adjust compression factor throughout training by evaluating model progress and assessing gradient information loss associated with compression. GraVAC works in an online, black-box manner without any prior assumptions about a model or its hyperparameters, while achieving the same or better accuracy than dense SGD (i.e., no compression) in the same number of iterations/epochs. As opposed to using a static compression factor, GraVAC reduces end-to-end training time for ResNet101, VGG16 and LSTM by 4.32x, 1.95x and 6.67x respectively. Compared to other adaptive schemes, our framework provides 1.94x to 5.63x overall speedup.
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- An overview of gradient descent optimization algorithms
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Mixed Precision Training
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- An Empirical Model of Large-Batch Training
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Understanding Top-k Sparsification in Distributed Deep Learning
- Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
- Critical Learning Periods in Deep Neural Networks
- The Early Phase of Neural Network Training
- RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
- An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
- Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
- AdaScale SGD: A User-Friendly Algorithm for Distributed Training
- On the Utility of Gradient Compression in Distributed Training Systems
- Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
- ScaDLES: Scalable Deep Learning over Streaming data at the Edge
- Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
- Pufferfish: Communication-efficient Models At No Extra Cost