Large-Scale Deep Learning Optimizations: A Comprehensive Survey
arXiv:2111.00856
Abstract
Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend longer training time on more computation and communication. In this survey, we aim to provide a clear sketch about the optimizations for large-scale deep learning with regard to the model accuracy and model efficiency. We investigate algorithms that are most commonly used for optimizing, elaborate the debatable topic of generalization gap arises in large-batch training, and review the SOTA strategies in addressing the communication overhead and reducing the memory footprints.
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- One weird trick for parallelizing convolutional neural networks
- Large Batch Training of Convolutional Networks
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes
- Optimization for deep learning: theory and algorithms
- Parallel SGD: When does averaging help?
- AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
- On the Generalization Benefit of Noise in Stochastic Gradient Descent
- Go Wider Instead of Deeper