Large-Scale Deep Learning Optimizations: A Comprehensive Survey

Abstract

Deep learning have achieved promising results on a wide spectrum of AI applications. Larger datasets and models consistently yield better performance. However, we generally spend longer training time on more computation and communication. In this survey, we aim to provide a clear sketch about the optimizations for large-scale deep learning with regard to the model accuracy and model efficiency. We investigate algorithms that are most commonly used for optimizing, elaborate the debatable topic of generalization gap arises in large-batch training, and review the SOTA strategies in addressing the communication overhead and reducing the memory footprints.

References in corpus (35)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• Mixed Precision Training
• One weird trick for parallelizing convolutional neural networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Large Batch Training of Convolutional Networks
• Revisiting Small Batch Training for Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• New insights and perspectives on the natural gradient method
• Three Factors Influencing Minima in SGD
• Sparsified SGD with Memory
• On Empirical Comparisons of Optimizers for Deep Learning
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Optimization for deep learning: theory and algorithms
• On Layer Normalization in the Transformer Architecture
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Image Classification at Supercomputer Scale
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Parallel SGD: When does averaging help?
• AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training
• Natural Compression for Distributed Deep Learning
• Scalable Second Order Optimization for Deep Learning
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Memory-Efficient Adaptive Optimization
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Go Wider Instead of Deeper