Extrapolation for Large-batch Training in Deep Learning
arXiv:2006.05720
Abstract
Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batch size to a substantial fraction of the training data for improving training time is the persistent degradation in performance (generalization gap). To address this issue, recent work propose to add small perturbations to the model parameters when computing the stochastic gradients and report improved generalization performance due to smoothing effects. However, this approach is poorly understood; it requires often model-specific noise and fine-tuning. To alleviate these drawbacks, we propose to use instead computationally efficient extrapolation (extragradient) to stabilize the optimization trajectory while still benefiting from smoothing to avoid sharp minima. This principled approach is well grounded from an optimization perspective and we show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer. We demonstrate that in a variety of experiments the scheme allows scaling to much larger batch sizes than before whilst reaching or surpassing SOTA accuracy.
References in corpus (30)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Densely Connected Convolutional Networks
- Visualizing the Loss Landscape of Neural Nets
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Large Batch Training of Convolutional Networks
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- Revisiting Small Batch Training for Deep Neural Networks
- Deep Networks with Stochastic Depth
- Don't Use Large Mini-Batches, Use Local SGD
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
- A Variational Inequality Perspective on Generative Adversarial Networks
- Measuring the Effects of Data Parallelism on Neural Network Training
- On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
- An Empirical Model of Large-Batch Training
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
- Implicit Regularization in Deep Learning
- The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
- Reducing Noise in GAN Training with Variance Reduced Extragradient
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks
- Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
- Revisiting Stochastic Extragradient
- Stochastic Nonconvex Optimization with Large Minibatches
- Large-Batch Training for LSTM and Beyond
- Accelerated Extra-Gradient Descent: A Novel Accelerated First-Order Method
- Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD