On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Abstract

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

Accepted as a conference paper at ICLR 2017

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks

Cited by in corpus (29)

• Revisiting Distributed Synchronous SGD
• Large Batch Training of Convolutional Networks
• A Closer Look at Memorization in Deep Networks
• Exploring Generalization in Deep Learning
• Implicit Regularization in Deep Learning
• Scaling Deep Learning on GPU and Knights Landing clusters
• Analysis and Optimization of Convolutional Neural Network Architectures
• ImageNet Training in Minutes
• Mixed Precision Training With 8-bit Floating Point
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
• A Scale Invariant Flatness Measure for Deep Network Minima
• Stability and Generalization of Learning Algorithms that Converge to Global Optima
• On Scale-out Deep Learning Training for Cloud and HPC
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
• Information Bottleneck and its Applications in Deep Learning
• Implicit Regularization in Over-parameterized Neural Networks
• Practical Processing of Mobile Sensor Data for Continual Deep Learning Predictions
• On improving deep learning generalization with adaptive sparse connectivity
• Mini-batch stochastic gradient descent with dynamic sample sizes
• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• Training on the Edge: The why and the how
• Improved visible to IR image transformation using synthetic data augmentation with cycle-consistent adversarial networks
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
• Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input