On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Abstract

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

Accepted as a conference paper at ICLR 2017

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Playing Atari with Deep Reinforcement Learning
• The Loss Surfaces of Multilayer Networks
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Gradient Descent Converges to Minimizers
• Training Recurrent Neural Networks by Diffusion

Cited by in corpus (201)

• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Large Batch Training of Convolutional Networks
• A Closer Look at Memorization in Deep Networks
• Revisiting Small Batch Training for Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Exploring Generalization in Deep Learning
• A Point-Cloud Deep Learning Framework for Prediction of Fluid Flow Fields on Irregular Geometries
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Training Tips for the Transformer Model
• Understanding and Robustifying Differentiable Architecture Search
• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• Model compression via distillation and quantization
• Time-to-Event Prediction with Neural Networks and Cox Regression
• Local SGD Converges Fast and Communicates Little
• What is being transferred in transfer learning?
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Biologically-informed neural networks guide mechanistic modeling from sparse experimental data
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• Nonlinear Information Bottleneck
• Fantastic Generalization Measures and Where to Find Them
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Optimization for deep learning: theory and algorithms
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Stronger generalization bounds for deep nets via a compression approach
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Implicit Regularization in Deep Learning
• Magnetostatics and micromagnetics with physics informed neural networks
• Scaling Deep Learning on GPU and Knights Landing clusters
• A newcomer's guide to deep learning for inverse design in nano-photonics
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Efficiency of quantum versus classical annealing in non-convex learning problems
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• The Case for Bayesian Deep Learning
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Multiple Sclerosis Lesion Segmentation from Brain MRI via Fully Convolutional Neural Networks
• A Progressive Batching L-BFGS Method for Machine Learning
• Analysis and Optimization of Convolutional Neural Network Architectures
• Three Mechanisms of Weight Decay Regularization
• Point-Cloud Deep Learning of Porous Media for Permeability Prediction
• Where is the Information in a Deep Neural Network?
• ImageNet Training in Minutes
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• A Survey of Optimization Methods from a Machine Learning Perspective
• A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis
• DeLTA: GPU Performance Model for Deep Learning Applications with In-depth Memory System Traffic Analysis
• Mixed Precision Training With 8-bit Floating Point
• Understanding Batch Normalization
• Towards Binary-Valued Gates for Robust LSTM Training
• Capstan: A Vector RDA for Sparsity
• Understanding training and generalization in deep learning by Fourier analysis
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• Large batch size training of neural networks with adversarial training and second-order information
• Small-GAN: Speeding Up GAN Training Using Core-sets
• CASI: A Convolutional Neural Network Approach for Shell Identification
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Laplacian Smoothing Gradient Descent
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• CASTLE: Regularization via Auxiliary Causal Graph Discovery
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
• How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks
• A Scale Invariant Flatness Measure for Deep Network Minima
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• Stability and Generalization of Learning Algorithms that Converge to Global Optima
• On Scale-out Deep Learning Training for Cloud and HPC
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Reducing Energy Bloat in Large Model Training
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• LCA: Loss Change Allocation for Neural Network Training
• Assessment of Neural Network Augmented Reynolds Averaged Navier Stokes Turbulence Model in Extrapolation Modes
• Communication-efficient distributed SGD with Sketching
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Generalization Bounds for Convolutional Neural Networks
• Achieving Adversarial Robustness via Sparsity
• Taming Momentum in a Distributed Asynchronous Environment
• Generalization Bounds for Neural Belief Propagation Decoders
• Test time Adaptation through Perturbation Robustness
• First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise
• Information Bottleneck and its Applications in Deep Learning
• Understanding Generalization in Deep Learning via Tensor Methods
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Implicit Regularization in Over-parameterized Neural Networks
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Is Support Set Diversity Necessary for Meta-Learning?
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• Decentralized Markov Chain Gradient Descent
• Neural networks for insurance pricing with frequency and severity data: a benchmark study from data preprocessing to technical tariff
• Trust Region Based Adversarial Attack on Neural Networks
• The Bayesian Learning Rule
• Robustness to Augmentations as a Generalization metric
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
• Practical Processing of Mobile Sensor Data for Continual Deep Learning Predictions
• Using Mode Connectivity for Loss Landscape Analysis
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• The Impact of Local Geometry and Batch Size on Stochastic Gradient Descent for Nonconvex Problems
• On improving deep learning generalization with adaptive sparse connectivity
• Noisy Recurrent Neural Networks
• Arithmetic with Language Models: from Memorization to Computation
• On the Effectiveness of Regularization Against Membership Inference Attacks
• Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Neural Complexity Measures
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Learning from Images: Proactive Caching with Parallel Convolutional Neural Networks
• Mini-batch stochastic gradient descent with dynamic sample sizes
• Pufferfish: Communication-efficient Models At No Extra Cost
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training
• Variance reduction for Riemannian non-convex optimization with batch size adaptation
• A Convex Neural Network Solver for DCOPF with Generalization Guarantees
• Improved Techniques For Weakly-Supervised Object Localization
• Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
• Optimizing Deeper Transformers on Small Datasets
• On the Convergence of SARAH and Beyond
• A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
• Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?
• Gradient Energy Matching for Distributed Asynchronous Gradient Descent
• Epigenetic evolution of deep convolutional models
• Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• Character-Level Feature Extraction with Densely Connected Networks
• Inefficiency of K-FAC for Large Batch Size Training
• DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
• Implicit bias of deep linear networks in the large learning rate phase
• Activated Gradients for Deep Neural Networks
• GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
• Tangent Space Separability in Feedforward Neural Networks
• Improved generalization by noise enhancement
• Why flatness does and does not correlate with generalization for deep neural networks
• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• Understanding Deep Architectures with Reasoning Layer
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• What training reveals about neural network complexity
• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
• Generalized Resubstitution for Classification Error Estimation
• Consistent Lock-free Parallel Stochastic Gradient Descent for Fast and Stable Convergence
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
• Acceleration via Fractal Learning Rate Schedules
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• Extrapolation for Large-batch Training in Deep Learning
• The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data
• SaaS: Speed as a Supervisor for Semi-supervised Learning
• Large Batch Training Does Not Need Warmup
• Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
• How do SGD hyperparameters in natural training affect adversarial robustness?
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Training on the Edge: The why and the how
• Improved visible to IR image transformation using synthetic data augmentation with cycle-consistent adversarial networks
• The role of invariance in spectral complexity-based generalization bounds
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
• Explicitly Bayesian Regularizations in Deep Learning
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Unique Properties of Flat Minima in Deep Networks
• Contrastive Weight Regularization for Large Minibatch SGD
• Arithmetic addition of two integers by deep image classification networks: experiments to quantify their autonomous reasoning ability
• Improving the Deconvolution of Spectrum at Finite Temperature via Neural Network
• Regularization in ResNet with Stochastic Depth
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
• A Resizable Mini-batch Gradient Descent based on a Multi-Armed Bandit
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Sequenced-Replacement Sampling for Deep Learning
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• LRTuner: A Learning Rate Tuner for Deep Neural Networks
• Long Short Term Memory Networks for Bandwidth Forecasting in Mobile Broadband Networks under Mobility
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD
• Filtering Statistics on Networks
• Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input
• Multilayer Lookahead: a Nested Version of Lookahead
• Trade-offs of Local SGD at Scale: An Empirical Study
• Logit Attenuating Weight Normalization
• Think Global, Act Local: Relating DNN generalisation and node-level SNR
• Addressing Algorithmic Bottlenecks in Elastic Machine Learning with Chicle
• Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data