Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Abstract

Background: Deep learning models are typically trained using stochastic gradient descent or one of its variants. These methods update the weights using their gradient, estimated from a small fraction of the training data. It has been observed that when using large batch sizes there is a persistent degradation in generalization performance - known as the "generalization gap" phenomena. Identifying the origin of this gap and closing it had remained an open problem. Contributions: We examine the initial high learning rate training phase. We find that the weight distance from its initialization grows logarithmically with the number of weight updates. We therefore propose a "random walk on random landscape" statistical model which is known to exhibit similar "ultra-slow" diffusion behavior. Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used. We further investigate different techniques to train models in the large-batch regime and present a novel algorithm named "Ghost Batch Normalization" which enables significant decrease in the generalization gap without increasing the number of updates. To validate our findings we conduct several additional experiments on MNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices and beliefs concerning training of deep models and suggest they may not be optimal to achieve good generalization.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• An overview of gradient descent optimization algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• The Loss Surfaces of Multilayer Networks
• Large Batch Training of Convolutional Networks
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Exponentially vanishing sub-optimal local minima in multilayer neural networks

Cited by in corpus (100)

• Visualizing the Loss Landscape of Neural Nets
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Revisiting Small Batch Training for Deep Neural Networks
• Training Tips for the Transformer Model
• Random Search and Reproducibility for Neural Architecture Search
• Time-to-Event Prediction with Neural Networks and Cox Regression
• Three Factors Influencing Minima in SGD
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Generalization in Deep Learning
• Regularization for Deep Learning: A Taxonomy
• Scaling description of generalization with number of parameters in deep learning
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• An Empirical Model of Large-Batch Training
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• HG-means: A scalable hybrid genetic algorithm for minimum sum-of-squares clustering
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Gravitational-wave surrogate models powered by artificial neural networks: The ANN-Sur for waveform generation
• A Progressive Batching L-BFGS Method for Machine Learning
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Generalization in Deep Networks: The Role of Distance from Initialization
• Augment your batch: better training with larger batches
• Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Traditional and Heavy-Tailed Self Regularization in Neural Network Models
• Understanding Batch Normalization
• An Alternative View: When Does SGD Escape Local Minima?
• Identifying Generalization Properties in Neural Networks
• Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Toward Interpretable Music Tagging with Self-Attention
• A Neural Network Transformer Model for Composite Microstructure Homogenization
• SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
• MegDet: A Large Mini-Batch Object Detector
• On the different regimes of Stochastic Gradient Descent
• EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
• On the diffusion approximation of nonconvex stochastic gradient descent
• Learning to Segment Inputs for NMT Favors Character-Level Processing
• Convergence of Gradient Descent on Separable Data
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Neural Sign Language Translation based on Human Keypoint Estimation
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Generalization in Machine Learning via Analytical Learning Theory
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Pay Attention to the cough: Early Diagnosis of COVID-19 using Interpretable Symptoms Embeddings with Cough Sound Signal Processing
• Stochastic Nonconvex Optimization with Large Minibatches
• The Two Regimes of Deep Network Training
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• How noise affects the Hessian spectrum in overparameterized neural networks
• Full deep neural network training on a pruned weight budget
• On the use of neural networks for the structural characterization of polymeric porous materials
• How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data
• Normalization of Neural Networks using Analytic Variance Propagation
• Strength of Minibatch Noise in SGD
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• Escaping Saddle Points Faster with Stochastic Momentum
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Accelerating Distributed ML Training via Selective Synchronization
• Implicit Regularization via Neural Feature Alignment
• NIPS - Not Even Wrong? A Systematic Review of Empirically Complete Demonstrations of Algorithmic Effectiveness in the Machine Learning and Artificial Intelligence Literature
• Unsupervised Domain Adaptation by Uncertain Feature Alignment
• A Simple Dynamic Learning Rate Tuning Algorithm For Automated Training of DNNs
• Automated Deepfake Detection
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
• Improved generalization by noise enhancement
• Simple Modifications to Improve Tabular Neural Networks
• What training reveals about neural network complexity
• Concurrent Adversarial Learning for Large-Batch Training
• Pre-interpolation loss behaviour in neural networks
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
• Extrapolation for Large-batch Training in Deep Learning
• Genetic-algorithm-optimized neural networks for gravitational wave classification
• A New Benchmark and Progress Toward Improved Weakly Supervised Learning
• Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization
• Self-paced Data Augmentation for Training Neural Networks
• Gradient Sparification for Asynchronous Distributed Training
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• How do SGD hyperparameters in natural training affect adversarial robustness?
• SIPA: A Simple Framework for Efficient Networks
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Why to "grow" and "harvest" deep learning models?
• Data optimization for large batch distributed training of deep neural networks
• Painless step size adaptation for SGD
• Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models
• Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
• LRTuner: A Learning Rate Tuner for Deep Neural Networks
• Understanding How Over-Parametrization Leads to Acceleration: A case of learning a single teacher neuron
• Scalable Kernel Learning via the Discriminant Information
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Logit Attenuating Weight Normalization
• Multilayer Lookahead: a Nested Version of Lookahead