Stochastic Training is Not Necessary for Generalization
arXiv:2109.14119
Abstract
It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization even when comparing against a strong and well-researched baseline. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.
arXiv comment25 pages, 6 figures. Code published at github.com/JonasGeiping/fullbatchtraining. Decompressed version of paper published at ICLR 2022
References in corpus (39)
- On the difficulty of training Recurrent Neural Networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- DARTS: Differentiable Architecture Search
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Large Batch Training of Convolutional Networks
- Revisiting Small Batch Training for Deep Neural Networks
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Fantastic Generalization Measures and Where to Find Them
- An Empirical Model of Large-Batch Training
- Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
- Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
- Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
- Implicit Regularization in Deep Matrix Factorization
- Gradient Descent Can Take Exponential Time to Escape Saddle Points
- The large learning rate phase of deep learning: the catapult mechanism
- On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
- On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
- On the Origin of Implicit Regularization in Stochastic Gradient Descent
- On the Noisy Gradient Descent that Generalizes as SGD
- Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
- A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
- Understanding Generalization through Visualizations
- On the Generalization Benefit of Noise in Stochastic Gradient Descent
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
- On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
- Shape Matters: Understanding the Implicit Bias of the Noise Covariance
- Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent
- SGD Implicitly Regularizes Generalization Error
- Acceleration of Primal-Dual Methods by Preconditioning and Simple Subproblem Procedures
- Are wider nets better given the same number of parameters?
- Extreme Memorization via Scale of Initialization