Stochastic Training is Not Necessary for Generalization

Abstract

It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern architectures. To this end, we show that the implicit regularization of SGD can be completely replaced with explicit regularization even when comparing against a strong and well-researched baseline. Our observations indicate that the perceived difficulty of full-batch training may be the result of its optimization properties and the disproportionate time and effort spent by the ML community tuning optimizers and hyperparameters for small-batch training.

25 pages, 6 figures. Code published at github.com/JonasGeiping/fullbatchtraining. Decompressed version of paper published at ICLR 2022

References in corpus (15)

• On the difficulty of training Recurrent Neural Networks
• Large Batch Training of Convolutional Networks
• Fantastic Generalization Measures and Where to Find Them
• An Empirical Model of Large-Batch Training
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• The large learning rate phase of deep learning: the catapult mechanism
• On the Origin of Implicit Regularization in Stochastic Gradient Descent
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• SGD Implicitly Regularizes Generalization Error

Cited by in corpus (3)

• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Depth Without the Magic: Inductive Bias of Natural Gradient Descent
• Subspace Adversarial Training