Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model

Abstract

Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, through two different lenses: large scale experiments, and analysis of a simple noisy quadratic model (NQM). We experimentally demonstrate that optimization algorithms that employ preconditioning, specifically Adam and K-FAC, result in much larger critical batch sizes than stochastic gradient descent with momentum. We also demonstrate that the NQM captures many of the essential features of real neural network training, despite being drastically simpler to work with. The NQM predicts our results with preconditioned optimizers, previous results with accelerated gradient descent, and other results around optimal learning rates and large batch training, making it a useful tool to generate testable predictions about neural network optimization.

NeurIPS 2019

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• No More Pesky Learning Rates
• Three Factors Influencing Minima in SGD
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• L2 Regularization versus Batch and Weight Normalization
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• Critical Hyper-Parameters: No Random, No Cry
• The Effect of Network Width on the Performance of Large-batch Training
• The Physical Systems Behind Optimization Algorithms

Cited by in corpus (5)

• Scaling Laws for Neural Language Models
• Understanding the Role of Training Regimes in Continual Learning
• Multiplicative noise and heavy tails in stochastic optimization
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Two-Level K-FAC Preconditioning for Deep Learning