Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training

Abstract

We study the effect of mini-batching on the loss landscape of deep neural networks using spiked, field-dependent random matrix theory. We demonstrate that the magnitude of the extremal values of the batch Hessian are larger than those of the empirical Hessian. We also derive similar results for the Generalised Gauss-Newton matrix approximation of the Hessian. As a consequence of our theorems we derive an analytical expressions for the maximal learning rates as a function of batch size, informing practical training regimens for both stochastic gradient descent (linear scaling) and adaptive algorithms, such as Adam (square root scaling), for smooth, non-convex deep neural networks. Whilst the linear scaling for stochastic gradient descent has been derived under more restrictive conditions, which we generalise, the square root scaling rule for adaptive optimisers is, to our knowledge, completely novel. %For stochastic second-order methods and adaptive methods, we derive that the minimal damping coefficient is proportional to the ratio of the learning rate to batch size. We validate our claims on the VGG/WideResNet architectures on the CIFAR-$100$ and ImageNet datasets. Based on our investigations of the sub-sampled Hessian we develop a stochastic Lanczos quadrature based on the fly learning rate and momentum learner, which avoids the need for expensive multiple evaluations for these key hyper-parameters and shows good preliminary results on the Pre-Residual Architecure for CIFAR-$100$.

References in corpus (38)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• The Loss Surfaces of Multilayer Networks
• Visualizing the Loss Landscape of Neural Nets
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• New insights and perspectives on the natural gradient method
• Three Factors Influencing Minima in SGD
• On Empirical Comparisons of Optimizers for Deep Learning
• A Linearly-Convergent Stochastic L-BFGS Algorithm
• Measuring the Effects of Data Parallelism on Neural Network Training
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Distributions of Angles in Random Packing on Spheres
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Gradient Descent Happens in a Tiny Subspace
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• Krylov Subspace Descent for Deep Learning
• The large learning rate phase of deep learning: the catapult mechanism
• Sub-Sampled Newton Methods II: Local Convergence Rates
• Four Things Everyone Should Know to Improve Batch Normalization
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• A note on the Marchenko-Pastur law for a class of random matrices with dependent entries
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Deep Frank-Wolfe For Neural Network Optimization
• Semicircle Law for a Class of Random Matrices with Dependent Entries
• An information and field theoretic approach to the grand canonical ensemble
• Deep Curvature Suite

Cited by in corpus (3)

• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
• Efficient fine-tuning of 37-level GraphCast with the Canadian global deterministic analysis