The large learning rate phase of deep learning: the catapult mechanism

Abstract

The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confirm their predictions empirically in practical deep learning settings. The networks exhibit sharply distinct behaviors at small and large learning rates. The two regimes are separated by a phase transition. In the small learning rate phase, training can be understood using the existing theory of infinitely wide neural networks. At large learning rates the model captures qualitatively distinct phenomena, including the convergence of gradient descent dynamics to flatter minima. One key prediction of our model is a narrow range of large, stable learning rates. We find good agreement between our model's predictions and training dynamics in realistic deep learning settings. Furthermore, we find that the optimal performance in such settings is often found in the large learning rate phase. We believe our results shed light on characteristics of models trained at different learning rates. In particular, they fill a gap between existing wide neural network theory, and the nonlinear, large learning rate, training dynamics relevant to practice.

arXiv comment25 pages, 19 figures

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Sharp Minima Can Generalize For Deep Nets
• SGD Learns the Conjugate Kernel Class of the Network
• Kernel and Rich Regimes in Overparametrized Models
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• The Two Regimes of Deep Network Training
• Stochastic natural gradient descent draws posterior samples in function space

Cited by in corpus (44)

• Scaling Laws for Autoregressive Generative Modeling
• Explaining Neural Scaling Laws
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Learning to Optimize: A Primer and A Benchmark
• Understanding the Role of Training Regimes in Continual Learning
• On the Origin of Implicit Regularization in Stochastic Gradient Descent
• Finite Versus Infinite Neural Networks: an Empirical Study
• The Heavy-Tail Phenomenon in SGD
• On the training dynamics of deep networks with $L_2$ regularization
• Deep Learning on a Data Diet: Finding Important Examples Early in Training
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Feature Learning in Infinite-Width Neural Networks
• How to decay your learning rate
• Speedy Performance Estimation for Neural Architecture Search
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Deep Neural Networks for Detecting Statistical Model Misspecifications. The Case of Measurement Invariance
• Differentiable Physics: A Position Piece
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Stochastic Training is Not Necessary for Generalization
• Implicit Gradient Regularization
• Universal scaling laws in the gradient descent training of neural networks
• Continuous vs. Discrete Optimization of Deep Neural Networks
• Implicit bias of deep linear networks in the large learning rate phase
• Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Transformed CNNs: recasting pre-trained convolutional layers with self-attention
• Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Properties of the After Kernel
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data
• Multilayer Lookahead: a Nested Version of Lookahead
• Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change