Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

Abstract

We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2/\text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.

arXiv commentICLR 2021. v3 moves several figures from the appendix into the main text, and adds more discussion regarding Jastrzębski et al (2020): https://doi.org/10.48550/arXiv.2002.09572

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Pointer Sentinel Mixture Models
• No More Pesky Learning Rates
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Three Factors Influencing Minima in SGD
• AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
• The large learning rate phase of deep learning: the catapult mechanism
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?

Cited by in corpus (13)

• Automated Learning Rate Scheduler for Large-batch Training
• Leveraging Non-uniformity in First-order Non-convex Optimization
• Label Noise SGD Provably Prefers Flat Global Minimizers
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Stochastic Training is Not Necessary for Generalization
• Continuous vs. Discrete Optimization of Deep Neural Networks
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Acceleration via Fractal Learning Rate Schedules
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Rethinking the Variational Interpretation of Nesterov's Accelerated Method
• Neurons learn slower than they think