A Loss Curvature Perspective on Training Instability in Deep Learning

Abstract

In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning rates affect the loss Hessian observed during training, we also analyze the effects of model initialization, architectural choices, and common training heuristics such as gradient clipping and learning rate warmup. Our results demonstrate that successful model and hyperparameter choices allow the early optimization trajectory to either avoid -- or navigate out of -- regions of high curvature and into flatter regions that tolerate a higher learning rate. Our results suggest a unifying perspective on how disparate mitigation strategies for training instability ultimately address the same underlying failure mode of neural network optimization, namely poor conditioning. Inspired by the conditioning perspective, we show that learning rate warmup can improve training stability just as much as batch normalization, layer normalization, MetaInit, GradInit, and Fixup initialization.

arXiv comment20 pages, 16 figures

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• On the difficulty of training Recurrent Neural Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Three Factors Influencing Minima in SGD
• On Empirical Comparisons of Optimizers for Deep Learning
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Carbon Emissions and Large Neural Network Training
• Fixup Initialization: Residual Learning Without Normalization
• On Layer Normalization in the Transformer Architecture
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• The large learning rate phase of deep learning: the catapult mechanism
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Characterizing signal propagation to close the performance gap in unnormalized ResNets