Catapult Dynamics and Phase Transitions in Quadratic Nets

Abstract

Neural networks trained with gradient descent can undergo non-trivial phase transitions as a function of the learning rate. In \cite{lewkowycz2020large} it was discovered that wide neural nets can exhibit a catapult phase for super-critical learning rates, where the training loss grows exponentially quickly at early times before rapidly decreasing to a small value. During this phase the top eigenvalue of the neural tangent kernel (NTK) also undergoes significant evolution. In this work, we will prove that the catapult phase exists in a large class of models, including quadratic models and two-layer, homogenous neural nets. To do this, we show that for a certain range of learning rates the weight norm decreases whenever the loss becomes large. We also empirically study learning rates beyond this theoretically derived range and show that the activation map of ReLU nets trained with super-critical learning rates becomes increasingly sparse as we increase the learning rate.

46 pages, many figures

References in corpus (19)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Phase Retrieval via Wirtinger Flow: Theory and Algorithms
• Deep Neural Networks as Gaussian Processes
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• A Geometric Analysis of Phase Retrieval
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Asymptotics of Wide Networks from Feynman Diagrams
• Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
• Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• On the Maximum Hessian Eigenvalue and Generalization
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• SGD with Large Step Sizes Learns Sparse Features
• Implicit bias of deep linear networks in the large learning rate phase
• Maximal Initial Learning Rates in Deep ReLU Networks
• Quadratic models for understanding catapult dynamics of neural networks
• On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions
• Second-order regression models exhibit progressive sharpening to the edge of stability