Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks

Abstract

Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes easy-to-generalize, hard-to-fit patterns, it generalizes worse on hard-to-generalize, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• A Convergence Theory for Deep Learning via Over-Parameterization
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Large Batch Training of Convolutional Networks
• Improving Generalization Performance by Switching from Adam to SGD
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On the Convergence Rate of Training Recurrent Neural Networks
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares
• A Walk with SGD
• SGD on Neural Networks Learns Functions of Increasing Complexity
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent

Cited by in corpus (35)

• Shortcut Learning in Deep Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Mathematical Models of Overparameterized Neural Networks
• Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• How to decay your learning rate
• Transforming task representations to perform novel tasks
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• The Two Regimes of Deep Network Training
• False Sense of Security: Leveraging XAI to Analyze the Reasoning and True Performance of Context-less DGA Classifiers
• Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Stochastic Training is Not Necessary for Generalization
• Constraint-Based Regularization of Neural Networks
• Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect
• Stochastic Anderson Mixing for Nonconvex Stochastic Optimization
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Loss Barcode: A Topological Measure of Escapability in Loss Landscapes
• Extrapolation for Large-batch Training in Deep Learning
• Acceleration via Fractal Learning Rate Schedules
• Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Adaptive Margin Circle Loss for Speaker Verification
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Catapult Dynamics and Phase Transitions in Quadratic Nets
• A study on the plasticity of neural networks
• Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data
• Multilayer Lookahead: a Nested Version of Lookahead
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control