Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems
arXiv:2005.07360
Abstract
Learning rate schedule can significantly affect generalization performance in modern neural networks, but the reasons for this are not yet understood. Li-Wei-Ma (2019) recently proved this behavior can exist in a simplified non-convex neural-network setting. In this note, we show that this phenomenon can exist even for convex learning problems -- in particular, linear regression in 2 dimensions. We give a toy convex problem where learning rate annealing (large initial learning rate, followed by small learning rate) can lead gradient descent to minima with provably better generalization than using a small learning rate throughout. In our case, this occurs due to a combination of the mismatch between the test and train loss landscapes, and early-stopping.
4 pages plus appendix
References in corpus (8)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- One weird trick for parallelizing convolutional neural networks
- An Empirical Model of Large-Batch Training
- Sharp Minima Can Generalize For Deep Nets
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- The large learning rate phase of deep learning: the catapult mechanism
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks