Second-order step-size tuning of SGD for non-convex optimization
arXiv:2103.03570 · doi:10.1007/s11063-021-10705-5
Abstract
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case. For doing so, one estimates curvature, based on a local quadratic model and using only noisy gradient approximations. One obtains a new stochastic first-order method (Step-Tuned SGD), enhanced by second-order information, which can be seen as a stochastic version of the classical Barzilai-Borwein method. Our theoretical results ensure almost sure convergence to the critical set and we provide convergence rates. Experiments on deep residual network training illustrate the favorable properties of our approach. For such networks we observe, during training, both a sudden drop of the loss and an improvement of test accuracy at medium stages, yielding better results than SGD, RMSprop, or ADAM.
arXiv commentTo appear in Neural Processing Letters (accepted Nov. 2021)
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- The NumPy array: a structure for efficient numerical computation
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Barzilai-Borwein Step Size for Stochastic Gradient Descent
- Natasha 2: Faster Non-Convex Optimization Than SGD
- A mathematical model for automatic differentiation in machine learning
- On Noisy Negative Curvature Descent: Competing with Gradient Descent for Faster Non-convex Optimization
- Incorporating the Barzilai-Borwein Adaptive Step Size into Sugradient Methods for Deep Network Training