Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic
arXiv:1910.08597
Abstract
This paper proposes SplitSGD, a new dynamic learning rate schedule for stochastic optimization. This method decreases the learning rate for better adaptation to the local geometry of the objective function whenever a stationary phase is detected, that is, the iterates are likely to bounce at around a vicinity of a local minimum. The detection is performed by splitting the single thread into two and using the inner product of the gradients from the two threads as a measure of stationarity. Owing to this simple yet provably valid stationarity detection, SplitSGD is easy-to-implement and essentially does not incur additional computational cost than standard SGD. Through a series of extensive experiments, we show that this method is appropriate for both convex problems and training (non-convex) neural networks, with performance compared favorably to other stochastic optimization methods. Importantly, this method is observed to be very robust with a set of default parameters for a wide range of problems and, moreover, can yield better generalization performance than other adaptive gradient methods such as Adam.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- An overview of gradient descent optimization algorithms
- Don't Decay the Learning Rate, Increase the Batch Size
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Practical recommendations for gradient-based training of deep architectures
- An Empirical Model of Large-Batch Training
- Fluctuation-dissipation relations for stochastic gradient descent
- HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
- Coupling Adaptive Batch Sizes with Learning Rates
- Gradient Diversity: a Key Ingredient for Scalable Distributed Learning