Statistical Adaptive Stochastic Gradient Methods

Abstract

We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods. SALSA first uses a smoothed stochastic line-search procedure to gradually increase the learning rate, then automatically switches to a statistical method to decrease the learning rate. The line search procedure ``warms up'' the optimization process, reducing the need for expensive trial and error in setting an initial learning rate. The method for decreasing the learning rate is based on a new statistical test for detecting stationarity when using a constant step size. Unlike in prior work, our test applies to a broad class of stochastic gradient algorithms without modification. The combined method is highly robust and autonomous, and it matches the performance of the best hand-tuned learning rate schedules in our experiments on several deep learning tasks.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Pointer Sentinel Mixture Models
• Batch means and spectral variance estimators in Markov chain Monte Carlo
• Accelerating Stochastic Gradient Descent For Least Squares Regression
• Fluctuation-dissipation relations for stochastic gradient descent
• Using Statistics to Automate Stochastic Optimization

Cited by in corpus (6)

• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Automated Learning Rate Scheduler for Large-batch Training
• On the Convergence of Step Decay Step-Size for Stochastic Optimization
• Half-Space Proximal Stochastic Gradient Method for Group-Sparsity Regularized Problem
• Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
• Stochastic Polyak Stepsize with a Moving Target