On the Convergence of Step Decay Step-Size for Stochastic Optimization
arXiv:2102.09393
Abstract
The convergence of stochastic gradient descent is highly dependent on the step-size, especially on non-convex problems such as neural network training. Step decay step-size schedules (constant and then cut) are widely used in practice because of their excellent convergence and generalization qualities, but their theoretical properties are not yet well understood. We provide the convergence results for step decay in the non-convex regime, ensuring that the gradient norm vanishes at an rate. We also provide the convergence guarantees for general (possibly non-smooth) convex problems, ensuring an convergence rate. Finally, in the strongly convex case, we establish an rate for smooth problems, which we also prove to be tight, and an rate without the smoothness assumption. We illustrate the practical efficiency of the step decay step-size in several large scale deep neural network training tasks.
References in corpus (11)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Decoupled Weight Decay Regularization
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
- On the Number of Iterations for Dantzig-Wolfe Optimization and Packing-Covering Approximation Algorithms
- Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
- Using Statistics to Automate Stochastic Optimization
- Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition
- Statistical Adaptive Stochastic Gradient Methods
- From low probability to high confidence in stochastic convex optimization