Revisiting the Polyak step size
arXiv:1905.00313
Abstract
This paper revisits the Polyak step size schedule for convex optimization problems, proving that a simple variant of it simultaneously attains near optimal convergence rates for the gradient descent algorithm, for all ranges of strong convexity, smoothness, and Lipschitz parameters, without a-priory knowledge of these parameters.
Cited by in corpus (8)
- Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
- On the Adaptivity of Stochastic Gradient-Based Optimization
- Adaptive Gradient Descent without Descent
- Training Neural Networks for and by Interpolation
- Complexity Guarantees for Polyak Steps with Momentum
- Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descent
- Acceleration via Fractal Learning Rate Schedules
- Approximately Exact Line Search