Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
arXiv:1911.08380
Abstract
In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated and non-accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.
arXiv comment18 pages
References in corpus (11)
- An overview of gradient descent optimization algorithms
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm
- A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noise
- Universal gradient descent
- Optimal mini-batch and step sizes for SAGA
- Generalized Mirror Prox for Monotone Variational Inequalities: Universality and Inexact Oracle
- The Complexity of Finding Stationary Points with Stochastic Gradient Descent
- On Primal-Dual Approach for Distributed Stochastic Convex Optimization over Networks
- Optimal Adaptive and Accelerated Stochastic Gradient Descent
- Variance-based stochastic extragradient methods with line search for stochastic variational inequalities
Cited by in corpus (4)
- Large Intelligent Surface Aided Physical Layer Security Transmission
- Cancer-inspired Genomics Mapper Model for the Generation of Synthetic DNA Sequences with Desired Genomics Signatures
- Stochastic Saddle-Point Optimization for Wasserstein Barycenters
- Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization