Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization

Abstract

In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated and non-accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.

18 pages

References in corpus (2)

• A Universal Algorithm for Variational Inequalities Adaptive to Smoothness and Noise
• The Complexity of Finding Stationary Points with Stochastic Gradient Descent

Cited by in corpus (4)

• Large Intelligent Surface Aided Physical Layer Security Transmission
• Cancer-inspired Genomics Mapper Model for the Generation of Synthetic DNA Sequences with Desired Genomics Signatures
• Stochastic Saddle-Point Optimization for Wasserstein Barycenters
• Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization