AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods
arXiv:2102.09700
Abstract
We present AI-SARAH, a practical variant of SARAH. As a variant of SARAH, this algorithm employs the stochastic recursive gradient yet adjusts step-size based on local geometry. AI-SARAH implicitly computes step-size and efficiently estimates local Lipschitz smoothness of stochastic functions. It is fully adaptive, tune-free, straightforward to implement, and computationally efficient. We provide technical insight and intuitive illustrations on its design and convergence. We conduct extensive empirical analysis and demonstrate its strong performance compared with its classical counterparts and other state-of-the-art first-order methods in solving convex machine learning problems.
References in corpus (10)
- SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
- Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
- AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
- Barzilai-Borwein Step Size for Stochastic Gradient Descent
- A Simple Practical Accelerated Method for Finite Sums
- Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
- Linearly convergent stochastic heavy ball method for minimizing generalization error
- Stochastic Hamiltonian Gradient Methods for Smooth Games
- SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation
- Adaptive Step Sizes in Variance Reduction via Regularization