Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

Abstract

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.

References in corpus (1)

• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

Cited by in corpus (11)

• ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization
• Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Rates
• One Sample Stochastic Frank-Wolfe
• Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
• Momentum Schemes with Stochastic Variance Reduction for Nonconvex Composite Optimization
• Variance reduction for Riemannian non-convex optimization with batch size adaptation
• Momentum with Variance Reduction for Nonconvex Composition Optimization
• Variance-Reduced Off-Policy Memory-Efficient Policy Search
• Escape saddle points faster on manifolds via perturbed Riemannian stochastic recursive gradient
• Accelerating Mini-batch SARAH by Step Size Rules
• DTN: A Learning Rate Scheme with Convergence Rate of $\mathcal{O}(1/t)$ for SGD