Riemannian stochastic recursive momentum method for non-convex optimization

Abstract

We propose a stochastic recursive momentum method for Riemannian non-convex optimization that achieves a near-optimal complexity of $\tilde{\mathcal{O}}({\epsilon }^{-3})$ to find $\epsilon$-approximate solution with one sample. That is, our method requires $\mathcal{O}(1)$ gradient evaluations per iteration and does not require restarting with a large batch gradient, which is commonly used to obtain the faster rate. Extensive experiment results demonstrate the superiority of our proposed algorithm.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Manopt, a Matlab toolbox for optimization on manifolds
• Kernel Methods on Riemannian Manifolds with Gaussian RBF Kernels
• Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
• Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
• R-SPIDER: A Fast Riemannian Stochastic Optimization Algorithm with Curvature Independent Rate
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds
• An Alternative to EM for Gaussian Mixture Models: Batch and Stochastic Riemannian Optimization
• Variance reduction for Riemannian non-convex optimization with batch size adaptation