On Riemannian Stochastic Approximation Schemes with Fixed Step-Size
arXiv:2102.07586
Abstract
This paper studies fixed step-size stochastic approximation (SA) schemes, including stochastic gradient schemes, in a Riemannian framework. It is motivated by several applications, where geodesics can be computed explicitly, and their use accelerates crude Euclidean methods. A fixed step-size scheme defines a family of time-homogeneous Markov chains, parametrized by the step-size. Here, using this formulation, non-asymptotic performance bounds are derived, under Lyapunov conditions. Then, for any step-size, the corresponding Markov chain is proved to admit a unique stationary distribution, and to be geometrically ergodic. This result gives rise to a family of stationary distributions indexed by the step-size, which is further shown to converge to a Dirac measure, concentrated at the solution of the problem at hand, as the step-size goes to 0. Finally, the asymptotic rate of this convergence is established, through an asymptotic expansion of the bias, and a central limit theorem.
arXiv comment37 pages, 4 figures, to appear in AISTAT21
References in corpus (10)
- Online EM Algorithm for Latent Data Models
- Complete Dictionary Recovery over the Sphere II: Recovery by Riemannian Trust-region Method
- The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
- Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
- Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
- Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
- On the Effectiveness of Richardson Extrapolation in Machine Learning
- Variance reduction for Riemannian non-convex optimization with batch size adaptation
- Convergence Analysis of Riemannian Stochastic Approximation Schemes
- Stochastic Primal-Dual Method on Riemannian Manifolds with Bounded Sectional Curvature