Riemannian adaptive stochastic gradient algorithms on matrix manifolds

Abstract

Adaptive stochastic gradient algorithms in the Euclidean space have attracted much attention lately. Such explorations on Riemannian manifolds, on the other hand, are relatively new, limited, and challenging. This is because of the intrinsic non-linear structure of the underlying manifold and the absence of a canonical coordinate system. In machine learning applications, however, most manifolds of interest are represented as matrices with notions of row and column subspaces. In addition, the implicit manifold-related constraints may also lie on such subspaces. For example, the Grassmann manifold is the set of column subspaces. To this end, such a rich structure should not be lost by transforming matrices to just a stack of vectors while developing optimization algorithms on manifolds. We propose novel stochastic gradient algorithms for problems on Riemannian matrix manifolds by adapting the row and column subspaces of gradients. Our algorithms are provably convergent and they achieve the convergence rate of order $\mathcal{O}(\log (T)/\sqrt{T})$, where $T$ is the number of iterations. Our experiments illustrate the efficacy of the proposed algorithms on several applications.

In International Conference on Machine Learning (ICML), PMLR 97:3262-3271, 2019

References in corpus (16)

• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Manopt, a Matlab toolbox for optimization on manifolds
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Global rates of convergence for nonconvex optimization on manifolds
• Riemannian Adaptive Optimization Methods
• Adaptive Bound Optimization for Online Convex Optimization
• Geomstats: A Python Package for Riemannian Geometry in Machine Learning
• Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
• Averaging Stochastic Gradient Descent on Riemannian Manifolds
• McTorch, a manifold optimization library for deep learning
• A dual framework for low-rank tensor completion
• Minimum weight norm models do not always generalize well for over-parameterized problems

Cited by in corpus (8)

• A Riemannian Block Coordinate Descent Method for Computing the Projection Robust Wasserstein Distance
• No-go Theorem for Acceleration in the Hyperbolic Plane
• Variance reduction for Riemannian non-convex optimization with batch size adaptation
• Projection Robust Wasserstein Distance and Riemannian Optimization
• Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing
• Global Riemannian Acceleration in Hyperbolic and Spherical Spaces
• Riemannian stochastic recursive momentum method for non-convex optimization
• Manifold Optimization Assisted Gaussian Variational Approximation