Stochastic gradient descent on Riemannian manifolds

Abstract

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Euclidian case, the gradient descent algorithm converges to a critical point of the cost function. The algorithm has numerous potential applications, and is illustrated here by four examples. In particular a novel gossip algorithm on the set of covariance matrices is derived and tested numerically.

A slightly shorter version has been published in IEEE Transactions Automatic Control

Cited by in corpus (35)

• An elementary introduction to information geometry
• Riemannian conjugate gradient methods: General framework and specific algorithms with convergence analyses
• Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
• HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric Regularization
• Product Knowledge Graph Embedding for E-commerce
• HICF: Hyperbolic Informative Collaborative Filtering
• Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering
• Walking on the Edge: Fast, Low-Distortion Adversarial Examples
• Beyond Convexity -- Contraction and Global Convergence of Gradient Descent
• Optimizing quantum circuits with Riemannian gradient flow
• Adaptive regularization with cubics on manifolds
• Performance of Hyperbolic Geometry Models on Top-N Recommendation Tasks
• Unsupervised Deep Metric Learning via Orthogonality based Probabilistic Loss
• Computation for Latent Variable Model Estimation: A Unified Stochastic Proximal Framework
• Hyperbolic Geometric Graph Representation Learning for Hierarchy-imbalance Node Classification
• Dual-Geometric Space Embedding Model for Two-View Knowledge Graphs
• Hyperbolic Node Embedding for Signed Networks
• Learning Co-Sparse Analysis Operators with Separable Structures
• Fast, asymptotically efficient, recursive estimation in a Riemannian manifold
• Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data
• A stochastic algorithm finding $p$-means on the circle
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Riemannian Smoothing Gradient Type Algorithms]{Riemannian Smoothing Gradient Type Algorithms for Nonsmooth Optimization Problem on Compact Riemannian Submanifold Embedded in Euclidean Space
• Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning
• Network Consensus in the Wasserstein Metric Space of Probability Measures
• A universal framework for learning the elliptical mixture model
• Efficient Random Walks on Riemannian Manifolds
• Linear Convergence of the Subspace Constrained Mean Shift Algorithm: From Euclidean to Directional Data
• Spherical Coordinates from Persistent Cohomology
• Riemannian Stochastic Variance-Reduced Cubic Regularized Newton Method for Submanifold Optimization
• Stochastic Modified Flows for Riemannian Stochastic Gradient Descent
• Convergence of variational Monte Carlo simulation and scale-invariant pre-training
• Hyperbolic recurrent neural network as the first type of non-Euclidean neural quantum state ansatz
• Stochastic Augmented Lagrangian Method in Riemannian Shape Manifolds
• Trainable Quantum Neural Network for Multiclass Image Classification with the Power of Pre-trained Tree Tensor Networks