Riemannian Adaptive Optimization Methods

Abstract

Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools - namely Adam , Adagrad and the more recent Amsgrad - remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincare ball.

arXiv commentAccepted at International Conference on Learning Representations (ICLR), 2019

References in corpus (7)

• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Conic geometric optimisation on the manifold of positive definite matrices
• Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
• Averaging Stochastic Gradient Descent on Riemannian Manifolds
• Towards Riemannian Accelerated Gradient Methods
• Riemannian Stein Variational Gradient Descent for Bayesian Inference

Cited by in corpus (33)

• Poincaré GloVe: Hyperbolic Word Embeddings
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Graph Neural Networks on SPD Manifolds for Motor Imagery Classification: A Perspective from the Time-Frequency Analysis
• Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs
• Riemannian geometry and automatic differentiation for optimization problems of quantum physics and quantum technologies
• Graph-based Modeling of Online Communities for Fake News Detection
• Riemannian Continuous Normalizing Flows
• Multi-relational Poincaré Graph Embeddings
• Riemannian adaptive stochastic gradient algorithms on matrix manifolds
• HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems
• Hyperbolic Deep Neural Networks: A Survey
• Quaternion Equivariant Capsule Networks for 3D Point Clouds
• Hyperbolic Interaction Model For Hierarchical Multi-Label Classification
• Efficient Quantum Circuit Compilation for Near-Term Quantum Advantage
• Time Series Alignment with Global Invariances
• Solving general elliptical mixture models through an approximate Wasserstein manifold
• Hyperbolic Busemann Learning with Ideal Prototypes
• Projection Robust Wasserstein Distance and Riemannian Optimization
• Tensor Analysis with n-Mode Generalized Difference Subspace
• Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing
• Unsupervised Hierarchy Matching with Optimal Transport over Hyperbolic Spaces
• Solving Optimization Problems over the Stiefel Manifold by Smooth Exact Penalty Function
• DeepHGCN: Toward Deeper Hyperbolic Graph Convolutional Networks
• Hyperbolic Neural Networks++
• Unit Ball Model for Embedding Hierarchical Structures in the Complex Hyperbolic Space
• HypLL: The Hyperbolic Learning Library
• A Riemannian Primal-dual Algorithm Based on Proximal Operator and its Application in Metric Learning
• Extracting Event Temporal Relations via Hyperbolic Geometry
• Manifold Optimization Assisted Gaussian Variational Approximation
• GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition
• Hermitian Symmetric Spaces for Graph Embeddings
• HyperText: Endowing FastText with Hyperbolic Geometry