Trivializations for Gradient-Based Optimization on Manifolds

Abstract

We introduce a framework to study the transformation of problems with manifold constraints into unconstrained problems through parametrizations in terms of a Euclidean space. We call these parametrizations "trivializations". We prove conditions under which a trivialization is sound in the context of gradient-based optimization and we show how two large families of trivializations have overall favorable properties, but also suffer from a performance issue. We then introduce "dynamic trivializations", which solve this problem, and we show how these form a family of optimization methods that lie between trivializations and Riemannian gradient descent, and combine the benefits of both of them. We then show how to implement these two families of trivializations in practice for different matrix manifolds. To this end, we prove a formula for the gradient of the exponential of matrices, which can be of practical interest on its own. Finally, we show how dynamic trivializations improve the performance of existing methods on standard tasks designed to test long-term memory within neural networks.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Global rates of convergence for nonconvex optimization on manifolds
• Geometric Optimization Methods for Adaptive Filtering
• Sylvester Normalizing Flows for Variational Inference
• Full-Capacity Unitary Recurrent Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Improving Variational Auto-Encoders using Householder Flow
• Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
• Towards Riemannian Accelerated Gradient Methods
• Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform

Cited by in corpus (10)

• Sampling using $SU(N)$ gauge equivariant flows
• Lipschitz Recurrent Neural Networks
• Neural Canonical Transformation with Symplectic Flows
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• Scaling Scaling Laws with Board Games
• Solving Optimization Problems over the Stiefel Manifold by Smooth Exact Penalty Function
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Training Invertible Linear Layers through Rank-One Perturbations
• Unraveling the Mechanism of Drug Binding to SARS-CoV-2 RNA Pseudoknot with Thermodynamics-Driven Machine Learning