Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Abstract

We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Global rates of convergence for nonconvex optimization on manifolds
• On orthogonality and learning recurrent networks with long term dependencies
• Riemannian Adaptive Optimization Methods
• Geometric Optimization Methods for Adaptive Filtering
• Full-Capacity Unitary Recurrent Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds
• Riemannian stochastic variance reduced gradient algorithm with retraction and vector transport
• Generalized BackPropagation, Étude De Cas: Orthogonality
• Optimization on Submanifolds of Convolution Kernels in CNNs
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform

Cited by in corpus (38)

• Efficiently Modeling Long Sequences with Structured State Spaces
• Normalizing Flows for Probabilistic Modeling and Inference
• Concept Whitening for Interpretable Image Recognition
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Sampling using $SU(N)$ gauge equivariant flows
• Quantum Methods for Neural Networks and Application to Medical Image Classification
• Parseval Proximal Neural Networks
• Lipschitz Recurrent Neural Networks
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Trivializations for Gradient-Based Optimization on Manifolds
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Here comes the SU(N): multivariate quantum gates and gradients
• Classical and Quantum Algorithms for Orthogonal Neural Networks
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• CKConv: Continuous Kernel Convolution For Sequential Data
• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• Orthogonalizing Convolutional Layers with the Cayley Transform
• UnICORNN: A recurrent model for learning very long time dependencies
• NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation
• Orthogonal Convolutional Neural Networks
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Noisy Recurrent Neural Networks
• Untangling tradeoffs between recurrence and self-attention in neural networks
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Orthogonal Over-Parameterized Training
• What if Neural Networks had SVDs?
• Global Riemannian Acceleration in Hyperbolic and Spherical Spaces
• Solving Optimization Problems over the Stiefel Manifold by Smooth Exact Penalty Function
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• RotoGrad: Gradient Homogenization in Multitask Learning
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Deep Unitary Convolutional Neural Networks
• One Reflection Suffice
• A Projection Algorithm for the Unitary Weights
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization
• Spectral Tensor Train Parameterization of Deep Learning Layers