Full-Capacity Unitary Recurrent Neural Networks

Abstract

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be learned? To address this question, we propose full-capacity uRNNs that optimize their recurrence matrix over all unitary matrices, leading to significantly improved performance over uRNNs that use a restricted-capacity recurrence matrix. Our contribution consists of two main components. First, we provide a theoretical argument to determine if a unitary parameterization has restricted capacity. Using this argument, we show that a recently proposed unitary parameterization has restricted capacity for hidden state dimension greater than 7. Second, we show how a complete, full-capacity unitary recurrence matrix can be optimized over the differentiable manifold of unitary matrices. The resulting multiplicative gradient step is very simple and does not require gradient clipping or learning rate adaptation. We confirm the utility of our claims by empirically evaluating our new full-capacity uRNNs on both synthetic and natural data, achieving superior performance compared to both LSTMs and the original restricted-capacity uRNNs.

arXiv comment9 pages, to appear in NIPS

References in corpus (4)

• On the difficulty of training Recurrent Neural Networks
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Recurrent Models of Visual Attention
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Cited by in corpus (56)

• Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks
• Concept Whitening for Interpretable Image Recognition
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Stable Recurrent Models
• R-Transformer: Recurrent Neural Network Enhanced Transformer
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Sorting out Lipschitz function approximation
• Parseval Proximal Neural Networks
• Lipschitz Recurrent Neural Networks
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Trivializations for Gradient-Based Optimization on Manifolds
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Continual Learning in Low-rank Orthogonal Subspaces
• Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
• Neural Rough Differential Equations for Long Time Series
• The Echo Index and multistability in input-driven recurrent neural networks
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Recurrent Quantum Neural Networks
• Can recurrent neural networks warp time?
• Heavy Ball Neural Ordinary Differential Equations
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• 3D-Rotation-Equivariant Quaternion Neural Networks
• UnICORNN: A recurrent model for learning very long time dependencies
• Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
• Orthogonal Convolutional Neural Networks
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• Orthogonal Over-Parameterized Training
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform
• Encoding-based Memory Modules for Recurrent Neural Networks
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• MC-LSTM: Mass-Conserving LSTM
• Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• Orthogonal Wasserstein GANs
• A Differential Geometry Perspective on Orthogonal Recurrent Models
• Controllable Orthogonalization in Training DNNs
• Learning to Adaptively Scale Recurrent Neural Networks
• On regularization for a convolutional kernel in neural networks
• Memory and attention in deep learning
• Long Expressive Memory for Sequence Modeling
• Neural Networks with Complex-Valued Weights Have No Spurious Local Minima
• Deep Unitary Convolutional Neural Networks
• Implicit Bias of Linear RNNs
• Quantum-inspired Multimodal Fusion for Video Sentiment Analysis
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• Sequence Prediction using Spectral RNNs
• Depth Enables Long-Term Memory for Recurrent Neural Networks
• Utilizing Complex-valued Network for Learning to Compare Image Patches
• CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
• Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Complex Evolution Recurrent Neural Networks (ceRNNs)
• Learning with Hyperspherical Uniformity