On orthogonality and learning recurrent networks with long term dependencies

Abstract

It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.

References in corpus (7)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Unitary Evolution Recurrent Neural Networks
• Generalized BackPropagation, Étude De Cas: Orthogonality
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• Regularizing RNNs by Stabilizing Activations
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

Cited by in corpus (47)

• Concept Whitening for Interpretable Image Recognition
• Stable Recurrent Models
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Analysis and Optimization of Convolutional Neural Network Architectures
• Parseval Proximal Neural Networks
• Lipschitz Recurrent Neural Networks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
• Continual Learning with Gated Incremental Memories for sequential data processing
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
• HUBERT Untangles BERT to Improve Transfer across NLP Tasks
• Deep Learning Methods for Vessel Trajectory Prediction based on Recurrent Neural Networks
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Projection Based Weight Normalization for Deep Neural Networks
• On the Initialization of Long Short-Term Memory Networks
• Continual Learning in Recurrent Neural Networks
• Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies
• How Robust are Deep Neural Networks?
• Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding
• Kronecker Recurrent Units
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Information Geometry of Orthogonal Initializations and Training
• Orthogonal Convolutional Neural Networks
• Constraint-Based Regularization of Neural Networks
• A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Deep Learning for Inverse Problems: Bounds and Regularizers
• Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform
• RePr: Improved Training of Convolutional Filters
• On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
• Controllable Orthogonalization in Training DNNs
• Rotational Unit of Memory
• Deep Unitary Convolutional Neural Networks
• Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
• Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization
• Push for Center Learning via Orthogonalization and Subspace Masking for Person Re-Identification