Recurrent Orthogonal Networks and Long-Memory Tasks

Abstract

Although RNNs have been shown to be powerful tools for processing sequential data, finding architectures or optimization strategies that allow them to model very long term dependencies is still an active area of research. In this work, we carefully analyze two synthetic datasets originally outlined in (Hochreiter and Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store information over many time steps. We explicitly construct RNN solutions to these problems, and using these constructions, illuminate both the problems themselves and the way in which RNNs store different types of information in their hidden states. These constructions furthermore explain the success of recent methods that specify unitary initializations or constraints on the transition matrices.

Cited by in corpus (18)

• Review: Deep Learning in Electron Microscopy
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Residual Connections Encourage Iterative Inference
• Lipschitz Recurrent Neural Networks
• Machine Learning on Sequential Data Using a Recurrent Weighted Average
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Improving the Gating Mechanism of Recurrent Neural Networks
• Continual Learning with Gated Incremental Memories for sequential data processing
• Orthogonal Recurrent Neural Networks with Scaled Cayley Transform
• Information Geometry of Orthogonal Initializations and Training
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Orthogonal Over-Parameterized Training
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks
• Rotational Unit of Memory
• Mapping high-performance RNNs to in-memory neuromorphic chips
• Optimization on Product Submanifolds of Convolution Kernels
• Stochastic Flows and Geometric Optimization on the Orthogonal Group