Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory

Abstract

Several variants of recurrent neural networks (RNNs) with orthogonal or unitary recurrent matrices have recently been developed to mitigate the vanishing/exploding gradient problem and to model long-term dependencies of sequences. However, with the eigenvalues of the recurrent matrix on the unit circle, the recurrent state retains all input information which may unnecessarily consume model capacity. In this paper, we address this issue by proposing an architecture that expands upon an orthogonal/unitary RNN with a state that is generated by a recurrent matrix with eigenvalues in the unit disc. Any input to this state dissipates in time and is replaced with new inputs, simulating short-term memory. A gradient descent algorithm is derived for learning such a recurrent matrix. The resulting method, called the Eigenvalue Normalized RNN (ENRNN), is shown to be highly competitive in several experiments.

References in corpus (8)

• On the difficulty of training Recurrent Neural Networks
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Full-Capacity Unitary Recurrent Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Complex Gated Recurrent Neural Networks
• Improved memory in recurrent neural networks with sequential non-normal dynamics

Cited by in corpus (1)

• Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs