Higher Order Recurrent Neural Networks

Abstract

In this paper, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted paths. By extending the popular recurrent structure in RNNs, we provide the models with better short-term memory mechanism to learn long term dependency in sequences. Analogous to digital filters in signal processing, we call these structures as higher order RNNs (HORNNs). Similar to RNNs, HORNNs can also be learned using the back-propagation through time method. HORNNs are generally applicable to a variety of sequence modelling tasks. In this work, we have examined HORNNs for the language modeling task using two popular data sets, namely the Penn Treebank (PTB) and English text8 data sets. Experimental results have shown that the proposed HORNNs yield the state-of-the-art performance on both data sets, significantly outperforming the regular RNNs as well as the popular LSTMs.

9 pages

References in corpus (10)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Deep Residual Learning for Image Recognition
• Recurrent Neural Network Regularization
• Going Deeper with Convolutions
• Deeply-Supervised Nets
• Highway Networks
• A Clockwork RNN
• Learning Longer Memory in Recurrent Neural Networks
• Regularization and nonlinearities for neural language models: when are they needed?

Cited by in corpus (14)

• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Long-term Forecasting using Higher Order Tensor RNNs
• Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies
• Self-Attentive Residual Decoder for Neural Machine Translation
• RRA: Recurrent Residual Attention for Sequence Learning
• State-Regularized Recurrent Neural Networks
• Advanced LSTM: A Study about Better Time Dependency Modeling in Emotion Recognition
• Does Higher Order LSTM Have Better Accuracy for Segmenting and Labeling Sequence Data?
• Higher Order Recurrent Space-Time Transformer for Video Action Prediction
• Non-local Recurrent Neural Memory for Supervised Sequence Modeling
• Assessment of End-to-End and Sequential Data-driven Learning of Fluid Flows
• On the Memory Mechanism of Tensor-Power Recurrent Models
• High Order Recurrent Neural Networks for Acoustic Modelling
• On Computability, Learnability and Extractability of Finite State Machines from Recurrent Neural Networks