Learning Longer Memory in Recurrent Neural Networks

Abstract

Recurrent neural network is a powerful model that learns temporal patterns in sequential data. For a long time, it was believed that recurrent networks are difficult to train using simple optimizers, such as stochastic gradient descent, due to the so-called vanishing gradient problem. In this paper, we show that learning longer term patterns in real data, such as in natural language, is perfectly possible using gradient descent. This is achieved by using a slight structural modification of the simple recurrent neural network architecture. We encourage some of the hidden units to change their state slowly by making part of the recurrent weight matrix close to identity, thus forming kind of a longer term memory. We evaluate our model in language modeling experiments, where we obtain similar performance to the much more complex Long Short Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997).

References in corpus (3)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Recurrent Neural Network Regularization
• Regularization and nonlinearities for neural language models: when are they needed?

Cited by in corpus (50)

• End-To-End Memory Networks
• Resnet in Resnet: Generalizing Residual Architectures
• Recent Advances in Recurrent Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Long Short-Term Memory-Networks for Machine Reading
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Efficient softmax approximation for GPUs
• DeepBach: a Steerable Model for Bach Chorales Generation
• Review: Deep Learning in Electron Microscopy
• Recurrent Dropout without Memory Loss
• Compositional Vector Space Models for Knowledge Base Completion
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• DR-RNN: A deep residual recurrent neural network for model reduction
• What is the best RNN-cell structure to forecast each time series behavior?
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Larger-Context Language Modelling
• Higher Order Recurrent Neural Networks
• Gated Recurrent Neural Tensor Network
• A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models
• Learning with hidden variables
• A recurrent neural network without chaos
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Feedforward Sequential Memory Neural Networks without Recurrent Feedback
• Fourier Neural Networks: A Comparative Study
• Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
• Unsupervised Learning from Continuous Video in a Scalable Predictive Recurrent Network
• A Linear Dynamical System Model for Text
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Constrained Deep Learning using Conditional Gradient and Applications in Computer Vision
• Top-down Tree Long Short-Term Memory Networks
• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• Tree Memory Networks for Modelling Long-term Temporal Dependencies
• A Simple LSTM model for Transition-based Dependency Parsing
• An Attention Mechanism for Answer Selection Using a Combined Global and Local View
• Simplified Long Short-term Memory Recurrent Neural Networks: part III
• Learning long-term dependencies for action recognition with a biologically-inspired deep network
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Self-organized Hierarchical Softmax
• Stochastic Learning of Nonstationary Kernels for Natural Language Modeling
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Prototypical Recurrent Unit
• Learning Simpler Language Models with the Differential State Framework
• Using General Adversarial Networks for Marketing: A Case Study of Airbnb
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Learning Multi-Sense Word Distributions using Approximate Kullback-Leibler Divergence
• Universal Network Representation for Heterogeneous Information Networks