Encoding-based Memory Modules for Recurrent Neural Networks

Abstract

Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. We extend the memorization component with a modular memory that encodes the hidden state sequence at different sampling frequencies. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. The experimental results on synthetic and real-world datasets show that specializing the training algorithm to train the memorization component always improves the final performance whenever the memorization of long sequences is necessary to solve the problem.

preprint submitted at Elsevier Neural Networks

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Recent Trends in Deep Learning Based Natural Language Processing
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Hierarchical Multiscale Recurrent Neural Networks
• Unitary Evolution Recurrent Neural Networks
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Learning to Transduce with Unbounded Memory
• Full-Capacity Unitary Recurrent Neural Networks

Cited by in corpus (2)

• Hopfield Networks is All You Need
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization