A Taxonomy for Neural Memory Networks

Abstract

In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our arguments. And we use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem.

References in corpus (13)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Visualizing and Understanding Recurrent Networks
• End-To-End Memory Networks
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Learning to Transduce with Unbounded Memory
• Capacity and Trainability in Recurrent Neural Networks
• MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
• The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations

Cited by in corpus (6)

• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• Graph Element Networks: adaptive, structured computation and memory
• Memory and attention in deep learning
• External-Memory Networks for Low-Shot Learning of Targets in Forward-Looking-Sonar Imagery
• Learning to Summarize Long Texts with Memory Compression and Transfer