Memory Augmented Neural Networks with Wormhole Connections

Abstract

Recent empirical results on long-term dependency tasks have shown that neural networks augmented with an external memory can learn the long-term dependency tasks more easily and achieve better generalization than vanilla recurrent neural networks (RNN). We suggest that memory augmented neural networks can reduce the effects of vanishing gradients by creating shortcut (or wormhole) connections. Based on this observation, we propose a novel memory augmented neural network model called TARDIS (Temporal Automatic Relation Discovery in Sequences). The controller of TARDIS can store a selective set of embeddings of its own previous hidden states into an external memory and revisit them as and when needed. For TARDIS, memory acts as a storage for wormhole connections to the past to propagate the gradients more effectively and it helps to learn the temporal dependencies. The memory structure of TARDIS has similarities to both Neural Turing Machines (NTM) and Dynamic Neural Turing Machines (D-NTM), but both read and write operations of TARDIS are simpler and more efficient. We use discrete addressing for read/write operations which helps to substantially to reduce the vanishing gradient problem with very long sequences. Read and write operations in TARDIS are tied with a heuristic once the memory becomes full, and this makes the learning problem simpler when compared to NTM or D-NTM type of architectures. We provide a detailed analysis on the gradient propagation in general for MANNs. We evaluate our models on different long-term dependency tasks and report competitive results in all of them.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• One-shot Learning with Memory-Augmented Neural Networks
• Neural GPUs Learn Algorithms
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• On Singular Value Inequalities for the Sum of Two Matrices

Cited by in corpus (22)

• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• Improving the Gating Mechanism of Recurrent Neural Networks
• SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
• Continual Learning in Recurrent Neural Networks
• DrNAS: Dirichlet Neural Architecture Search
• Integrating Episodic Memory into a Reinforcement Learning Agent using Reservoir Sampling
• Hierarchical Multi-scale Attention Networks for Action Recognition
• Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory
• Explicit-Blurred Memory Network for Analyzing Patient Electronic Health Records
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• Learning to Remember, Forget and Ignore using Attention Control in Memory
• Learning Simpler Language Models with the Differential State Framework
• Knowledge Efficient Deep Learning for Natural Language Processing
• On-The-Fly Information Retrieval Augmentation for Language Models
• Memory and attention in deep learning
• Harmonic-Percussive Source Separation with Deep Neural Networks and Phase Recovery
• Multi Resolution LSTM For Long Term Prediction In Neural Activity Video
• Understanding Memory Modules on Learning Simple Algorithms
• ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network
• Memory-Augmented Recurrent Networks for Dialogue Coherence
• Using Multi-task and Transfer Learning to Solve Working Memory Tasks
• Partially Non-Recurrent Controllers for Memory-Augmented Neural Networks