Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes

Abstract

We extend neural Turing machine (NTM) model into a dynamic neural Turing machine (D-NTM) by introducing a trainable memory addressing scheme. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies including both linear and nonlinear ones. We implement the D-NTM with both continuous, differentiable and discrete, non-differentiable read/write mechanisms. We investigate the mechanisms and effects of learning to read and write into a memory through experiments on Facebook bAbI tasks using both a feedforward and GRUcontroller. The D-NTM is evaluated on a set of Facebook bAbI tasks and shown to outperform NTM and LSTM baselines. We have done extensive analysis of our model and different variations of NTM on bAbI task. We also provide further experimental results on sequential pMNIST, Stanford Natural Language Inference, associative recall and copy tasks.

13 pages, 3 figures

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Neural Conversational Model
• Dynamic Memory Networks for Visual and Textual Question Answering
• Reasoning about Entailment with Neural Attention
• Large-scale Simple Question Answering with Memory Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• A large annotated corpus for learning natural language inference
• Describing Videos by Exploiting Temporal Structure
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Recurrent Dropout without Memory Loss
• Noisy Activation Functions
• Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
• Hierarchical Memory Networks
• Learning Simple Algorithms from Examples
• Lie Access Neural Turing Machine
• Structured Memory for Neural Turing Machines

Cited by in corpus (16)

• End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
• Search Engine Guided Non-Parametric Neural Machine Translation
• Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
• A Read-Write Memory Network for Movie Story Understanding
• Lie Access Neural Turing Machine
• Implementing Neural Turing Machines
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Data
• Learning Numeracy: Binary Arithmetic with Neural Turing Machines
• Learning to Remember, Forget and Ignore using Attention Control in Memory
• Modeling Past and Future for Neural Machine Translation
• A Memory Network Approach for Story-based Temporal Summarization of 360° Videos
• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning
• Memory-Augmented Recurrent Networks for Dialogue Coherence
• Understanding Memory Modules on Learning Simple Algorithms
• Partially Non-Recurrent Controllers for Memory-Augmented Neural Networks