Learning to Transduce with Unbounded Memory

Abstract

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.

14 pages, 4 figures, NIPS 2015

References in corpus (3)

• Sequence to Sequence Learning with Neural Networks
• Sequence Transduction with Recurrent Neural Networks
• Neural Turing Machines

Cited by in corpus (42)

• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• A Convolutional Attention Network for Extreme Summarization of Source Code
• Associative Long Short-Term Memory
• Programming with a Differentiable Forth Interpreter
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• On the Turing Completeness of Modern Neural Network Architectures
• Compositional generalization through meta sequence-to-sequence learning
• Learning Continuous Semantic Representations of Symbolic Expressions
• A Taxonomy for Neural Memory Networks
• Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
• Text normalization using memory augmented neural networks
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• Learning Efficient Algorithms with Hierarchical Attentive Memory
• A modular architecture for transparent computation in Recurrent Neural Networks
• Neural Attribute Machines for Program Generation
• CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network
• Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) Time
• Strongly-Typed Recurrent Neural Networks
• Learning to Generate with Memory
• On the Principles of Differentiable Quantum Programming Languages
• State-Regularized Recurrent Neural Networks
• Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks
• Mutual Information Scaling and Expressive Power of Sequence Models
• Compositional Generalization via Neural-Symbolic Stack Machines
• Lexicon Learning for Few-Shot Neural Sequence Modeling
• Compositional Generalization with Tree Stack Memory Units
• Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
• Neural Status Registers
• Programmable Agents
• Encoding-based Memory Modules for Recurrent Neural Networks
• Neural Machine Translation: A Review and Survey
• Motion Planning for Heterogeneous Unmanned Systems under Partial Observation from UAV
• Learning Hierarchical Structures with Differentiable Nondeterministic Stacks
• Memory and attention in deep learning
• Recursive Sketches for Modular Deep Learning
• Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication
• Explainable Neural Computation via Stack Neural Module Networks
• Understanding Memory Modules on Learning Simple Algorithms
• Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop
• Neural Program Meta-Induction
• Partially Non-Recurrent Controllers for Memory-Augmented Neural Networks
• Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language