Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
arXiv:1911.03329
Abstract
We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack-based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory-augmented architectures are easy to train in an end-to-end fashion and can learn the Dyck languages over as many as six parenthesis-pairs, in addition to two deterministic palindrome languages and the string-reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory-augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Theoretical Limitations of Self-Attention in Neural Sequence Models
- Learning to Transduce with Unbounded Memory
- Reinforcement Learning Neural Turing Machines - Revised
- Lie Access Neural Turing Machine
- Learning Operations on a Stack with Neural Turing Machines
Cited by in corpus (11)
- Can RNNs learn Recursive Nested Subject-Verb Agreements?
- On the Linguistic Capacity of Real-Time Counter Automata
- Formal Language Theory Meets Modern NLP
- A Survey of Neural Networks and Formal Languages
- Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
- Self-Attention Networks Can Process Bounded Hierarchical Languages
- Reservoir Stack Machines
- Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
- How Can Self-Attention Networks Recognize Dyck-n Languages?
- Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language
- Recognizing Long Grammatical Sequences Using Recurrent Networks Augmented With An External Differentiable Stack