Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Abstract

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.

References in corpus (4)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Learning to Execute
• High-Performance Neural Networks for Visual Object Classification
• Learning Longer Memory in Recurrent Neural Networks

Cited by in corpus (52)

• End-To-End Memory Networks
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Universal Transformers
• Control of Memory, Active Perception, and Action in Minecraft
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
• RobustFill: Neural Program Learning under Noisy I/O
• Associative Long Short-Term Memory
• Programming with a Differentiable Forth Interpreter
• Analysing Mathematical Reasoning Abilities of Neural Models
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Neural Programmer: Inducing Latent Programs with Gradient Descent
• On the Turing Completeness of Modern Neural Network Architectures
• On the Binding Problem in Artificial Neural Networks
• MazeBase: A Sandbox for Learning from Games
• Learning Continuous Semantic Representations of Symbolic Expressions
• A Taxonomy for Neural Memory Networks
• Addressing Some Limitations of Transformers with Feedback Memory
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Text normalization using memory augmented neural networks
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• A modular architecture for transparent computation in Recurrent Neural Networks
• Strong Generalization and Efficiency in Neural Programs
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) Time
• Can RNNs learn Recursive Nested Subject-Verb Agreements?
• VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop
• Towards Better Opioid Antagonists Using Deep Reinforcement Learning
• Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks
• Mutual Information Scaling and Expressive Power of Sequence Models
• Compositional Generalization via Neural-Symbolic Stack Machines
• Compositional Generalization with Tree Stack Memory Units
• Cooperative Learning of Disjoint Syntax and Semantics
• Encoding-based Memory Modules for Recurrent Neural Networks
• I-BERT: Inductive Generalization of Transformer to Arbitrary Context Lengths
• Neural Machine Translation: A Review and Survey
• Motion Planning for Heterogeneous Unmanned Systems under Partial Observation from UAV
• Neural Program Synthesis By Self-Learning
• PaccMann${}^{RL}$ on SARS-CoV-2: Designing antiviral candidates with conditional generative models
• Learning Hierarchical Structures with Differentiable Nondeterministic Stacks
• Recursive Sketches for Modular Deep Learning
• Hierarchical Variational Imitation Learning of Control Programs
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer Architecture
• Memory-Augmented Temporal Dynamic Learning for Action Recognition
• pix2rule: End-to-end Neuro-symbolic Rule Learning
• Finding Syntactic Representations in Neural Stacks
• Common-Description Learning: A Framework for Learning Algorithms and Generating Subproblems from Few Examples
• Understanding Memory Modules on Learning Simple Algorithms
• What do Entity-Centric Models Learn? Insights from Entity Linking in Multi-Party Dialogue
• On learning an interpreted language with recurrent models