Reservoir Stack Machines

Abstract

Memory-augmented neural networks equip a recurrent neural network with an explicit memory to support tasks that require information storage without interference over long times. A key motivation for such research is to perform classic computation tasks, such as parsing. However, memory-augmented neural networks are notoriously hard to train, requiring many backpropagation epochs and a lot of data. In this paper, we introduce the reservoir stack machine, a model which can provably recognize all deterministic context-free languages and circumvents the training problem by training only the output layer of a recurrent net and employing auxiliary information during training about the desired interaction with a stack. In our experiments, we validate the reservoir stack machine against deep and shallow networks from the literature on three benchmark tasks for Neural Turing machines and six deterministic context-free languages. Our results show that the reservoir stack machine achieves zero error, even on test sequences longer than the training data, requiring only a few seconds of training time and 100 training sequences.

arXiv commentin print at the Journal Neurocomputing

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
• Learning to Compose Words into Sentences with Reinforcement Learning
• Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
• Implementing Neural Turing Machines
• Reservoir Memory Machines as Neural Computers