Neural Random-Access Machines

Abstract

In this paper, we propose and investigate a new neural network architecture called Neural Random Access Machine. It can manipulate and dereference pointers to an external variable-size random-access memory. The model is trained from pure input-output examples using backpropagation. We evaluate the new model on a number of simple algorithmic tasks whose solutions require pointer manipulation and dereferencing. Our results show that the proposed model can learn to solve algorithmic tasks of such type and is capable of operating on simple data structures like linked-lists and binary trees. For easier tasks, the learned solutions generalize to sequences of arbitrary length. Moreover, memory access during inference can be done in a constant time under some assumptions.

ICLR submission, 17 pages, 9 figures, 6 tables (with bibliography and appendix)

References in corpus (1)

• Recurrent Neural Networks with External Memory for Language Understanding

Cited by in corpus (11)

• Adding Gradient Noise Improves Learning for Very Deep Networks
• RobustFill: Neural Program Learning under Noisy I/O
• On the Turing Completeness of Modern Neural Network Architectures
• On the Binding Problem in Artificial Neural Networks
• Program Synthesis with Large Language Models
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) Time
• On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamics
• Towards Modular Algorithm Induction
• Progress Extrapolating Algorithmic Learning to Arbitrary Sequence Lengths
• BF++: a language for general-purpose program synthesis