Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Abstract

Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs $1,000\times$ faster and with $3,000\times$ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring $100,000$s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.

in 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• End-To-End Memory Networks
• Mondrian Forests: Efficient Online Random Forests
• Neural Turing Machines

Cited by in corpus (21)

• Universal Transformers
• Reformer: The Efficient Transformer
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• Compressive Transformers for Long-Range Sequence Modelling
• Metalearned Neural Memory
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Learning to Remember More with Less Memorization
• Memory-Based Graph Networks
• Reservoir Memory Machines as Neural Computers
• MEMO: A Deep Network for Flexible Combination of Episodic Memories
• Distributed Associative Memory Network with Memory Refreshing Loss
• TrustMAE: A Noise-Resilient Defect Classification Framework using Memory-Augmented Auto-Encoders with Trust Regions
• Neural Stored-program Memory
• A short note on the decision tree based neural turing machine
• Knowledge Efficient Deep Learning for Natural Language Processing
• On-The-Fly Information Retrieval Augmentation for Language Models
• A Neural Turing~Machine for Conditional Transition Graph Modeling
• Memory based fusion for multi-modal deep learning
• Large Product Key Memory for Pretrained Language Models
• Reservoir Stack Machines
• Understanding Memory Modules on Learning Simple Algorithms