Self-Attentive Associative Memory

Abstract

Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.

arXiv comment19 pages

References in corpus (6)

• A simple neural network module for relational reasoning
• Using Fast Weights to Attend to the Recent Past
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Metalearned Neural Memory
• Learning to Remember More with Less Memorization
• Dual Memory Neural Computer for Asynchronous Two-view Sequential Learning

Cited by in corpus (4)

• Graph Neural Networks Extract High-Resolution Cultivated Land Maps from Sentinel-2 Image Series
• Distributed Associative Memory Network with Memory Refreshing Loss
• A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19
• Learning Associative Inference Using Fast Weight Memory