Compositional Attention: Disentangling Search and Retrieval

Abstract

Multi-head, key-value attention is the backbone of the widely successful Transformer model and its variants. This attention mechanism uses multiple parallel key-value attention blocks (called heads), each performing two fundamental computations: (1) search - selection of a relevant entity from a set via query-key interactions, and (2) retrieval - extraction of relevant features from the selected entity via a value matrix. Importantly, standard attention heads learn a rigid mapping between search and retrieval. In this work, we first highlight how this static nature of the pairing can potentially: (a) lead to learning of redundant parameters in certain tasks, and (b) hinder generalization. To alleviate this problem, we propose a novel attention mechanism, called Compositional Attention, that replaces the standard head structure. The proposed mechanism disentangles search and retrieval and composes them in a dynamic, flexible and context-dependent manner through an additional soft competition stage between the query-key combination and value pairing. Through a series of numerical experiments, we show that it outperforms standard multi-head attention on a variety of tasks, including some out-of-distribution settings. Through our qualitative analysis, we demonstrate that Compositional Attention leads to dynamic specialization based on the type of retrieval needed. Our proposed mechanism generalizes multi-head attention, allows independent scaling of search and retrieval, and can easily be implemented in lieu of standard attention heads in any network architecture.

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Linformer: Self-Attention with Linear Complexity
• A simple neural network module for relational reasoning
• Generating Long Sequences with Sparse Transformers
• Pointer Sentinel Mixture Models
• A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
• Rethinking Attention with Performers
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Coordination Among Neural Modules Through a Shared Global Workspace
• Emergent Symbols through Binding in External Memory
• Compositional Generalization for Primitive Substitutions
• Transformers with Competitive Ensembles of Independent Mechanisms
• Compositional Generalization via Neural-Symbolic Stack Machines