Adaptively Sparse Transformers

Abstract

Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with $\alpha$-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the $\alpha$ parameter -- which controls the shape and sparsity of $\alpha$-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.

arXiv commentConference on Empirical Methods in Natural Language Processing (EMNLP), 2019, Hong Kong, China

References in corpus (7)

• Convolutional Sequence to Sequence Learning
• Generating Long Sequences with Sparse Transformers
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Pay Less Attention with Lightweight and Dynamic Convolutions
• On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization
• Latent Alignment and Variational Attention
• A Regularized Framework for Sparse and Structured Neural Attention

Cited by in corpus (9)

• Reducing Transformer Depth on Demand with Structured Dropout
• BASE Layers: Simplifying Training of Large, Sparse Models
• ETC: Encoding Long and Structured Inputs in Transformers
• Blockwise Self-Attention for Long Document Understanding
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Understanding Multi-Head Attention in Abstractive Summarization
• Order in the Court: Explainable AI Methods Prone to Disagreement
• Weak-Attention Suppression For Transformer Based Speech Recognition
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph