Are Sixteen Heads Really Better than One?

Abstract

Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions. In particular, multi-headed attention is a driving force behind many recent state-of-the-art NLP models such as Transformer-based MT models and BERT. These models apply multiple attention mechanisms in parallel, with each attention "head" potentially focusing on different parts of the input, which makes it possible to express sophisticated functions beyond the simple weighted average. In this paper we make the surprising observation that even if models have been trained using multiple heads, in practice, a large percentage of attention heads can be removed at test time without significantly impacting performance. In fact, some layers can even be reduced to a single head. We further examine greedy algorithms for pruning down models, and the potential speed, memory efficiency, and accuracy improvements obtainable therefrom. Finally, we analyze the results with respect to which parts of the model are more reliant on having multiple heads, and provide precursory evidence that training dynamics play a role in the gains provided by multi-head attention.

arXiv commentNeurIPS 2019

References in corpus (5)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• A Deep Reinforced Model for Abstractive Summarization
• Opening the Black Box of Deep Neural Networks via Information
• Pruning Filters for Efficient ConvNets
• Weighted Transformer Network for Machine Translation

Cited by in corpus (17)

• Reducing Transformer Depth on Demand with Structured Dropout
• Reweighted Proximal Pruning for Large-Scale Language Representation
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Blockwise Self-Attention for Long Document Understanding
• I-BERT: Integer-only BERT Quantization
• Low-Rank Bottleneck in Multi-head Attention Models
• Understanding Multi-Head Attention in Abstractive Summarization
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
• Hessian-Aware Pruning and Optimal Neural Implant
• Sampling Equivariant Self-attention Networks for Object Detection in Aerial Images
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures
• How BPE Affects Memorization in Transformers
• The Brownian motion in the transformer model