Attention is not Explanation

Abstract

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work, we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful `explanations' for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code for all experiments is available at https://github.com/successar/AttentionExplanation.

Accepted as NAACL 2019 Long Paper

References in corpus (1)

• Understanding Neural Networks through Representation Erasure

Cited by in corpus (14)

• WT5?! Training Text-to-Text Models to Explain their Predictions
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• Adversarial Infidelity Learning for Model Interpretation
• Normalized Attention Without Probability Cage
• Understanding Multi-Head Attention in Abstractive Summarization
• A multi-component framework for the analysis and design of explainable artificial intelligence
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Spatial--spectral FFPNet: Attention-Based Pyramid Network for Segmentation and Classification of Remote Sensing Images
• Interpretable Question Answering on Knowledge Bases and Text
• Attention improves concentration when learning node embeddings
• Attention or memory? Neurointerpretable agents in space and time
• Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability
• Games for Fairness and Interpretability