Attention is not Explanation

Abstract

Attention mechanisms have seen wide adoption in neural NLP models. In addition to improving predictive performance, these are often touted as affording transparency: models equipped with attention provide a distribution over attended-to input units, and this is often presented (at least implicitly) as communicating the relative importance of inputs. However, it is unclear what relationship exists between attention weights and model outputs. In this work, we perform extensive experiments across a variety of NLP tasks that aim to assess the degree to which attention weights provide meaningful `explanations' for predictions. We find that they largely do not. For example, learned attention weights are frequently uncorrelated with gradient-based measures of feature importance, and one can identify very different attention distributions that nonetheless yield equivalent predictions. Our findings show that standard attention modules do not provide meaningful explanations and should not be treated as though they do. Code for all experiments is available at https://github.com/successar/AttentionExplanation.

Accepted as NAACL 2019 Long Paper

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Axiomatic Attribution for Deep Networks
• Teaching Machines to Read and Comprehend
• Dynamic Coattention Networks For Question Answering
• Understanding Neural Networks through Representation Erasure
• Pathologies of Neural Models Make Interpretations Difficult

Cited by in corpus (90)

• Pre-trained Models for Natural Language Processing: A Survey
• A Survey on the Explainability of Supervised Machine Learning
• A General Survey on Attention Mechanisms in Deep Learning
• Attention in Natural Language Processing
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
• Video Transformers: A Survey
• A Survey on Aspect-Based Sentiment Classification
• A Practical Survey on Faster and Lighter Transformers
• On the Explainability of Natural Language Processing Deep Models
• Biomedical Question Answering: A Survey of Approaches and Challenges
• WT5?! Training Text-to-Text Models to Explain their Predictions
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Language Models are Open Knowledge Graphs
• On Identifiability in Transformers
• The State of Human-centered NLP Technology for Fact-checking
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
• On Interpretability of Artificial Neural Networks: A Survey
• Explain and Predict, and then Predict Again
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Context-Aware Attention for Understanding Twitter Abuse
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Interpreting Black Box Models via Hypothesis Testing
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training
• A Neural Network Transformer Model for Composite Microstructure Homogenization
• Self-Explaining Structures Improve NLP Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Machine learning surrogates for efficient hydrologic modeling: Insights from stochastic simulations of managed aquifer recharge
• Adversarial Infidelity Learning for Model Interpretation
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
• Convolution-enhanced Evolving Attention Networks
• Towards Understanding Neural Machine Translation with Word Importance
• Normalized Attention Without Probability Cage
• Evolving Attention with Residual Convolutions
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Understanding Multi-Head Attention in Abstractive Summarization
• A multi-component framework for the analysis and design of explainable artificial intelligence
• On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations
• ProtoryNet - Interpretable Text Classification Via Prototype Trajectories
• BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
• X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust
• Understanding How Encoder-Decoder Architectures Attend
• Interpretation of NLP models through input marginalization
• A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
• It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• An attention model to analyse the risk of agitation and urinary tract infections in people with dementia
• Challenges for cognitive decoding using deep learning methods
• Weakly Supervised Reasoning by Neuro-Symbolic Approaches
• Explainability in AI Based Applications: A Framework for Comparing Different Techniques
• This Reads Like That: Deep Learning for Interpretable Natural Language Processing
• Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Explicit-Blurred Memory Network for Analyzing Patient Electronic Health Records
• Predicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream Transformer
• Attention Flows are Shapley Value Explanations
• Attention improves concentration when learning node embeddings
• Does Attention Mechanism Possess the Feature of Human Reading? A Perspective of Sentiment Classification Task
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• VisBERT: Hidden-State Visualizations for Transformers
• Collaborative Graph Learning with Auxiliary Text for Temporal Event Prediction in Healthcare
• Neural Machine Translation: A Review and Survey
• Spatial--spectral FFPNet: Attention-Based Pyramid Network for Segmentation and Classification of Remote Sensing Images
• Language Model Evaluation in Open-ended Text Generation
• Interpretable Question Answering on Knowledge Bases and Text
• Why Attentions May Not Be Interpretable?
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• Logic Traps in Evaluating Attribution Scores
• A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
• SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning
• Multi-Domain Transformer-Based Counterfactual Augmentation for Earnings Call Analysis
• Transformer-F: A Transformer network with effective methods for learning universal sentence representation
• On the Learning Dynamics of Attention Networks
• Towards explainable message passing networks for predicting carbon dioxide adsorption in metal-organic frameworks
• Social Media Unrest Prediction during the {COVID}-19 Pandemic: Neural Implicit Motive Pattern Recognition as Psychometric Signs of Severe Crises
• Enriched Annotations for Tumor Attribute Classification from Pathology Reports with Limited Labeled Data
• Explainability-aided Domain Generalization for Image Classification
• Attention or memory? Neurointerpretable agents in space and time
• Deep Dynamic Effective Connectivity Estimation from Multivariate Time Series
• Games for Fairness and Interpretability
• Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability
• Corpus-level and Concept-based Explanations for Interpretable Document Classification
• The Brownian motion in the transformer model
• A Framework for Rationale Extraction for Deep QA models