Attention Interpretability Across NLP Tasks

Abstract

The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

References in corpus (3)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Teaching Machines to Read and Comprehend
• Conditional Image Generation with PixelCNN Decoders

Cited by in corpus (19)

• Attention in Natural Language Processing
• On the Explainability of Natural Language Processing Deep Models
• Deep learning of contagion dynamics on complex networks
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• Self-Explaining Structures Improve NLP Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Hard-Coded Gaussian Attention for Neural Machine Translation
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• Understanding Multi-Head Attention in Abstractive Summarization
• A Simple Framework for Uncertainty in Contrastive Learning
• Approximating How Single Head Attention Learns
• On the Learning Dynamics of Attention Networks
• Logic Traps in Evaluating Attribution Scores
• Rationalizing Medical Relation Prediction from Corpus-level Statistics
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• Structured Self-Attention Weights Encode Semantics in Sentiment Analysis
• Text analysis and deep learning: A network approach
• Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience