Visualizing Attention in Transformer-Based Language Representation Models

Abstract

We present an open-source tool for visualizing multi-head self-attention in Transformer-based language representation models. The tool extends earlier work by visualizing attention at three levels of granularity: the attention-head level, the model level, and the neuron level. We describe how each of these views can help to interpret the model, and we demonstrate the tool on the BERT model and the OpenAI GPT-2 model. We also present three use cases for analyzing GPT-2: detecting model bias, identifying recurring patterns, and linking neurons to model behavior.

References in corpus (1)

• Reasoning about Entailment with Neural Attention

Cited by in corpus (12)

• Visualizing and Measuring the Geometry of BERT
• On the Explainability of Natural Language Processing Deep Models
• Language Models are Open Knowledge Graphs
• On Identifiability in Transformers
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• Tree Transformer: Integrating Tree Structures into Self-Attention
• Attention Approximates Sparse Distributed Memory
• Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures
• AttViz: Online exploration of self-attention for transparent neural language modeling
• On the Importance of Local Information in Transformer Based Models
• Corpus-level and Concept-based Explanations for Interpretable Document Classification
• IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?