On Identifiability in Transformers

Abstract

In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.

Published as a conference paper at ICLR 2020

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Opening the Black Box of Deep Neural Networks via Information
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Attention is not Explanation
• Universal Transformers
• A BERT Baseline for the Natural Questions
• Adding Interpretable Attention to Neural Translation Models Improves Word Alignment
• Visualizing Attention in Transformer-Based Language Representation Models
• Investigating the Successes and Failures of BERT for Passage Re-Ranking
• Attending to Mathematical Language with Transformers
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?

Cited by in corpus (10)

• Post-hoc Interpretability for Neural NLP: A Survey
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• Interpreting Black Box Models via Hypothesis Testing
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching
• BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
• It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers
• Attention for Causal Relationship Discovery from Biological Neural Dynamics
• Accurate Word Alignment Induction from Neural Machine Translation
• Circles are like Ellipses, or Ellipses are like Circles? Measuring the Degree of Asymmetry of Static and Contextual Embeddings and the Implications to Representation Learning