Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Abstract

The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input. Prior work strives to attribute model decisions to individual input features with different saliency measures, but they fail to explain how these input features interact with each other to reach predictions. In this paper, we propose a self-attention attribution method to interpret the information interactions inside Transformer. We take BERT as an example to conduct extensive studies. Firstly, we apply self-attention attribution to identify the important attention heads, while others can be pruned with marginal performance degradation. Furthermore, we extract the most salient dependencies in each layer to construct an attribution tree, which reveals the hierarchical interactions inside Transformer. Finally, we show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.

arXiv commentAAAI-2021

References in corpus (8)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Automatic Rule Extraction from Long Short Term Memory Networks
• Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers
• Inducing Syntactic Trees from BERT Representations

Cited by in corpus (4)

• BEiT: BERT Pre-Training of Image Transformers
• On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
• Understanding Self-Attention of Self-Supervised Audio Transformers
• Logic Traps in Evaluating Attribution Scores