Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

Abstract

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• Axiomatic Attribution for Deep Networks
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Attention is not Explanation
• Semi-supervised Sequence Learning
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Normal vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction
• GAM: Explainable Visual Similarity and Classification via Gradient Activation Maps

Cited by in corpus (5)

• Evaluating the Effectiveness of XAI Techniques for Encoder-Based Language Models
• GAM: Explainable Visual Similarity and Classification via Gradient Activation Maps
• Reasoning-Grounded Natural Language Explanations for Language Models
• ASAP: Visual Analytics for Identifying and Analyzing Image Patterns in AI-generated Images
• Emoception: Selective Affective Layer Fine-Tuning of Video Vision Transformers for Player Arousal Change Recognition From Gameplay Footage