Order in the Court: Explainable AI Methods Prone to Disagreement

Abstract

By computing the rank correlation between attention weights and feature-additive explanation methods, previous analyses either invalidate or support the role of attention-based explanations as a faithful and plausible measure of salience. To investigate whether this approach is appropriate, we compare LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations, applied to two neural architectures trained on single- and pair-sequence language tasks. In most cases, we find that none of our chosen methods agree. Based on our empirical observations and theoretical objections, we conclude that rank correlation does not measure the quality of feature-additive methods. Practitioners should instead use the numerous and rigorous diagnostic methods proposed by the community.

Accepted for presentation at the ICML Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI

References in corpus (8)

• Towards A Rigorous Science of Interpretable Machine Learning
• Understanding Neural Networks through Representation Erasure
• The (Un)reliability of saliency methods
• Investigating the influence of noise and distractors on the interpretation of neural networks
• Interpretation of Prediction Models Using the Input Gradient
• Evaluating the Correctness of Explainable AI Algorithms for Classification
• Understanding Multi-Head Attention in Abstractive Summarization
• Attention Flows are Shapley Value Explanations

Cited by in corpus (2)

• Seven challenges for harmonizing explainability requirements
• Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools