Order in the Court: Explainable AI Methods Prone to Disagreement
arXiv:2105.03287
Abstract
By computing the rank correlation between attention weights and feature-additive explanation methods, previous analyses either invalidate or support the role of attention-based explanations as a faithful and plausible measure of salience. To investigate whether this approach is appropriate, we compare LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations, applied to two neural architectures trained on single- and pair-sequence language tasks. In most cases, we find that none of our chosen methods agree. Based on our empirical observations and theoretical objections, we conclude that rank correlation does not measure the quality of feature-additive methods. Practitioners should instead use the numerous and rigorous diagnostic methods proposed by the community.
arXiv commentAccepted for presentation at the ICML Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI
References in corpus (17)
- Decoupled Weight Decay Regularization
- Towards A Rigorous Science of Interpretable Machine Learning
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Axiomatic Attribution for Deep Networks
- Learning Important Features Through Propagating Activation Differences
- Understanding Neural Networks through Representation Erasure
- A large annotated corpus for learning natural language inference
- A Benchmark for Interpretability Methods in Deep Neural Networks
- The (Un)reliability of saliency methods
- On the Convergence Proof of AMSGrad and a New Version
- Investigating the influence of noise and distractors on the interpretation of neural networks
- Interpretable Credit Application Predictions With Counterfactual Explanations
- Interpretation of Prediction Models Using the Input Gradient
- Adaptively Sparse Transformers
- Evaluating the Correctness of Explainable AI Algorithms for Classification
- Understanding Multi-Head Attention in Abstractive Summarization
- Attention Flows are Shapley Value Explanations