The effectiveness of feature attribution methods and its correlation with automatic evaluation scores

Abstract

Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers have either proposed new feature attribution methods, discussed or harnessed these tools in their work. However, despite humans being the target end-users, most attribution methods were only evaluated on proxy automatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et al. 2018). In this paper, we conduct the first user study to measure attribution map effectiveness in assisting humans in ImageNet classification and Stanford Dogs fine-grained classification, and when an image is natural or adversarial (i.e., contains adversarial perturbations). Overall, feature attribution is surprisingly not more effective than showing humans nearest training-set examples. On a harder task of fine-grained dog categorization, presenting attribution maps to humans does not help, but instead hurts the performance of human-AI teams compared to AI alone. Importantly, we found automatic attribution-map evaluation measures to correlate poorly with the actual human-AI team performance. Our findings encourage the community to rigorously test their methods on the downstream human-in-the-loop applications and to rethink the existing evaluation metrics.

arXiv commentNeurIPS 2021; 10 pages of Main text; 28 pages of Appendix

References in corpus (16)

• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• European Union regulations on algorithmic decision-making and a "right to explanation"
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• Near-Optimally Teaching the Crowd to Classify
• Quantifying Interpretability and Trust in Machine Learning Systems
• Generating Counterfactual Explanations with Natural Language
• Towards falsifiable interpretability research
• Feature Removal Is a Unifying Principle for Model Explanation Methods
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• Challenging common interpretability assumptions in feature attribution explanations
• Explainable AI for Natural Adversarial Images

Cited by in corpus (5)

• Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
• The Impact of Imperfect XAI on Human-AI Decision-Making
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• Saliency strikes back: How filtering out high frequencies improves white-box explanations