Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods

Abstract

For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on explanations --- feature-additivity and feature-selection --- lead to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that current post-hoc explainers are either validated under simplistic scenarios (on simple models such as linear regression, or on models trained on syntactic datasets), or, when applied to real-world neural networks, explainers are commonly validated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.

References in corpus (10)

• Learning Important Features Through Propagating Activation Differences
• Understanding Black-box Predictions via Influence Functions
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Sanity Checks for Saliency Maps
• A large annotated corpus for learning natural language inference
• e-SNLI: Natural Language Inference with Natural Language Explanations
• Model Agnostic Supervised Local Explanations
• Nothing Else Matters: Model-Agnostic Explanations By Identifying Prediction Invariance
• Learning Attitudes and Attributes from Multi-Aspect Reviews
• Textual Explanations for Self-Driving Vehicles

Cited by in corpus (8)

• On Tackling Explanation Redundancy in Decision Trees
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• Explaining Deep Neural Networks
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• Efficient Explanations With Relevant Sets
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• On Efficiently Explaining Graph-Based Classifiers
• To what extent do human explanations of model behavior align with actual model behavior?