Evaluations and Methods for Explanation through Robustness Analysis

Abstract

Feature based explanations, that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model. In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis. In contrast to existing evaluations which require us to specify some way to "remove" features that could inevitably introduces biases and artifacts, we make use of the subtler notion of smaller adversarial perturbations. By optimizing towards our proposed evaluation criteria, we obtain new explanations that are loosely necessary and sufficient for a prediction. We further extend the explanation to extract the set of features that would move the current prediction to a target class by adopting targeted adversarial attack for the robustness analysis. Through experiments across multiple domains and a user study, we validate the usefulness of our evaluation criteria and our derived explanations.

arXiv commentTo appear in ICLR 2021

References in corpus (13)

• Towards A Rigorous Science of Interpretable Machine Learning
• Towards Deep Learning Models Resistant to Adversarial Attacks
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Understanding Neural Networks through Representation Erasure
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Efficient Formal Safety Analysis of Neural Networks
• Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
• Model Agnostic Supervised Local Explanations
• Structured Adversarial Attack: Towards General Implementation and Better Interpretability
• Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach
• Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems
• Representer Point Selection for Explaining Deep Neural Networks

Cited by in corpus (4)

• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• Combining Similarity and Adversarial Learning to Generate Visual Explanation: Application to Medical Image Classification
• Where and When: Space-Time Attention for Audio-Visual Explanations
• Understanding Misclassifications by Attributes