Explanations can be manipulated and geometry is to blame

Abstract

Explanation methods aim to make neural networks more trustworthy and interpretable. In this paper, we demonstrate a property of explanation methods which is disconcerting for both of these purposes. Namely, we show that explanations can be manipulated arbitrarily by applying visually hardly perceptible perturbations to the input that keep the network's output approximately constant. We establish theoretically that this phenomenon can be related to certain geometrical properties of neural networks. This allows us to derive an upper bound on the susceptibility of explanations to manipulations. Based on this result, we propose effective mechanisms to enhance the robustness of explanations.

References in corpus (9)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Learning how to explain neural networks: PatternNet and PatternAttribution
• The (Un)reliability of saliency methods
• Interpretation of Neural Networks is Fragile

Cited by in corpus (31)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
• Toward Explainable AI for Regression Models
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects
• "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• Model extraction from counterfactual explanations
• Saliency Methods for Explaining Adversarial Attacks
• Adversarial Infidelity Learning for Model Interpretation
• Best of both worlds: local and global explanations with human-understandable concepts
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
• A simple defense against adversarial attacks on heatmap explanations
• Robustness of Visual Explanations to Common Data Augmentation
• Consistent Counterfactuals for Deep Models
• A Game-Theoretic Taxonomy of Visual Concepts in DNNs
• Learning Fair Rule Lists
• A Survey on the Robustness of Feature Importance and Counterfactual Explanations
• Aggregating explanation methods for stable and robust explainability
• Robust and Stable Black Box Explanations
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• Selective Ensembles for Consistent Predictions
• Black-box Attacks on Image Activity Prediction and its Natural Language Explanations
• Do Input Gradients Highlight Discriminative Features?
• AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators
• Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning