Interpretation of Neural Networks is Fragile

Abstract

In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions. For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification. How to interpret black-box predictors is thus an important and active area of research. A fundamental question is: how much can we trust the interpretation itself? In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations. We systematically characterize the fragility of several widely-used feature-importance interpretation methods (saliency maps, relevance propagation, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile. Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

arXiv commentPublished as a conference paper at AAAI 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Do Deep Nets Really Need to be Deep?
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Understanding Black-box Predictions via Influence Functions
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Parseval Networks: Improving Robustness to Adversarial Examples
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

Cited by in corpus (29)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Pathologies of Neural Models Make Interpretations Difficult
• Explanations can be manipulated and geometry is to blame
• Benchmarking Attribution Methods with Relative Feature Importance
• Towards Automatic Concept-based Explanations
• Deep Neural Generative Model of Functional MRI Images for Psychiatric Disorder Diagnosis
• Interpreting deep learning models for weak lensing
• Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
• Regularizing Black-box Models for Improved Interpretability
• A Distributional Framework for Data Valuation
• Investigating the significance of adversarial attacks and their relation to interpretability for radar-based human activity recognition systems
• Adversarial Reprogramming of Neural Networks
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• ML-LOO: Detecting Adversarial Examples with Feature Attribution
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• XRAI: Better Attributions Through Regions
• Interpreting Neural Networks Using Flip Points
• Interpreting Adversarial Examples with Attributes
• Towards Explanation of DNN-based Prediction with Guided Feature Inversion
• Regularizing Black-box Models for Improved Interpretability (HILL 2019 Version)
• Regional Image Perturbation Reduces $L_p$ Norms of Adversarial Examples While Maintaining Model-to-model Transferability
• Unifying Model Explainability and Robustness via Machine-Checkable Concepts
• A Tour of Convolutional Networks Guided by Linear Interpreters
• DANCE: Enhancing saliency maps using decoys
• Seven Myths in Machine Learning Research
• Saliency strikes back: How filtering out high frequencies improves white-box explanations