Fooling Neural Network Interpretations via Adversarial Model Manipulation

Abstract

We ask whether the neural network interpretation methods can be fooled via adversarial model manipulation, which is defined as a model fine-tuning step that aims to radically alter the explanations without hurting the accuracy of the original models, e.g., VGG19, ResNet50, and DenseNet121. By incorporating the interpretation results directly in the penalty term of the objective function for fine-tuning, we show that the state-of-the-art saliency map based interpreters, e.g., LRP, Grad-CAM, and SimpleGrad, can be easily fooled with our model manipulation. We propose two types of fooling, Passive and Active, and demonstrate such foolings generalize well to the entire validation set as well as transfer to other interpretation methods. Our results are validated by both visually showing the fooled explanations and reporting quantitative metrics that measure the deviations from the original explanations. We claim that the stability of neural network interpretation method with respect to our adversarial model manipulation is an important criterion to check for developing robust and reliable neural network interpretation method.

References in corpus (13)

• Explaining and Harnessing Adversarial Examples
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Interpretable machine learning: definitions, methods, and applications
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• SmoothGrad: removing noise by adding noise
• Towards Robust Interpretability with Self-Explaining Neural Networks
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Investigating the influence of noise and distractors on the interpretation of neural networks
• Interpretable Deep Learning under Fire

Cited by in corpus (22)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Interpreting Deep Learning-Based Networking Systems
• Benchmarking Attribution Methods with Relative Feature Importance
• A Survey on Neural Network Interpretability
• What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
• Smoothed Geometry for Robust Attribution
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• Adversarial Infidelity Learning for Model Interpretation
• Explaining Bayesian Neural Networks
• Fooling Partial Dependence via Data Poisoning
• Learning Fair Rule Lists
• Adversarial Attacks and Defenses: An Interpretation Perspective
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• A Survey on the Robustness of Feature Importance and Counterfactual Explanations
• Jointly Attacking Graph Neural Network and its Explanations
• Aggregating explanation methods for stable and robust explainability
• Gradient-based Analysis of NLP Models is Manipulable
• Selective Ensembles for Consistent Predictions
• Black-box Attacks on Image Activity Prediction and its Natural Language Explanations
• Do Input Gradients Highlight Discriminative Features?
• Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning
• AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators