How to Manipulate CNNs to Make Them Lie: the GradCAM Case

Abstract

Recently many methods have been introduced to explain CNN decisions. However, it has been shown that some methods can be sensitive to manipulation of the input. We continue this line of work and investigate the explanation method GradCAM. Instead of manipulating the input, we consider an adversary that manipulates the model itself to attack the explanation. By changing weights and architecture, we demonstrate that it is possible to generate any desired explanation, while leaving the model's accuracy essentially unchanged. This illustrates that GradCAM cannot explain the decision of every CNN and provides a proof of concept showing that it is possible to obfuscate the inner workings of a CNN. Finally, we combine input and model manipulation. To this end we put a backdoor in the network: the explanation is correct unless there is a specific pattern present in the input, which triggers a malicious explanation. Our work raises new security concerns, especially in settings where explanations of models may be used to make decisions, such as in the medical domain.

arXiv commentPresented at BMVC 2019: Workshop on Interpretable and Explainable Machine Vision, Cardiff, UK. Updated to BMVC template

References in corpus (12)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Do Deep Nets Really Need to be Deep?
• Understanding Black-box Predictions via Influence Functions
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Learning how to explain neural networks: PatternNet and PatternAttribution
• The (Un)reliability of saliency methods
• Interpretation of Neural Networks is Fragile

Cited by in corpus (5)

• When Explanations Lie: Why Many Modified BP Attributions Fail
• Smoothed Geometry for Robust Attribution
• Zoom-CAM: Generating Fine-grained Pixel Annotations from Image Labels
• Gradient-based Analysis of NLP Models is Manipulable
• Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning