Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps

Abstract

Saliency Map, the gradient of the score function with respect to the input, is the most basic technique for interpreting deep neural network decisions. However, saliency maps are often visually noisy. Although several hypotheses were proposed to account for this phenomenon, there are few works that provide rigorous analyses of noisy saliency maps. In this paper, we firstly propose a new hypothesis that noise may occur in saliency maps when irrelevant features pass through ReLU activation functions. Then, we propose Rectified Gradient, a method that alleviates this problem through layer-wise thresholding during backpropagation. Experiments with neural networks trained on CIFAR-10 and ImageNet showed effectiveness of our method and its superiority to other attribution methods.

arXiv commentAccepted at the 2019 ICCV Workshop on Interpreting and Explaining Visual AI Models (VXAI 2019)

References in corpus (10)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Theoretically Principled Trade-off between Robustness and Accuracy
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• On the importance of single directions for generalization
• Gradients of Counterfactuals
• Bridging Adversarial Robustness and Gradient Interpretability

Cited by in corpus (5)

• Interpreting deep learning models for weak lensing
• When Explanations Lie: Why Many Modified BP Attributions Fail
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods
• On Spectral Properties of Gradient-based Explanation Methods