Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods

Abstract

Saliency methods interpret the prediction of a neural network by showing the importance of input elements for that prediction. A popular family of saliency methods utilize gradient information. In this work, we empirically show that two approaches for handling the gradient information, namely positive aggregation, and positive propagation, break these methods. Though these methods reflect visually salient information in the input, they do not explain the model prediction anymore as the generated saliency maps are insensitive to the predicted output and are insensitive to model parameter randomization. Specifically for methods that aggregate the gradients of a chosen layer such as GradCAM++ and FullGrad, exclusively aggregating positive gradients is detrimental. We further support this by proposing several variants of aggregation methods with positive handling of gradient information. For methods that backpropagate gradient information such as LRP, RectGrad, and Guided Backpropagation, we show the destructive effect of exclusively propagating positive gradient information.

arXiv commentICML 2020 - Workshop on Human Interpretability in Machine Learning - Spotlight paper - Video at http://whi2020.online/poster_40.html

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Axiomatic Attribution for Deep Networks
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Sanity Checks for Saliency Maps
• A Benchmark for Interpretability Methods in Deep Neural Networks
• The (Un)reliability of saliency methods
• A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
• Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps
• Improving Feature Attribution through Input-specific Network Pruning