A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

Abstract

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.

arXiv comment21 pages, ICML 2018 (We revised the proofs of Theorem 1 and 2 in Appendix)

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Intriguing properties of neural networks
• How transferable are features in deep neural networks?
• Striving for Simplicity: The All Convolutional Net
• Learning Important Features Through Propagating Activation Differences
• Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Sub-Gaussian estimators of the mean of a random vector

Cited by in corpus (13)

• Interpretable machine learning: definitions, methods, and applications
• Impossibility Theorems for Feature Attribution
• Visualizing Deep Networks by Optimizing with Integrated Gradients
• Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
• Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• Hide-and-Seek: A Template for Explainable AI
• Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods
• Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings
• Explaining Deep Learning Representations by Tracing the Training Process