Full-Gradient Representation for Neural Network Visualization

Abstract

We introduce a new tool for interpreting neural net responses, namely full-gradients, which decomposes the neural net response into input sensitivity and per-neuron sensitivity components. This is the first proposed representation which satisfies two key properties: completeness and weak dependence, which provably cannot be satisfied by any saliency map-based interpretability method. For convolutional nets, we also propose an approximate saliency map representation, called FullGrad, obtained by aggregating the full-gradient components. We experimentally evaluate the usefulness of FullGrad in explaining model behaviour with two quantitative tests: pixel perturbation and remove-and-retrain. Our experiments reveal that our method explains model behaviour correctly, and more comprehensively than other methods in the literature. Visual inspection also reveals that our saliency maps are sharper and more tightly confined to object regions than other methods.

arXiv commentNeurIPS 2019

Cited by in corpus (24)

• Impossibility Theorems for Feature Attribution
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Accurate Virus Identification with Interpretable Raman Signatures by Machine Learning
• AeroRIT: A New Scene for Hyperspectral Image Analysis
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Knee Injury Detection using MRI with Efficiently-Layered Network (ELNet)
• Deeply Explain CNN via Hierarchical Decomposition
• Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
• Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• Improving Feature Attribution through Input-specific Network Pruning
• Optimising for Interpretability: Convolutional Dynamic Alignment Networks
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications
• On a Sparse Shortcut Topology of Artificial Neural Networks
• Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
• Explaining neural network predictions of material strength
• Evaluation of Saliency-based Explainability Method
• Explainability Guided Multi-Site COVID-19 CT Classification
• Shapley Explanation Networks
• Measuring and improving the quality of visual explanations
• Towards Understanding the Effectiveness of Attention Mechanism
• Weakly Supervised Recovery of Semantic Attributes
• How to Explain Neural Networks: an Approximation Perspective
• Ada-SISE: Adaptive Semantic Input Sampling for Efficient Explanation of Convolutional Neural Networks