Gradients of Counterfactuals

Abstract

Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a ligand-based virtual screening network with categorical features and an LSTM based language model for the Penn Treebank dataset. We visualize how interior gradients better capture feature importance. Furthermore, interior gradients are applicable to a wide variety of deep networks, and have the attribution property that the feature importance scores sum to the the prediction score. Best of all, interior gradients can be computed just as easily as gradients. In contrast, previous methods are complex to implement, which hinders practical adoption.

References in corpus (3)

• Striving for Simplicity: The All Convolutional Net
• Understanding Neural Networks Through Deep Visualization
• Axiomatic Attribution for Multilinear Functions

Cited by in corpus (21)

• Learning Important Features Through Propagating Activation Differences
• A Survey on the Explainability of Supervised Machine Learning
• A new interpretable unsupervised anomaly detection method based on residual explanation
• DeepFaceLIFT: Interpretable Personalized Models for Automatic Estimation of Self-Reported Pain
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• Mutual Information for Explainable Deep Learning of Multiscale Systems
• Explaining Visual Models by Causal Attribution
• Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps
• Towards Interrogating Discriminative Machine Learning Models
• Generalizability Analysis of Graph-based Trajectory Predictor with Vectorized Representation
• A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks
• Investigating Saturation Effects in Integrated Gradients
• Explaining Regression Based Neural Network Model
• Ontology-based Interpretable Machine Learning for Textual Data
• The Generalizability of Explanations
• Do Input Gradients Highlight Discriminative Features?
• An Experimental Study of Semantic Continuity for Deep Learning Models
• Rank Projection Trees for Multilevel Neural Network Interpretation
• Balancing Robustness and Sensitivity using Feature Contrastive Learning
• Explainability-Aware One Point Attack for Point Cloud Neural Networks
• Logic Constraints to Feature Importances