Interpretation of Prediction Models Using the Input Gradient

Abstract

State of the art machine learning algorithms are highly optimized to provide the optimal prediction possible, naturally resulting in complex models. While these models often outperform simpler more interpretable models by order of magnitudes, in terms of understanding the way the model functions, we are often facing a "black box". In this paper we suggest a simple method to interpret the behavior of any predictive model, both for regression and classification. Given a particular model, the information required to interpret it can be obtained by studying the partial derivatives of the model with respect to the input. We exemplify this insight by interpreting convolutional and multi-layer neural networks in the field of natural language processing.

Presented at NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems

References in corpus (3)

• Convolutional Neural Networks for Sentence Classification
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings

Cited by in corpus (21)

• Feature Importance Ranking for Deep Learning
• Sampling, Intervention, Prediction, Aggregation: A Generalized Framework for Model-Agnostic Interpretations
• Detecting Statistical Interactions from Neural Network Weights
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Learning Global Pairwise Interactions with Bayesian Neural Networks
• Saliency Learning: Teaching the Model Where to Pay Attention
• Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
• False Discovery Rate Control via Data Splitting
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• Order in the Court: Explainable AI Methods Prone to Disagreement
• Fibres of Failure: Classifying errors in predictive processes
• NLS: an accurate and yet easy-to-interpret regression method
• Explainable Product Search with a Dynamic Relation Embedding Model
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• Combining Similarity and Adversarial Learning to Generate Visual Explanation: Application to Medical Image Classification
• Evaluating the Reliability of Self-Explanations in Large Language Models
• Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
• Neural Gaussian Mirror for Controlled Feature Selection in Neural Networks
• Training Machine Learning Models by Regularizing their Explanations
• Distilling neural networks into skipgram-level decision lists