Interpretation of Prediction Models Using the Input Gradient

Abstract

State of the art machine learning algorithms are highly optimized to provide the optimal prediction possible, naturally resulting in complex models. While these models often outperform simpler more interpretable models by order of magnitudes, in terms of understanding the way the model functions, we are often facing a "black box". In this paper we suggest a simple method to interpret the behavior of any predictive model, both for regression and classification. Given a particular model, the information required to interpret it can be obtained by studying the partial derivatives of the model with respect to the input. We exemplify this insight by interpreting convolutional and multi-layer neural networks in the field of natural language processing.

Presented at NIPS 2016 Workshop on Interpretable Machine Learning in Complex Systems

References in corpus (1)

• Convolutional Neural Networks for Sentence Classification

Cited by in corpus (8)

• Saliency Learning: Teaching the Model Where to Pay Attention
• Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
• NLS: an accurate and yet easy-to-interpret regression method
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• Combining Similarity and Adversarial Learning to Generate Visual Explanation: Application to Medical Image Classification
• Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution
• Distilling neural networks into skipgram-level decision lists