Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability

Abstract

Any prediction from a model is made by a combination of learning history and test stimuli. This provides significant insights for improving model interpretability: {\it because of which part(s) of which training example(s), the model attends to which part(s) of a test example}. Unfortunately, existing methods to interpret a model's predictions are only able to capture a single aspect of either test stimuli or learning history, and evidences from both are never combined or integrated. In this paper, we propose an efficient and differentiable approach to make it feasible to interpret a model's prediction by jointly examining training history and test stimuli. Test stimuli is first identified by gradient-based methods, signifying {\it the part of a test example that the model attends to}. The gradient-based saliency scores are then propagated to training examples using influence functions to identify {\it which part(s) of which training example(s)} make the model attends to the test stimuli. The system is differentiable and time efficient: the adoption of saliency scores from gradient-based methods allows us to efficiently trace a model's prediction through test stimuli, and then back to training examples through influence functions. We demonstrate that the proposed methodology offers clear explanations about neural model decisions, along with being useful for performing error analysis, crafting adversarial examples and fixing erroneously classified examples.

References in corpus (22)

• LSTM: A Search Space Odyssey
• Striving for Simplicity: The All Convolutional Net
• Learning Important Features Through Propagating Activation Differences
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Understanding Black-box Predictions via Influence Functions
• Visualizing and Understanding Recurrent Networks
• Attention is not Explanation
• Understanding Neural Networks through Representation Erasure
• Generating Natural Adversarial Examples
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Visualizing and Understanding Neural Models in NLP
• When Are Tree Structures Necessary for Deep Learning of Representations?
• HotFlip: White-Box Adversarial Examples for Text Classification
• CXPlain: Causal Explanations for Model Interpretation under Uncertainty
• Attention Interpretability Across NLP Tasks
• Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemble
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• Unsupervised Learning of Neural Networks to Explain Neural Networks
• Certified Robustness to Adversarial Word Substitutions
• LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• Interpretable Adversarial Training for Text

Cited by in corpus (5)

• Self-Explaining Structures Improve NLP Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• A General Framework for Defending Against Backdoor Attacks via Influence Graph
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation