Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
arXiv:2005.06676
Abstract
Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which 'saliency maps' may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.
ACL 2020
References in corpus (9)
- Towards A Rigorous Science of Interpretable Machine Learning
- Axiomatic Attribution for Deep Networks
- Learning Important Features Through Propagating Activation Differences
- Understanding Black-box Predictions via Influence Functions
- SmoothGrad: removing noise by adding noise
- Understanding Neural Networks through Representation Erasure
- e-SNLI: Natural Language Inference with Natural Language Explanations
- Pathologies of Neural Models Make Interpretations Difficult
- Deep Weighted Averaging Classifiers
Cited by in corpus (4)
- Explaining and Improving Model Behavior with k Nearest Neighbor Representations
- Interpreting Deep Learning Models in Natural Language Processing: A Review
- Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
- OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts