Understanding Neural Networks through Representation Erasure

Abstract

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

References in corpus (2)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Sparse Overcomplete Word Vector Representations

Cited by in corpus (9)

• ML-LOO: Detecting Adversarial Examples with Feature Attribution
• Interpretable Adversarial Training for Text
• Towards Interrogating Discriminative Machine Learning Models
• Second-Order Word Embeddings from Nearest Neighbor Topological Features
• Interpretable Question Answering on Knowledge Bases and Text
• Investigating how well contextual features are captured by bi-directional recurrent neural network models
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition
• Understanding Memory Modules on Learning Simple Algorithms
• A Deep Network with Visual Text Composition Behavior