Interpreting Deep Learning Models in Natural Language Processing: A Review

Abstract

Neural network models have achieved state-of-the-art performances in a wide range of natural language processing (NLP) tasks. However, a long-standing criticism against neural network models is the lack of interpretability, which not only reduces the reliability of neural NLP systems but also limits the scope of their applications in areas where interpretability is essential (e.g., health care applications). In response, the increasing interest in interpreting neural NLP models has spurred a diverse array of interpretation methods over recent years. In this survey, we provide a comprehensive review of various interpretation methods for neural models in NLP. We first stretch out a high-level taxonomy for interpretation methods in NLP, i.e., training-based approaches, test-based approaches, and hybrid approaches. Next, we describe sub-categories in each category in detail, e.g., influence-function based methods, KNN-based methods, attention-based models, saliency-based methods, perturbation-based methods, etc. We point out deficiencies of current methods and suggest some avenues for future research.

References in corpus (73)

• Distilling the Knowledge in a Neural Network
• LSTM: A Search Space Odyssey
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Categorical Reparameterization with Gumbel-Softmax
• Character-level Convolutional Networks for Text Classification
• Language Models are Few-Shot Learners
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Learning Important Features Through Propagating Activation Differences
• Understanding Neural Networks Through Deep Visualization
• A Structured Self-attentive Sentence Embedding
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Understanding Black-box Predictions via Influence Functions
• Convolutional Neural Networks for Sentence Classification
• Visualizing and Understanding Recurrent Networks
• ERNIE: Enhanced Representation through Knowledge Integration
• SmoothGrad: removing noise by adding noise
• A C-LSTM Neural Network for Text Classification
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Attention is not Explanation
• Understanding Neural Networks through Representation Erasure
• A large annotated corpus for learning natural language inference
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Assessing BERT's Syntactic Abilities
• Learning how to explain neural networks: PatternNet and PatternAttribution
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Towards a Human-like Open-Domain Chatbot
• Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Visualizing and Understanding Neural Models in NLP
• Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions
• Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Rethinking Attention with Performers
• Understanding Back-Translation at Scale
• Entailment as Few-Shot Learner
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Hierarchical interpretations for neural network predictions
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Extraction of Salient Sentences from Labelled Documents
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Full-Gradient Representation for Neural Network Visualization
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF
• Attention Interpretability Across NLP Tasks
• DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
• A Unified MRC Framework for Named Entity Recognition
• BertGCN: Transductive Text Classification by Combining GCN and BERT
• A Game Theoretic Approach to Class-wise Selective Rationalization
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• Chinese NER Using Lattice LSTM
• LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
• Learning Gender-Neutral Word Embeddings
• Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
• The Price of Interpretability
• Explaining and Improving Model Behavior with k Nearest Neighbor Representations
• Embedding Words as Distributions with a Bayesian Skip-gram Model
• Self-Explaining Structures Improve NLP Models
• Towards Explainable NLP: A Generative Explanation Framework for Text Classification
• Lessons on Parameter Sharing across Layers in Transformers
• Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction
• Inducing Syntactic Trees from BERT Representations
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• Model Explainability in Deep Learning Based Natural Language Processing
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
• Distribution Matching for Rationalization