Understanding Neural Networks through Representation Erasure

Abstract

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

References in corpus (13)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Intriguing properties of neural networks
• Sequence Level Training with Recurrent Neural Networks
• Visualizing and Understanding Recurrent Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Visualizing and Understanding Neural Models in NLP
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Aspect Level Sentiment Classification with Deep Memory Network
• Representation of linguistic form and function in recurrent neural networks
• LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
• Sparse Overcomplete Word Vector Representations

Cited by in corpus (145)

• Towards Explainable Artificial Intelligence
• Attention is not Explanation
• BAE: BERT-based Adversarial Examples for Text Classification
• Generating Natural Adversarial Examples
• Learning to Generate Reviews and Discovering Sentiment
• Pathologies of Neural Models Make Interpretations Difficult
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• Representations of language in a model of visually grounded speech signal
• On the Explainability of Natural Language Processing Deep Models
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
• Hierarchical interpretations for neural network predictions
• On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations
• Before Name-calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation
• Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
• Attack Graph Convolutional Networks by Adding Fake Nodes
• Patient representation learning and interpretable evaluation using clinical notes
• On Interpretability of Artificial Neural Networks: A Survey
• Explain and Predict, and then Predict Again
• Toward Accurate Interpretable Predictions of Materials Properties within Transformer Language Models
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• How does this interaction affect me? Interpretable attribution for feature interactions
• Description Based Text Classification with Reinforcement Learning
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data
• Security Matters: A Survey on Adversarial Machine Learning
• Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training
• IcoRating: A Deep-Learning System for Scam ICO Identification
• Combating Adversarial Misspellings with Robust Word Recognition
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Self-Explaining Structures Improve NLP Models
• ferret: a Framework for Benchmarking Explainers on Transformers
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
• ML-LOO: Detecting Adversarial Examples with Feature Attribution
• On Learning and Learned Data Representation by Capsule Networks
• Evaluations and Methods for Explanation through Robustness Analysis
• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• Saliency Learning: Teaching the Model Where to Pay Attention
• HotelRec: a Novel Very Large-Scale Hotel Recommendation Dataset
• Revisiting the Hierarchical Multiscale LSTM
• What made you do this? Understanding black-box decisions with sufficient input subsets
• Towards Understanding Neural Machine Translation with Word Importance
• Heterogeneous Graph Neural Networks with Post-hoc Explanations for Multi-modal and Explainable Land Use Inference
• Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
• Interpretable Adversarial Training for Text
• Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models
• Do End-to-End Speech Recognition Models Care About Context?
• Learning to Attack: Towards Textual Adversarial Attacking in Real-world Situations
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
• Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations
• Order in the Court: Explainable AI Methods Prone to Disagreement
• DQI: Measuring Data Quality in NLP
• Identify Susceptible Locations in Medical Records via Adversarial Attacks on Deep Predictive Models
• Adversarial Attacks and Defenses: An Interpretation Perspective
• AQEyes: Visual Analytics for Anomaly Detection and Examination of Air Quality Data
• Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness
• ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models
• Interpretation of NLP models through input marginalization
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Towards Interrogating Discriminative Machine Learning Models
• Semantics Preserving Adversarial Learning
• Evaluating Saliency Methods for Neural Language Models
• What Context Features Can Transformer Language Models Use?
• LS-Tree: Model Interpretation When the Data Are Linguistic
• Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks
• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script
• Finding Generalizable Evidence by Learning to Convince Q&A Models
• Learning Credible Deep Neural Networks with Rationale Regularization
• Visualizing RNN States with Predictive Semantic Encodings
• Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
• Reliable Deep Grade Prediction with Uncertainty Estimation
• Measuring Faithful and Plausible Visual Grounding in VQA
• Second-Order Word Embeddings from Nearest Neighbor Topological Features
• Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness
• It Takes Two Flints to Make a Fire: Multitask Learning of Neural Relation and Explanation Classifiers
• Discretized Integrated Gradients for Explaining Language Models
• This Reads Like That: Deep Learning for Interpretable Natural Language Processing
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• Analysis Methods in Neural Language Processing: A Survey
• DSReg: Using Distant Supervision as a Regularizer
• Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity
• DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction
• Attention Flows are Shapley Value Explanations
• Interpretable Question Answering on Knowledge Bases and Text
• Generalizing Natural Language Analysis through Span-relation Representations
• Tasty Burgers, Soggy Fries: Probing Aspect Robustness in Aspect-Based Sentiment Analysis
• SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• Neural Image Compression and Explanation
• MSSRNet: Manipulating Sequential Style Representation for Unsupervised Text Style Transfer
• Investigating how well contextual features are captured by bi-directional recurrent neural network models
• On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation
• Evaluating the Reliability of Self-Explanations in Large Language Models
• Lost In Translation: Generating Adversarial Examples Robust to Round-Trip Translation
• Geometry matters: Exploring language examples at the decision boundary
• On Attribution of Recurrent Neural Network Predictions via Additive Decomposition
• Exclusion and Inclusion -- A model agnostic approach to feature importance in DNNs
• Making Document-Level Information Extraction Right for the Right Reasons
• Human-grounded Evaluations of Explanation Methods for Text Classification
• Evaluating the Values of Sources in Transfer Learning
• Challenge AI Mind: A Crowd System for Proactive AI Testing
• Extractive Adversarial Networks: High-Recall Explanations for Identifying Personal Attacks in Social Media Posts
• Explainability-aided Domain Generalization for Image Classification
• Logic Traps in Evaluating Attribution Scores
• SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning
• Interpreting Deep Learning Model Using Rule-based Method
• Training Machine Learning Models by Regularizing their Explanations
• Fine-grained Interpretation and Causation Analysis in Deep NLP Models
• Natural Adversarial Sentence Generation with Gradient-based Perturbation
• A Deep Network with Visual Text Composition Behavior
• Detecting and Reducing Bias in a High Stakes Domain
• Sparsity Emerges Naturally in Neural Language Models
• Explaining the Deep Natural Language Processing by Mining Textual Interpretable Features
• Target Model Agnostic Adversarial Attacks with Query Budgets on Language Understanding Models
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition
• Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience
• T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLP
• Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification
• Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers
• MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding
• Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing
• Improving Moderation of Online Discussions via Interpretable Neural Models
• A Framework for Rationale Extraction for Deep QA models
• Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
• Adversarial Machine Learning in Text Analysis and Generation
• Distilling neural networks into skipgram-level decision lists
• AURORA: Auditing PageRank on Large Graphs
• Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
• Understanding Memory Modules on Learning Simple Algorithms
• Counterfactual Explanation with Multi-Agent Reinforcement Learning for Drug Target Prediction