Visualizing and Understanding Neural Models in NLP

Abstract

While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret. For example it's not clear how they achieve {\em compositionality}, building sentence meaning from the meanings of words and phrases. In this paper we describe four strategies for visualizing compositionality in neural models for NLP, inspired by similar work in computer vision. We first plot unit values to visualize compositionality of negation, intensification, and concessive clauses, allow us to see well-known markedness asymmetries in negation. We then introduce three simple and straightforward methods for visualizing a unit's {\em salience}, the amount it contributes to the final composed meaning: (1) gradient back-propagation, (2) the variance of a token from the average word node, (3) LSTM-style gates that measure information flow. We test our methods on sentiment using simple recurrent nets and LSTMs. Our general-purpose methods may have wide applications for understanding compositionality and other semantic properties of deep networks , and also shed light on why LSTMs outperform simple recurrent nets,

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• A Neural Conversational Model
• Visualizing and Understanding Recurrent Networks
• Addressing the Rare Word Problem in Neural Machine Translation
• Understanding Deep Image Representations by Inverting Them
• Sparse Overcomplete Word Vector Representations

Cited by in corpus (54)

• Understanding Neural Networks through Representation Erasure
• Predicting Process Behaviour using Deep Learning
• Selective Encoding for Abstractive Sentence Summarization
• Pathologies of Neural Models Make Interpretations Difficult
• Explainable AI: current status and future directions
• Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study
• End-to-End Neural Sentence Ordering Using Pointer Network
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Neural Sentence Ordering
• deepMiRGene: Deep Neural Network based Precursor microRNA Prediction
• Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk
• Analyzing COVID-19 on Online Social Media: Trends, Sentiments and Emotions
• Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data
• IcoRating: A Deep-Learning System for Scam ICO Identification
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Self-Explaining Structures Improve NLP Models
• Deep Motif Dashboard: Visualizing and Understanding Genomic Sequences Using Deep Neural Networks
• xGEMs: Generating Examplars to Explain Black-Box Models
• Representation of linguistic form and function in recurrent neural networks
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Techniques for visualizing LSTMs applied to electrocardiograms
• Deep Active Learning for Text Classification with Diverse Interpretations
• How recurrent networks implement contextual processing in sentiment analysis
• An Interpretability Illusion for BERT
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
• Do Vision Models Encode Object-Level Semantic Relatedness? A Cognitive Psychology-Inspired Benchmark
• Interpretation of NLP models through input marginalization
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks
• Visualizing RNN States with Predictive Semantic Encodings
• DNN2LR: Interpretation-inspired Feature Crossing for Real-world Tabular Data
• Interpreting search result rankings through intent modeling
• Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search
• Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition
• GRACE: Generating Concise and Informative Contrastive Sample to Explain Neural Network Model's Prediction
• DSReg: Using Distant Supervision as a Regularizer
• Does it care what you asked? Understanding Importance of Verbs in Deep Learning QA System
• Corpus-Level End-to-End Exploration for Interactive Systems
• Investigating how well contextual features are captured by bi-directional recurrent neural network models
• Effective Visualization and Analysis of Recommender Systems
• Learning Representations of Social Media Users
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings
• Not All Features Are Equal: Feature Leveling Deep Neural Networks for Better Interpretation
• Understanding Recurrent Neural State Using Memory Signatures
• Multi-Granular Text Encoding for Self-Explaining Categorization
• SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning
• Investigating Inner Properties of Multimodal Representation and Semantic Compositionality with Brain-based Componential Semantics
• DNN2LR: Automatic Feature Crossing for Credit Scoring
• Learning Syntactic and Dynamic Selective Encoding for Document Summarization
• Learning Noun Cases Using Sequential Neural Networks
• Task-specific Word Identification from Short Texts Using a Convolutional Neural Network
• A Deep Network with Visual Text Composition Behavior
• Sparsity Emerges Naturally in Neural Language Models