Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Abstract

There is a lot of research interest in encoding variable length sentences into fixed length vectors, in a way that preserves the sentence meanings. Two common methods include representations based on averaging word vectors, and representations based on the hidden states of recurrent neural networks such as LSTMs. The sentence vectors are used as features for subsequent machine learning tasks or for pre-training in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they capture. We propose a framework that facilitates better understanding of the encoded representations. We define prediction tasks around isolated aspects of sentence structure (namely sentence length, word content, and word order), and score representations by the ability to train a classifier to solve each prediction task when using the representation as input. We demonstrate the potential contribution of the approach by analyzing different sentence representation mechanisms. The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Visualizing and Understanding Recurrent Networks
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• rnn : Recurrent Library for Torch
• Representation of linguistic form and function in recurrent neural networks

Cited by in corpus (44)

• Pointer Sentinel Mixture Models
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• What do Neural Machine Translation Models Learn about Morphology?
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Representations of language in a model of visually grounded speech signal
• Syntactic Structure from Deep Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
• Language Models are Open Knowledge Graphs
• Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias
• Linguistic Profiling of a Neural Language Model
• Local Interpretations for Explainable Natural Language Processing: A Survey
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Probing for the Usage of Grammatical Number
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Analyzing analytical methods: The case of phonology in neural models of spoken language
• Transformers and the representation of biomedical background knowledge
• Lessons from Natural Language Inference in the Clinical Domain
• Pre-Trained Models: Past, Present and Future
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Can LSTM Learn to Capture Agreement? The Case of Basque
• Probing artificial neural networks: insights from neuroscience
• Probing Neural Dialog Models for Conversational Understanding
• Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
• Data Augmentation for Neural Online Chat Response Selection
• Representing Sentences as Low-Rank Subspaces
• Discovery of Natural Language Concepts in Individual Units of CNNs
• Trick or TReAT: Thematic Reinforcement for Artistic Typography
• Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
• On the Interplay Between Fine-tuning and Composition in Transformers
• Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks
• Analysis of Bag-of-n-grams Representation's Properties Based on Textual Reconstruction
• Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• Lexicosyntactic Inference in Neural Models
• Probing Task-Oriented Dialogue Representation from Language Models
• Predicting In-game Actions from Interviews of NBA Players
• Learning Noun Cases Using Sequential Neural Networks
• RuSentEval: Linguistic Source, Encoder Force!
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
• TextContourNet: a Flexible and Effective Framework for Improving Scene Text Detection Architecture with a Multi-task Cascade
• Bag-of-Vector Embeddings of Dependency Graphs for Semantic Induction
• Evaluating the Representational Hub of Language and Vision Models