Evaluation of sentence embeddings in downstream and linguistic probing tasks

Abstract

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.

arXiv comment15 pages, 3 figures, 11 tables

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Universal Sentence Encoder
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• A large annotated corpus for learning natural language inference
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• A Survey of Word Embeddings Evaluation Methods
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Cited by in corpus (33)

• Linguistic Profiling of a Neural Language Model
• RP-DNN: A Tweet level propagation context based deep neural networks for early rumor detection in Social Media
• Investigating the Effects of Word Substitution Errors on Sentence Embeddings
• Neural Language Model Based Training Data Augmentation for Weakly Supervised Early Rumor Detection
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Evaluation of Sentence Representations in Polish
• Correlation Coefficients and Semantic Textual Similarity
• Text Embeddings for Retrieval From a Large Knowledge Base
• Finding Generalizable Evidence by Learning to Convince Q&A Models
• Inducing and Embedding Senses with Scaled Gumbel Softmax
• ntuer at SemEval-2019 Task 3: Emotion Classification with Word and Sentence Representations in RCNN
• Efficient Contextual Representation Learning Without Softmax Layer
• CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
• IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation
• Learning Robust, Transferable Sentence Representations for Text Classification
• DirectProbe: Studying Representations without Classifiers
• A Multi-Resolution Word Embedding for Document Retrieval from Large Unstructured Knowledge Bases
• P-SIF: Document Embeddings Using Partition Averaging
• Meta-Embeddings for Natural Language Inference and Semantic Similarity tasks
• Semantic Representation and Inference for NLP
• Cross-Lingual Transfer for Distantly Supervised and Low-resources Indonesian NER
• In Other News: A Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• Shareable Representations for Search Query Understanding
• COSTRA 1.0: A Dataset of Complex Sentence Transformations
• Retrofitting Contextualized Word Embeddings with Paraphrases
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
• SOAR: A Synthesis Approach for Data Science API Refactoring
• An Evaluation of Transfer Learning for Classifying Sales Engagement Emails at Large Scale
• Unsupervised Spoken Utterance Classification
• Correlations between Word Vector Sets
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training
• In Search for Linear Relations in Sentence Embedding Spaces