SentEval: An Evaluation Toolkit for Universal Sentence Representations

Abstract

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.

LREC 2018

References in corpus (3)

• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Charagram: Embedding Words and Sentences via Character n-grams

Cited by in corpus (61)

• CLEAR: Contrastive Learning for Sentence Representation
• Do Adversarially Robust ImageNet Models Transfer Better?
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
• Universal Text Representation from BERT: An Empirical Study
• Multilingual NMT with a language-independent attention bridge
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Pair-Level Supervised Contrastive Learning for Natural Language Inference
• Generating Diverse and Meaningful Captions
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Low Precision Decentralized Distributed Training over IID and non-IID Data
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Learning Compressed Sentence Representations for On-Device Text Processing
• Sentence Embeddings using Supervised Contrastive Learning
• Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation
• Continual Learning for Sentence Representations Using Conceptors
• DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective
• Contextual Lensing of Universal Sentence Representations
• 50 Ways to Bake a Cookie: Mapping the Landscape of Procedural Texts
• Semantic Similarity Measure of Natural Language Text through Machine Learning and a Keyword-Aware Cross-Encoder-Ranking Summarizer -- A Case Study Using UCGIS GIS&T Body of Knowledge
• When Do Discourse Markers Affect Computational Sentence Understanding?
• Neural Language Priors
• Learning Finer-class Networks for Universal Representations
• Improving Sentence Representations with Consensus Maximisation
• Learning Robust, Transferable Sentence Representations for Text Classification
• On the impressive performance of randomly weighted encoders in summarization tasks
• Is neural language acquisition similar to natural? A chronological probing study
• Backretrieval: An Image-Pivoted Evaluation Metric for Cross-Lingual Text Representations Without Parallel Corpora
• Word2rate: training and evaluating multiple word embeddings as statistical transitions
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Applying SoftTriple Loss for Supervised Language Model Fine Tuning
• Embedding Compression with Isotropic Iterative Quantization
• P-SIF: Document Embeddings Using Partition Averaging
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation
• Paraphrase Detection on Noisy Subtitles in Six Languages
• Lexicosyntactic Inference in Neural Models
• Modeling Text with Decision Forests using Categorical-Set Splits
• Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments
• Predictive Representation Learning for Language Modeling
• Fiction Sentence Expansion and Enhancement via Focused Objective and Novelty Curve Sampling
• COSTRA 1.0: A Dataset of Complex Sentence Transformations
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
• Hamming Sentence Embeddings for Information Retrieval
• RuSentEval: Linguistic Source, Encoder Force!
• Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
• Are Classes Clusters?
• In Search for Linear Relations in Sentence Embedding Spaces
• Sentence Embeddings for Russian NLU
• Learning and Evaluating Sparse Interpretable Sentence Embeddings
• Efficient Sentence Embedding via Semantic Subspace Analysis
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• TransAug: Translate as Augmentation for Sentence Embeddings