SimCSE: Simple Contrastive Learning of Sentence Embeddings

Abstract

This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation, and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to the previous best results. We also show -- both theoretically and empirically -- that the contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available.

Accepted to EMNLP 2021. The code and pre-trained models are available at https://github.com/princeton-nlp/simcse

References in corpus (4)

• Skip-Thought Vectors
• CLEAR: Contrastive Learning for Sentence Representation
• Whitening Sentence Representations for Better Semantics and Faster Retrieval
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Cited by in corpus (7)

• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• PAUSE: Positive and Annealed Unlabeled Sentence Embedding
• ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
• Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning