ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Abstract

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8\% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

Accepted by ACL2021, 10 pages, 7 figures, 4 tables

References in corpus (8)

• A Simple Framework for Contrastive Learning of Visual Representations
• Improving neural networks by preventing co-adaptation of feature detectors
• Skip-Thought Vectors
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• CLEAR: Contrastive Learning for Sentence Representation
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation

Cited by in corpus (9)

• SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
• ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Towards the Generalization of Contrastive Self-Supervised Learning
• Simple Contrastive Representation Adversarial Learning for NLP Tasks
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations
• Coarse to Fine: Video Retrieval before Moment Localization
• MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models