Self-Guided Contrastive Learning for BERT Sentence Representations
arXiv:2106.07345
Abstract
Although BERT and its variants have reshaped the NLP landscape, it still remains unclear how best to derive sentence embeddings from such pre-trained Transformers. In this work, we propose a contrastive learning method that utilizes self-guidance for improving the quality of BERT sentence representations. Our method fine-tunes BERT in a self-supervised fashion, does not rely on data augmentation, and enables the usual [CLS] token embeddings to function as sentence vectors. Moreover, we redesign the contrastive learning objective (NT-Xent) and apply it to sentence representation learning. We demonstrate with extensive experiments that our approach is more effective than competitive baselines on diverse sentence-related tasks. We also show it is efficient at inference and robust to domain shifts.
ACL 2021
References in corpus (15)
- Distributed Representations of Words and Phrases and their Compositionality
- A Simple Framework for Contrastive Learning of Visual Representations
- Universal Sentence Encoder
- Contrastive Representation Learning: A Framework and Review
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- A large annotated corpus for learning natural language inference
- Exploring Simple Siamese Representation Learning
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- CLEAR: Contrastive Learning for Sentence Representation
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
- On the Sentence Embeddings from Pre-trained Language Models
- SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models