CLEAR: Contrastive Learning for Sentence Representation

Abstract

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

10 pages, 2 figures

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• A Simple Framework for Contrastive Learning of Visual Representations
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Contrastive Multiview Coding
• Pointer Sentinel Mixture Models
• A large annotated corpus for learning natural language inference
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Contrastive Code Representation Learning
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
• Certified Robustness to Adversarial Word Substitutions

Cited by in corpus (35)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• A Survey on Data Augmentation for Text Classification
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation
• SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation
• ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Towards the Generalization of Contrastive Self-Supervised Learning
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• RadBERT-CL: Factually-Aware Contrastive Learning For Radiology Report Classification
• High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Weakly Supervised Learning for Analyzing Political Campaigns on Facebook
• SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
• MUSE: Music Recommender System with Shuffle Play Recommendation Enhancement
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
• DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Unsupervised Document Embedding via Contrastive Augmentation
• Supervised Graph Contrastive Pretraining for Text Classification
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
• SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene
• False Negative Distillation and Contrastive Learning for Personalized Outfit Recommendation
• Pairwise Supervised Contrastive Learning of Sentence Representations
• Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering
• A Survey on Green Deep Learning
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• Contrastive Document Representation Learning with Graph Attention Networks
• Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning