An efficient framework for learning sentence representations

Abstract

In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

arXiv commentICLR 2018

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• Convolutional Neural Networks for Sentence Classification
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Autoencoding beyond pixels using a learned similarity metric
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Towards Universal Paraphrastic Sentence Embeddings
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• Order-Embeddings of Images and Language
• Multilingual Distributed Representations without Word Alignment
• Generating Sentences by Editing Prototypes
• Learning Generic Sentence Representations Using Convolutional Neural Networks

Cited by in corpus (91)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Contrastive Representation Learning: A Framework and Review
• What Makes for Good Views for Contrastive Learning?
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• CLEAR: Contrastive Learning for Sentence Representation
• Evaluation of Text Generation: A Survey
• Leveraging BERT for Extractive Text Summarization on Lectures
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• CrossCBR: Cross-view Contrastive Learning for Bundle Recommendation
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• How to Fine-Tune BERT for Text Classification?
• ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
• Correlating neural and symbolic representations of language
• Encoding word order in complex embeddings
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Deep Learning-based Sentiment Classification: A Comparative Survey
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Contrastive Learning with Hard Negative Samples
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Transfer Fine-Tuning: A BERT Case Study
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• Extracting Sentence Embeddings from Pretrained Transformer Models
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• Contrastive Learning Inverts the Data Generating Process
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Machine Translation Evaluation with BERT Regressor
• Contrastive learning, multi-view redundancy, and linear models
• Debiased Contrastive Learning
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• Multilingual Question Answering from Formatted Text applied to Conversational Agents
• Information Leakage in Embedding Models
• One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases
• Sentence Embeddings using Supervised Contrastive Learning
• Smoothed Contrastive Learning for Unsupervised Sentence Embedding
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
• Adversarial Training with Contrastive Learning in NLP
• Sentence Analogies: Exploring Linguistic Relationships and Regularities in Sentence Embeddings
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Language-Driven Image Style Transfer
• Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
• Proof Artifact Co-training for Theorem Proving with Language Models
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• DeepChannel: Salience Estimation by Contrastive Learning for Extractive Document Summarization
• Unsupervised Learning of Sentence Representations Using Sequence Consistency
• Efficient Domain Adaptation of Sentence Embeddings Using Adapters
• UER: An Open-Source Toolkit for Pre-training Models
• Multi-view Sentence Representation Learning
• PAC-Bayesian Contrastive Unsupervised Representation Learning
• AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge
• Pre-Training for Query Rewriting in A Spoken Language Understanding System
• A Novel Distributed Representation of News (DRNews) for Stock Market Predictions
• When Do Discourse Markers Affect Computational Sentence Understanding?
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
• Prior Art Search and Reranking for Generated Patent Text
• A Survey of Embedding Space Alignment Methods for Language and Knowledge Graphs
• Dialog Intent Induction with Deep Multi-View Clustering
• Structure-Aware Generation Network for Recipe Generation from Images
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Improving Sentence Representations with Consensus Maximisation
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity
• Learning Robust, Transferable Sentence Representations for Text Classification
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• Understanding Chat Messages for Sticker Recommendation in Messaging Apps
• Event Presence Prediction Helps Trigger Detection Across Languages
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• Leveraging Sequence Embedding and Convolutional Neural Network for Protein Function Prediction
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• Learning Universal Sentence Representations with Mean-Max Attention Autoencoder
• Adversarial Decomposition of Text Representation
• Memory Consolidation for Contextual Spoken Language Understanding with Dialogue Logistic Inference
• A new approach for extracting the conceptual schema of texts based on the linguistic Thematic Progression theory
• A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• BURT: BERT-inspired Universal Representation from Twin Structure
• $C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
• Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• Predictions For Pre-training Language Models
• Are Classes Clusters?
• Hamming Sentence Embeddings for Information Retrieval
• Parameter-Efficient Neural Question Answering Models via Graph-Enriched Document Representations
• Parameter-free Sentence Embedding via Orthogonal Basis