Text and Code Embeddings by Contrastive Pre-Training

Abstract

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

Cited by in corpus (12)

• GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
• Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysis
• QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance
• Automatic Assessment of Divergent Thinking in Chinese Language with TransDis: A Transformer-Based Language Model Approach
• Multilingual Name Entity Recognition and Intent Classification Employing Deep Learning Architectures
• Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming
• MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation
• Enhancing Procedural Writing Through Personalized Example Retrieval: A Case Study on Cooking Recipes
• Using Ensemble Inference to Improve Recall of Clone Detection
• Vectorizing string entries for data processing on tables: when are larger language models better?
• UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer Learning