Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction

Abstract

Dense retrieval has shown promising results in many information retrieval (IR) related tasks, whose foundation is high-quality text representation learning for effective search. Some recent studies have shown that autoencoder-based language models are able to boost the dense retrieval performance using a weak decoder. However, we argue that 1) it is not discriminative to decode all the input texts and, 2) even a weak decoder has the bypass effect on the encoder. Therefore, in this work, we introduce a novel contrastive span prediction task to pre-train the encoder alone, but still retain the bottleneck ability of the autoencoder. % Therefore, in this work, we propose to drop out the decoder and introduce a novel contrastive span prediction task to pre-train the encoder alone. The key idea is to force the encoder to generate the text representation close to its own random spans while far away from others using a group-wise contrastive loss. In this way, we can 1) learn discriminative text representations efficiently with the group-wise contrastive learning over spans and, 2) avoid the bypass effect of the decoder thoroughly. Comprehensive experiments over publicly available retrieval benchmark datasets show that our approach can outperform existing pre-training methods for dense retrieval significantly.

arXiv commentAccepted to SIGIR 2022

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• ERNIE: Enhanced Representation through Knowledge Integration
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Overview of the TREC 2020 deep learning track
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Few-Shot Conversational Dense Retrieval
• RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
• PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval
• Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval
• B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval

Cited by in corpus (6)

• CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
• Continual Learning for Generative Retrieval over Dynamic Corpora
• Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
• HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
• A Contrastive Pre-training Approach to Learn Discriminative Autoencoder for Dense Retrieval
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval