Condenser: a Pre-training Architecture for Dense Retrieval

Abstract

Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

arXiv commentEMNLP 2021

References in corpus (10)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Universal Sentence Encoder
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling