Pre-training Tasks for Embedding-based Large-scale Retrieval

Abstract

We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset of candidate documents. The scoring phase then re-ranks the documents. Critically, the retrieval algorithm not only desires high recall but also requires to be highly efficient, returning candidates in time sublinear to the number of documents. Unlike the scoring phase witnessing significant advances recently due to the BERT-style pre-training tasks on cross-attention models, the retrieval phase remains less well studied. Most previous works rely on classic Information Retrieval (IR) methods such as BM-25 (token matching + TF-IDF weights). These models only accept sparse handcrafted features and can not be optimized for different downstream tasks of interest. In this paper, we conduct a comprehensive study on the embedding-based retrieval models. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks. With adequately designed paragraph-level pre-training tasks, the Transformer models can remarkably improve over the widely-used BM-25 as well as embedding models without Transformers. The paragraph-level pre-training tasks we studied are Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki Link Prediction (WLP), and the combination of all three.

Accepted by ICLR 2020

Cited by in corpus (43)

• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documents
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Self-supervised Learning for Large-scale Item Recommendations
• Embedding-based Zero-shot Retrieval through Query Generation
• Neural Passage Retrieval with Improved Negative Contrast
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification
• B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
• MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Taming Pretrained Transformers for Extreme Multi-label Text Classification
• Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval
• Condenser: a Pre-training Architecture for Dense Retrieval
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• CsFEVER and CTKFacts: Acquiring Czech data for fact verification
• Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• A Graph-guided Multi-round Retrieval Method for Conversational Open-domain Question Answering
• SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
• PSSL: Self-supervised Learning for Personalized Search with Contrastive Sampling
• Cross-Lingual Training with Dense Retrieval for Document Retrieval
• Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
• Towards Robust Neural Retrieval Models with Synthetic Pre-Training
• Dense Hierarchical Retrieval for Open-Domain Question Answering
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• Crossing Variational Autoencoders for Answer Retrieval
• Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
• An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
• Technical Question Answering across Tasks and Domains
• Extreme Multi-label Learning for Semantic Matching in Product Search
• Graph-based Multilingual Product Retrieval in E-commerce Search
• YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker