PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval

Abstract

Recently pre-trained language representation models such as BERT have shown great success when fine-tuned on downstream tasks including information retrieval (IR). However, pre-training objectives tailored for ad-hoc retrieval have not been well explored. In this paper, we propose Pre-training with Representative wOrds Prediction (PROP) for ad-hoc retrieval. PROP is inspired by the classical statistical language model for IR, specifically the query likelihood model, which assumes that the query is generated as the piece of text representative of the "ideal" document. Based on this idea, we construct the representative words prediction (ROP) task for pre-training. Given an input document, we sample a pair of word sets according to the document language model, where the set with higher likelihood is deemed as more representative of the document. We then pre-train the Transformer model to predict the pairwise preference between the two word sets, jointly with the Masked Language Model (MLM) objective. By further fine-tuning on a variety of representative downstream ad-hoc retrieval tasks, PROP achieves significant improvements over baselines without pre-training or with other pre-training methods. We also show that PROP can achieve exciting performance under both the zero- and low-resource IR settings. The code and pre-trained models are available at https://github.com/Albert-Ma/PROP.

arXiv commentAccepted by WSDM2021

References in corpus (6)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• CEDR: Contextualized Embeddings for Document Ranking
• DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval
• Content-Based Weak Supervision for Ad-Hoc Re-Ranking

Cited by in corpus (9)

• Information Retrieval: Recent Advances and Beyond
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking
• Learning To Rank Resources with GNN
• To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment
• Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
• A Contrastive Pre-training Approach to Learn Discriminative Autoencoder for Dense Retrieval
• Approximated Doubly Robust Search Relevance Estimation
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need