Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need

Abstract

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. \blfootnote{$†$ This work was done during an internship at Huawei.} In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods.

Camera-ready version, CIKM 2021 Full Paper, the codes are available at https://github.com/zhengyima/anchors

References in corpus (12)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Multi-Stage Document Ranking with BERT
• Understanding the Behaviors of BERT in Ranking
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Mining Missing Hyperlinks from Human Navigation Traces: A Case Study of Wikipedia
• Selective Weak Supervision for Neural Information Retrieval
• CoLAKE: Contextualized Language and Knowledge Embedding
• Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
• Neural Sentence Ordering Based on Constraint Graphs
• Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models