Understanding the Behaviors of BERT in Ranking

Abstract

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.

There is an error in Table 1 and we will update them to correct results. Please refer to MS MARCO Leaderboard for the actually evaluation results

References in corpus (2)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling

Cited by in corpus (24)

• Query Resolution for Conversational Search with Limited Supervision
• Hybrid Ranking Network for Text-to-SQL
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Latin BERT: A Contextual Language Model for Classical Philology
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• A Multi-task Learning Framework for Product Ranking with BERT
• P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• News Article Retrieval in Context for Event-centric Narrative Creation
• A Comparison of Supervised Learning to Match Methods for Product Search
• Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
• Context-based Transformer Models for Answer Sentence Selection
• An Audio-enriched BERT-based Framework for Spoken Multiple-choice Question Answering
• Beyond Lexical: A Semantic Retrieval Framework for Textual SearchEngine
• Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning
• A multi-perspective combined recall and rank framework for Chinese procedure terminology normalization
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
• DeText: A Deep Text Ranking Framework with BERT
• BERT Embeddings Can Track Context in Conversational Search
• Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering
• A neural document language modeling framework for spoken document retrieval
• TPRM: A Topic-based Personalized Ranking Model for Web Search
• ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss
• Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints