Understanding the Behaviors of BERT in Ranking
arXiv:1904.07531
Abstract
This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.
arXiv commentThere is an error in Table 1 and we will update them to correct results. Please refer to MS MARCO Leaderboard for the actually evaluation results
References in corpus (3)
Cited by in corpus (41)
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Natural Language Processing for Smart Healthcare
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Information Retrieval: Recent Advances and Beyond
- Query Resolution for Conversational Search with Limited Supervision
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Hybrid Ranking Network for Text-to-SQL
- Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
- Neural ranking models for document retrieval
- Latin BERT: A Contextual Language Model for Classical Philology
- Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
- A Multi-task Learning Framework for Product Ranking with BERT
- P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
- A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints
- Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
- An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
- News Article Retrieval in Context for Event-centric Narrative Creation
- Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
- BERT-QE: Contextualized Query Expansion for Document Re-ranking
- A Comparison of Supervised Learning to Match Methods for Product Search
- OpenMatch: An Open Source Library for Neu-IR Research
- Context-based Transformer Models for Answer Sentence Selection
- Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning
- Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
- Beyond Lexical: A Semantic Retrieval Framework for Textual SearchEngine
- Ranking Creative Language Characteristics in Small Data Scenarios
- An Audio-enriched BERT-based Framework for Spoken Multiple-choice Question Answering
- PSSL: Self-supervised Learning for Personalized Search with Contrastive Sampling
- Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
- GLOW : Global Weighted Self-Attention Network for Web Search
- DeText: A Deep Text Ranking Framework with BERT
- A multi-perspective combined recall and rank framework for Chinese procedure terminology normalization
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
- Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision
- Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering
- ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss
- TPRM: A Topic-based Personalized Ranking Model for Web Search
- A neural document language modeling framework for spoken document retrieval
- BERT Embeddings Can Track Context in Conversational Search
- Modularized Transfomer-based Ranking Framework
- Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints