Multi-Stage Document Ranking with BERT

Abstract

The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing. This work explores one such popular model, BERT, in the context of document ranking. We propose two variants, called monoBERT and duoBERT, that formulate the ranking problem as pointwise and pairwise classification, respectively. These two models are arranged in a multi-stage ranking architecture to form an end-to-end search system. One major advantage of this design is the ability to trade off quality against latency by controlling the admission of candidates into each pipeline stage, and by doing so, we are able to find operating points that offer a good balance between these two competing metrics. On two large-scale datasets, MS MARCO and TREC CAR, experiments show that our model produces results that are either at or comparable to the state of the art. Ablation studies show the contributions of each component and characterize the latency/quality tradeoff space.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• End-to-End Open-Domain Question Answering with BERTserini
• Passage Re-ranking with BERT
• Efficient Natural Language Response Suggestion for Smart Reply
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Cascade Ranking for Operational E-commerce Search
• SciBERT: A Pretrained Language Model for Scientific Text

Cited by in corpus (56)

• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Sparse, Dense, and Attentional Representations for Text Retrieval
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• The Information Retrieval Experiment Platform
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• StruBERT: Structure-aware BERT for Table Search and Matching
• Adaptive Re-Ranking with a Corpus Graph
• Learning a Product Relevance Model from Click-Through Data in E-Commerce
• Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
• Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims
• Entity-aware Transformers for Entity Search
• Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation
• Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach
• SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
• IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles
• Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• Semantic Labeling Using a Deep Contextualized Language Model
• Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
• Learning Passage Impacts for Inverted Indexes
• Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset
• Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders
• Investigating the Effects of Sparse Attention on Cross-Encoders
• BERT-QE: Contextualized Query Expansion for Document Re-ranking
• Improving Conversational Passage Re-ranking with View Ensemble
• An Optimal Algorithm for Finding Champions in Tournament Graphs
• Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation
• Ranking Creative Language Characteristics in Small Data Scenarios
• Long Document Ranking with Query-Directed Sparse Transformer
• Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning
• Knowledge-driven Answer Generation for Conversational Search
• Learning Representations for Axis-Aligned Decision Forests through Input Perturbation
• Pre-trained Language Model based Ranking in Baidu Search
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
• Prediction, Selection, and Generation: Exploration of Knowledge-Driven Conversation System
• Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning
• Conformer-Kernel with Query Term Independence at TREC 2020 Deep Learning Track
• An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
• GLOW : Global Weighted Self-Attention Network for Web Search
• Searching Personal Collections
• BERT for Target Apps Selection: Analyzing the Diversity and Performance of BERT in Unified Mobile Search
• Text-to-Text Multi-view Learning for Passage Re-ranking
• Quality and Cost Trade-offs in Passage Re-ranking Task
• Evolutionary Optimization of High-Coverage Budgeted Classifiers