Pretrained Transformers for Text Ranking: BERT and Beyond

Abstract

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading.

arXiv commentFinal preproduction version of volume in Synthesis Lectures on Human Language Technologies by Morgan & Claypool

References in corpus (80)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• TensorFlow: A system for large-scale machine learning
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Longformer: The Long-Document Transformer
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Scaling Laws for Neural Language Models
• Do Deep Nets Really Need to be Deep?
• Universal Sentence Encoder
• Enhanced LSTM for Natural Language Inference
• The Curious Case of Neural Text Degeneration
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Snorkel: Rapid Training Data Creation with Weak Supervision
• On the Cross-lingual Transferability of Monolingual Representations
• An Empirical Study of Smoothing Techniques for Language Modeling
• REALM: Retrieval-Augmented Language Model Pre-Training
• Generating Long Sequences with Sparse Transformers
• A large annotated corpus for learning natural language inference
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Passage Re-ranking with BERT
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Efficient Natural Language Response Suggestion for Smart Reply
• Document Expansion by Query Prediction
• Multi-Stage Document Ranking with BERT
• Representation learning for very short texts using weighted word embedding aggregation
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Hierarchical Neural Story Generation
• How multilingual is Multilingual BERT?
• TinyBERT: Distilling BERT for Natural Language Understanding
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering
• Modeling Diverse Relevance Patterns in Ad-hoc Retrieval
• TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank
• Language Models as Knowledge Bases?
• Cascade Ranking for Operational E-commerce Search
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• A Dual Embedding Space Model for Document Ranking
• A BERT Baseline for the Natural Questions
• Pre-training via Paraphrasing
• A Study of MatchPyramid Models on Ad-hoc Retrieval
• Zero-shot Text Classification With Generative Language Models
• Overview of the TREC 2019 deep learning track
• End-to-End Retrieval in Continuous Space
• A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
• Complementing Lexical Retrieval with Semantic Residual Embedding
• SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Overview of the TREC 2021 deep learning track
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Cross-Lingual Machine Reading Comprehension
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Structural Language Models of Code
• What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
• An Updated Duet Model for Passage Re-ranking
• Hierarchical Transformers for Multi-Document Summarization
• ABNIRML: Analyzing the Behavior of Neural IR Models
• Diagnosing BERT with Retrieval Heuristics
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• A Memory Efficient Baseline for Open Domain Question Answering
• A Replication Study of Dense Passage Retriever
• Investigating the Successes and Failures of BERT for Passage Re-Ranking
• DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling
• Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
• Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
• Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• Repeatability Corner Cases in Document Ranking: The Impact of Score Ties
• Conformer-Kernel with Query Term Independence for Document Retrieval
• RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemble
• Let's measure run time! Extending the IR replicability infrastructure to include performance aspects
• Content-Based Citation Recommendation
• TU Wien @ TREC Deep Learning '19 -- Simple Contextualization for Re-ranking
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
• NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval
• Learning Passage Impacts for Inverted Indexes
• Long Document Ranking with Query-Directed Sparse Transformer

Cited by in corpus (31)

• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Biomedical Question Answering: A Survey of Approaches and Challenges
• A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Neural ranking models for document retrieval
• Query Embedding Pruning for Dense Retrieval
• ABNIRML: Analyzing the Behavior of Neural IR Models
• Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
• Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation
• B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Intra-Document Cascading: Learning to Select Passages for Neural Document Ranking
• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
• Perception-Aware Bias Detection for Query Suggestions
• News Article Retrieval in Context for Event-centric Narrative Creation
• On Single and Multiple Representations in Dense Passage Retrieval
• Learning Dense Representations of Phrases at Scale
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• IntenT5: Search Result Diversification using Causal Language Models
• Cross-Lingual Training with Dense Retrieval for Document Retrieval
• Pre-trained Language Model based Ranking in Baidu Search
• Dealing with Typos for BERT-based Passage Retrieval and Ranking
• TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
• Text-to-Text Multi-view Learning for Passage Re-ranking
• MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
• JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments
• Contrastive Document Representation Learning with Graph Attention Networks