Semantic Models for the First-stage Retrieval: A Comprehensive Review

Abstract

Multi-stage ranking pipelines have been a practical solution in modern search systems, where the first-stage retrieval is to return a subset of candidate documents, and latter stages attempt to re-rank those candidates. Unlike re-ranking stages going through quick technique shifts during past decades, the first-stage retrieval has long been dominated by classical term-based models. Unfortunately, these models suffer from the vocabulary mismatch problem, which may block re-ranking stages from relevant documents at the very beginning. Therefore, it has been a long-term desire to build semantic models for the first-stage retrieval that can achieve high recall efficiently. Recently, we have witnessed an explosive growth of research interests on the first-stage semantic retrieval models. We believe it is the right time to survey current status, learn from existing methods, and gain some insights for future development. In this paper, we describe the current landscape of the first-stage retrieval models under a unified framework to clarify the connection between classical term-based retrieval methods, early semantic retrieval methods and neural semantic retrieval methods. Moreover, we identify some open challenges and envision some future directions, with the hope of inspiring more researches on these important yet less investigated topics.

arXiv commentAccepted by TOIS

References in corpus (72)

• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Distributed Representations of Sentences and Documents
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Additive Margin Softmax for Face Verification
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
• NormFace: L2 Hypersphere Embedding for Face Verification
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• REALM: Retrieval-Augmented Language Model Pre-Training
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Passage Re-ranking with BERT
• Embedding-based Retrieval in Facebook Search
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Efficient Natural Language Response Suggestion for Smart Reply
• Document Expansion by Query Prediction
• Multi-Stage Document Ranking with BERT
• Attention-over-Attention Neural Networks for Reading Comprehension
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Quasar: Datasets for Question Answering by Search and Reading
• Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Overview of the TREC 2020 deep learning track
• Cascade Ranking for Operational E-commerce Search
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• A Dual Embedding Space Model for Document Ranking
• Neural Vector Spaces for Unsupervised Information Retrieval
• Adversarial Sampling and Training for Semi-Supervised Information Retrieval
• Using Word Embeddings for Automatic Query Expansion
• Neural Models for Information Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• Expansion via Prediction of Importance with Contextualization
• Overview of the TREC 2019 deep learning track
• End-to-End Retrieval in Continuous Space
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Efficient Document Re-Ranking for Transformers by Precomputing Term Representations
• RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
• SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Query Expansion with Locally-Trained Word Embeddings
• Multi-Interest Network with Dynamic Routing for Recommendation at Tmall
• Generation-Augmented Retrieval for Open-domain Question Answering
• Embedding-based Zero-shot Retrieval through Query Generation
• Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search
• Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
• Relevance-based Word Embedding
• Adversarial Contrastive Estimation
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index
• A Discrete Hard EM Approach for Weakly Supervised Question Answering
• B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
• Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach
• Return of the Lernaean Hydra: Experimental Evaluation of Data Series Approximate Similarity Search
• Let's measure run time! Extending the IR replicability infrastructure to include performance aspects
• Optimizing Dense Retrieval Model Training with Hard Negatives
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• Representing Documents and Queries as Sets of Word Embedded Vectors for Information Retrieval
• DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding
• Multi-Hop Paragraph Retrieval for Open-Domain Question Answering
• Learning Passage Impacts for Inverted Indexes
• Extreme Classification via Adversarial Softmax Approximation
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Match${}^2$: A Matching over Matching Model for Similar Question Identification

Cited by in corpus (18)

• CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
• ConsRec: Learning Consensus Behind Interactions for Group Recommendation
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• Neural Retrievers are Biased Towards LLM-Generated Content
• Retrieving false claims on Twitter during the Russia-Ukraine conflict
• MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels
• A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking
• Towards Completeness-Oriented Tool Retrieval for Large Language Models
• Improving Neural Topic Models with Wasserstein Knowledge Distillation
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Fuzzy Norm-Explicit Product Quantization for Recommender Systems
• Improving information retrieval through correspondence analysis instead of latent semantic analysis
• Multi-Modality Transformer for E-Commerce: Inferring User Purchase Intention to Bridge the Query-Product Gap
• PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Optimizing Compound Retrieval Systems
• Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models
• JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments