Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval

Abstract

Term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. This paper proposes a Deep Contextualized Term Weighting framework that learns to map BERT's contextualized text representations to context-aware term weights for sentences and passages. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for passage retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms. This is novel because most deep neural network based ranking models have higher computational costs, and thus are restricted to later-stage rankers. Experiments on four datasets demonstrate that DeepCT's deep contextualized text understanding greatly improves the accuracy of first-stage retrieval algorithms.

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• Passage Re-ranking with BERT
• Document Expansion by Query Prediction
• An Updated Duet Model for Passage Re-ranking

Cited by in corpus (37)

• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
• Learning-to-Rank with BERT in TF-Ranking
• SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
• RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• ABNIRML: Analyzing the Behavior of Neural IR Models
• Adaptive Re-Ranking with a Corpus Graph
• To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
• Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation
• Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach
• SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Optimizing Dense Retrieval Model Training with Hard Negatives
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• Condenser: a Pre-training Architecture for Dense Retrieval
• Learning Passage Impacts for Inverted Indexes
• An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
• VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words
• Scientific Claim Verification with VERT5ERINI
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• GLOW : Global Weighted Self-Attention Network for Web Search
• Searching Personal Collections
• Modularized Transfomer-based Ranking Framework
• YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker
• Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder