CEDR: Contextualized Embeddings for Document Ranking

Abstract

Although considerable attention has been given to neural ranking architectures recently, far less attention has been paid to the term representations that are used as input to these models. In this work, we investigate how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking. Through experiments on TREC benchmarks, we find that several existing neural ranking architectures can benefit from the additional context provided by contextualized language models. Furthermore, we propose a joint approach that incorporates BERT's classification vector into existing neural models and show that it outperforms state-of-the-art ad-hoc ranking baselines. We call this joint approach CEDR (Contextualized Embeddings for Document Ranking). We also address practical challenges in using these models for ranking, including the maximum input length imposed by BERT and runtime performance impacts of contextualized language models.

arXiv commentAppeared in SIGIR 2019, 4 pages

References in corpus (2)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling

Cited by in corpus (40)

• Information Retrieval: Recent Advances and Beyond
• Query Resolution for Conversational Search with Limited Supervision
• Declarative Experimentation in Information Retrieval using PyTerrier
• Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
• Expansion via Prediction of Importance with Contextualization
• GERE: Generative Evidence Retrieval for Fact Verification
• Bringing order into the realm of Transformer-based language models for artificial intelligence and law
• Table Search Using a Deep Contextualized Language Model
• PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval
• Neural ranking models for document retrieval
• C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
• ABNIRML: Analyzing the Behavior of Neural IR Models
• Adaptive Re-Ranking with a Corpus Graph
• Diagnosing BERT with Retrieval Heuristics
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• Selective Weak Supervision for Neural Information Retrieval
• Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
• Efficient Neural Ranking using Forward Indexes
• A Reproducibility Study of PLAID
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• Graph-based Hierarchical Relevance Matching Signals for Ad-hoc Retrieval
• Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning
• Improving Bi-encoder Document Ranking Models with Two Rankers and Multi-teacher Distillation
• SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval
• Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?
• Composite Re-Ranking for Efficient Document Search with BERT
• On Approximate Nearest Neighbour Selection for Multi-Stage Dense Retrieval
• "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"
• Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
• Training Curricula for Open Domain Answer Re-Ranking
• A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models
• SQRQuerier: A Visual Querying Framework for Cross-national Survey Data Recycling
• OFAR: A Multimodal Evidence Retrieval Framework for Illegal Live-streaming Identification
• QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking
• CWRCzech: 100M Query-Document Czech Click Dataset and Its Application to Web Relevance Ranking
• Query-driven Segment Selection for Ranking Long Documents
• Shallow Cross-Encoders for Low-Latency Retrieval
• Ranking Significant Discrepancies in Clinical Reports
• COPER: a Query-adaptable Semantics-based Search Engine for Persian COVID-19 Articles
• On the Reproducibility of Learned Sparse Retrieval Adaptations for Long Documents