RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

Abstract

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

For corresponding code and data, see https://github.com/jingtaozhan/RepBERT-Index

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• REALM: Retrieval-Augmented Language Model Pre-Training
• Passage Re-ranking with BERT
• Document Expansion by Query Prediction
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks

Cited by in corpus (22)

• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• A Replication Study of Dense Passage Retriever
• CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• To Interpolate or not to Interpolate: PRF, Dense and Sparse Retrievers
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• Optimizing Dense Retrieval Model Training with Hard Negatives
• CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking
• PARK: Personalized academic retrieval with knowledge-graphs
• An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
• Asyncval: A Toolkit for Asynchronously Validating Dense Retriever Checkpoints during Training
• Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• Interpreting Dense Retrieval as Mixture of Topics
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Dealing with Typos for BERT-based Passage Retrieval and Ranking
• Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval