Sparse, Dense, and Attentional Representations for Text Retrieval

Abstract

Dual encoders perform retrieval by encoding documents and queries into dense lowdimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.

arXiv commentTo appear in TACL 2020. The arXiv version is a pre-MIT Press publication version

References in corpus (12)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• Passage Re-ranking with BERT
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Document Expansion by Query Prediction
• Multi-Stage Document Ranking with BERT
• Simple Applications of BERT for Ad Hoc Document Retrieval
• Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
• No Training Required: Exploring Random Encoders for Sentence Classification
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Overview of the TREC 2021 deep learning track
• Quantization based Fast Inner Product Search
• Can Unconditional Language Models Recover Arbitrary Sentences?

Cited by in corpus (33)

• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
• SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• Distilling Knowledge from Reader to Retriever for Question Answering
• Query Embedding Pruning for Dense Retrieval
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
• Generation-Augmented Retrieval for Open-domain Question Answering
• A Memory Efficient Baseline for Open Domain Question Answering
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Neural Passage Retrieval with Improved Negative Contrast
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• Optimizing Dense Retrieval Model Training with Hard Negatives
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
• Condenser: a Pre-training Architecture for Dense Retrieval
• Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation
• Ultra-Fast, Low-Storage, Highly Effective Coarse-grained Selection in Retrieval-based Chatbot by Using Deep Semantic Hashing
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• Matching-oriented Product Quantization For Ad-hoc Retrieval
• Interpreting Dense Retrieval as Mixture of Topics
• Dealing with Typos for BERT-based Passage Retrieval and Ranking
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Unsupervised Document Expansion for Information Retrieval with Stochastic Text Generation
• An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
• Understanding Hard Negatives in Noise Contrastive Estimation
• Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks
• Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
• Contrastive Document Representation Learning with Graph Attention Networks