Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Abstract

Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval. Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval. In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing. This paper presents Approximate nearest neighbor Negative Contrastive Estimation (ANCE), a training mechanism that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus, which is parallelly updated with the learning process to select more realistic negative training instances. This fundamentally resolves the discrepancy between the data distribution used in the training and testing of DR. In our experiments, ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and sparse retrieval baselines. It nearly matches the accuracy of sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned representation space and provides almost 100x speed-up.

References in corpus (20)

• A Simple Framework for Contrastive Learning of Visual Representations
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Improved Baselines with Momentum Contrastive Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• REALM: Retrieval-Augmented Language Model Pre-Training
• Deeper Text Understanding for IR with Contextual Neural Language Modeling
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• Multi-Stage Document Ranking with BERT
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Understanding the Behaviors of BERT in Ranking
• Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval
• Pre-training via Paraphrasing
• Sparse, Dense, and Attentional Representations for Text Retrieval
• Complementing Lexical Retrieval with Semantic Residual Embedding
• Overview of the TREC 2021 deep learning track
• Generalization through Memorization: Nearest Neighbor Language Models
• Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
• BERT-QE: Contextualized Query Expansion for Document Re-ranking

Cited by in corpus (73)

• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Sparse, Dense, and Attentional Representations for Text Retrieval
• A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
• The Information Retrieval Experiment Platform
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Fast Passage Re-ranking with Contextualized Exact Term Matching and Efficient Passage Expansion
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Query Embedding Pruning for Dense Retrieval
• Neural ranking models for document retrieval
• Adversarial Retriever-Ranker for dense text retrieval
• Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models
• Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
• A Replication Study of Dense Passage Retriever
• Neural Passage Retrieval with Improved Negative Contrast
• GNN-LM: Language Modeling based on Global Contexts via GNN
• mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Optimizing Dense Retrieval Model Training with Hard Negatives
• COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List
• Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking
• CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
• Learning Passage Impacts for Inverted Indexes
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Ultra-Fast, Low-Storage, Highly Effective Coarse-grained Selection in Retrieval-based Chatbot by Using Deep Semantic Hashing
• Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval
• Self-supervised Document Clustering Based on BERT with Data Augment
• Investigating the Effects of Sparse Attention on Cross-Encoders
• On Single and Multiple Representations in Dense Passage Retrieval
• Multistage BiCross encoder for multilingual access to COVID-19 health information
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• OpenMatch: An Open Source Library for Neu-IR Research
• IntenT5: Search Result Diversification using Causal Language Models
• Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
• Encoder Adaptation of Dense Passage Retrieval for Open-Domain Question Answering
• UserBERT: Contrastive User Model Pre-training
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• DoSSIER@COLIEE 2021: Leveraging dense retrieval and summarization-based re-ranking for case law retrieval
• Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?
• Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboard
• Contextualized Query Embeddings for Conversational Search
• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation
• GLOW : Global Weighted Self-Attention Network for Web Search
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• Dense Hierarchical Retrieval for Open-Domain Question Answering
• Shallow Cross-Encoders for Low-Latency Retrieval
• Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need
• Conformer-Kernel with Query Term Independence at TREC 2020 Deep Learning Track
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
• Unsupervised Document Expansion for Information Retrieval with Stochastic Text Generation
• An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking
• Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback
• Dealing with Typos for BERT-based Passage Retrieval and Ranking
• Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
• SAUCE: Truncated Sparse Document Signature Bit-Vectors for Fast Web-Scale Corpus Expansion
• Contrastive Document Representation Learning with Graph Attention Networks
• Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
• Towards Universal Dense Retrieval for Open-domain Question Answering
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• Cross-Batch Negative Sampling for Training Two-Tower Recommenders
• More Robust Dense Retrieval with Contrastive Dual Learning
• Scalable Personalised Item Ranking through Parametric Density Estimation
• YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker
• Extreme Multi-label Learning for Semantic Matching in Product Search