Neural Passage Retrieval with Improved Negative Contrast

Abstract

In this paper we explore the effects of negative sampling in dual encoder models used to retrieve passages for automatic question answering. We explore four negative sampling strategies that complement the straightforward random sampling of negatives, typically used to train dual encoder models. Out of the four strategies, three are based on retrieval and one on heuristics. Our retrieval-based strategies are based on the semantic similarity and the lexical overlap between questions and passages. We train the dual encoder models in two stages: pre-training with synthetic data and fine tuning with domain-specific data. We apply negative sampling to both stages. The approach is evaluated in two passage retrieval tasks. Even though it is not evident that there is one single sampling strategy that works best in all the tasks, it is clear that our strategies contribute to improving the contrast between the response and all the other passages. Furthermore, mixing the negatives from different strategies achieve performance on par with the best performing strategy in all tasks. Our results establish a new state-of-the-art level of performance on two of the open-domain question answering datasets that we evaluated.

References in corpus (9)

• A Deep Relevance Matching Model for Ad-hoc Retrieval
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Passage Re-ranking with BERT
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Efficient Natural Language Response Suggestion for Smart Reply
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Sparse, Dense, and Attentional Representations for Text Retrieval
• End-to-End Retrieval in Continuous Space
• Overview of the TREC 2021 deep learning track

Cited by in corpus (5)

• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Dense Hierarchical Retrieval for Open-Domain Question Answering
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking