Is Retriever Merely an Approximator of Reader?

Abstract

The state of the art in open-domain question answering (QA) relies on an efficient retriever that drastically reduces the search space for the expensive reader. A rather overlooked question in the community is the relationship between the retriever and the reader, and in particular, if the whole purpose of the retriever is just a fast approximation for the reader. Our empirical evidence indicates that the answer is no, and that the reader and the retriever are complementary to each other even in terms of accuracy only. We make a careful conjecture that the architectural constraint of the retriever, which has been originally intended for enabling approximate search, seems to also make the model more robust in large-scale search. We then propose to distill the reader into the retriever so that the retriever absorbs the strength of the reader while keeping its own benefit. Experimental results show that our method can enhance the document recall rate as well as the end-to-end QA accuracy of off-the-shelf retrievers in open-domain QA tasks.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• Machine Comprehension Using Match-LSTM and Answer Pointer
• Passage Re-ranking with BERT
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
• Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index

Cited by in corpus (6)

• Information Retrieval: Recent Advances and Beyond
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• Distilling Knowledge from Reader to Retriever for Question Answering
• Adversarial Retriever-Ranker for dense text retrieval
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering