Distilling Knowledge from Reader to Retriever for Question Answering

Abstract

The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Decoupled Weight Decay Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• REALM: Retrieval-Augmented Language Model Pre-Training
• End-to-End Open-Domain Question Answering with BERTserini
• Passage Re-ranking with BERT
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Pre-training via Paraphrasing
• Sparse, Dense, and Attentional Representations for Text Retrieval
• End-to-End Retrieval in Continuous Space
• Is Retriever Merely an Approximator of Reader?

Cited by in corpus (12)

• Information Retrieval: Recent Advances and Beyond
• Recursively Summarizing Books with Human Feedback
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents
• Adversarial Retriever-Ranker for dense text retrieval
• A Memory Efficient Baseline for Open Domain Question Answering
• Attention-guided Generative Models for Extractive Question Answering
• Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval
• Automatic Claim Review for Climate Science via Explanation Generation
• Recent Advances in Automated Question Answering In Biomedical Domain
• Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints
• Unsupervised Open-Domain Question Answering