Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation

Abstract

Open-domain question answering answers a question based on evidence retrieved from a large corpus. State-of-the-art neural approaches require intermediate evidence annotations for training. However, such intermediate annotations are expensive, and methods that rely on them cannot transfer to the more common setting, where only question-answer pairs are available. This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training, thereby generating no additional annotation cost. We introduce a novel approach (DistDR) that iteratively improves over a weak retriever by alternately finding evidence from the up-to-date model and encouraging the model to learn the most likely evidence. Without using any evidence labels, DistDR is on par with fully-supervised state-of-the-art methods on both multi-hop and single-hop QA benchmarks. Our analysis confirms that DistDR finds more accurate evidence over iterations, which leads to model improvements.

arXiv commentEMNLP 2021

References in corpus (8)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
• Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering
• Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval