Relation-Guided Pre-Training for Open-Domain Question Answering
arXiv:2109.10346
Abstract
Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pretraining with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations
References in corpus (6)
- REALM: Retrieval-Augmented Language Model Pre-Training
- Unsupervised Question Answering by Cloze Translation
- Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
- Differentiable Reasoning over a Virtual Knowledge Base
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
- Reasoning Over Virtual Knowledge Bases With Open Predicate Relations