Unsupervised Question Answering by Cloze Translation

Abstract

Obtaining training data for Question Answering (QA) is time-consuming and resource-intensive, and existing QA datasets are only available for limited domains and languages. In this work, we explore to what extent high quality training data is actually required for Extractive QA, and investigate the possibility of unsupervised Extractive QA. We approach this problem by first learning to generate context, question and answer triples in an unsupervised manner, which we then use to synthesize Extractive QA training data automatically. To generate such triples, we first sample random context paragraphs from a large corpus of documents and then random noun phrases or named entity mentions from these paragraphs as answers. Next we convert answers in context to "fill-in-the-blank" cloze questions and finally translate them into natural questions. We propose and compare various unsupervised ways to perform cloze-to-natural question translation, including training an unsupervised NMT model using non-aligned corpora of natural questions and cloze questions as well as a rule-based approach. We find that modern QA models can learn to answer human questions surprisingly well using only synthetic training data. We demonstrate that, without using the SQuAD training data at all, our approach achieves 56.4 F1 on SQuAD v1 (64.5 F1 when the answer is a Named entity mention), outperforming early supervised models.

arXiv commentTo appear in ACL 2019

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Cross-lingual Language Model Pretraining
• Enriching Word Vectors with Subword Information
• Unsupervised Statistical Machine Translation
• Phrase-Based & Neural Unsupervised Machine Translation
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Learning to Ask: Neural Question Generation for Reading Comprehension
• Dynamic Integration of Background Knowledge in Neural NLU Systems
• Multiple-Attribute Text Style Transfer
• Efficient and Robust Question Answering from Minimal Context over Documents
• A Minimal Span-Based Neural Constituency Parser
• Identifying Well-formed Natural Language Questions
• Evaluating Rewards for Question Generation Models

Cited by in corpus (32)

• CTRL: A Conditional Transformer Language Model for Controllable Generation
• How Context Affects Language Models' Factual Predictions
• Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
• Unsupervised Pre-training for Biomedical Question Answering
• Embedding-based Zero-shot Retrieval through Query Generation
• Multi-Stage Pre-training for Low-Resource Domain Adaptation
• Studying Strategically: Learning to Mask for Closed-book QA
• Answering Questions on COVID-19 in Real-Time
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering
• MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
• QAEA-DR: A Unified Text Augmentation Framework for Dense Retrieval
• Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
• Generating Query Focused Summaries from Query-Free Resources
• Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA
• Summary-Oriented Question Generation for Informational Queries
• Science Checker: Extractive-Boolean Question Answering For Scientific Fact Checking
• Let Me Know What to Ask: Interrogative-Word-Aware Question Generation
• Teaching Machine Comprehension with Compositional Explanations
• Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach
• Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals
• Unsupervised Open-Domain Question Answering
• When in Doubt, Ask: Generating Answerable and Unanswerable Questions, Unsupervised
• Open Aspect Target Sentiment Classification with Natural Language Prompts
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• ReadOnce Transformers: Reusable Representations of Text for Transformers
• Relation-Guided Pre-Training for Open-Domain Question Answering
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models
• Improving Unsupervised Question Answering via Summarization-Informed Question Generation
• Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systems