Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Abstract

Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a "semantic drift" problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this paper, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. Second, since the traditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model's ability to mimic human annotators in generating QA training data. Experiments show that our method achieves the new state-of-the-art performance w.r.t. traditional metrics, and also performs best on our QA-based evaluation metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: generate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.

arXiv comment15 pages (EMNLP 2019)

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Bidirectional Attention Flow for Machine Comprehension
• A Deep Reinforced Model for Abstractive Summarization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Dual Learning for Machine Translation
• Speaker-Follower Models for Vision-and-Language Navigation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Question Answering and Question Generation as Dual Tasks
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Learning to Generate Questions by Learning What not to Generate
• A Unified Query-based Generative Model for Question Generation and Question Answering
• Synthetic QA Corpora Generation with Roundtrip Consistency
• Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
• Improving Neural Question Generation using Answer Separation
• Evaluating Rewards for Question Generation Models

Cited by in corpus (6)

• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• Enhancing Question Generation with Commonsense Knowledge
• Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space