Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text

Abstract

Automatic question generation (QG) is a useful yet challenging task in NLP. Recent neural network-based approaches represent the state-of-the-art in this task. In this work, we attempt to strengthen them significantly by adopting a holistic and novel generator-evaluator framework that directly optimizes objectives that reward semantics and structure. The {\it generator} is a sequence-to-sequence model that incorporates the {\it structure} and {\it semantics} of the question being generated. The generator predicts an answer in the passage that the question can pivot on. Employing the copy and coverage mechanisms, it also acknowledges other contextually important (and possibly rare) keywords in the passage that the question needs to conform to, while not redundantly repeating words. The {\it evaluator} model evaluates and assigns a reward to each predicted question based on its conformity to the {\it structure} of ground-truth questions. We propose two novel QG-specific reward functions for text conformity and answer conformity of the generated question. The evaluator also employs structure-sensitive rewards based on evaluation measures such as BLEU, GLEU, and ROUGE-L, which are suitable for QG. In contrast, most of the previous works only optimize the cross-entropy loss, which can induce inconsistencies between training (objective) and testing (evaluation) measures. Our evaluation shows that our approach significantly outperforms state-of-the-art systems on the widely-used SQuAD benchmark as per both automatic and human evaluation.

arXiv comment10 pages, The SIGNLL Conference on Computational Natural Language Learning (CoNLL 2019)

References in corpus (6)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• A Deep Reinforced Model for Abstractive Summarization
• Sequence Level Training with Recurrent Neural Networks
• Deep Reinforcement Learning for Dialogue Generation
• Learning to Ask: Neural Question Generation for Reading Comprehension
• Automating Reading Comprehension by Generating Question and Answer Pairs

Cited by in corpus (5)

• Natural Question Generation with Reinforcement Learning Based Graph-to-Sequence Model
• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
• Text as Environment: A Deep Reinforcement Learning Text Readability Assessment Model
• Hindi Question Generation Using Dependency Structures
• Two Birds, One Stone: A Simple, Unified Model for Text Generation from Structured and Unstructured Data