Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

Abstract

We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

Cited by in corpus (23)

• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs
• Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
• Focused Hierarchical RNNs for Conditional Sequence Processing
• Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
• Contextualized Representations Using Textual Encyclopedic Knowledge
• Paraphrase Generation with Deep Reinforcement Learning
• Deep Reinforced Query Reformulation for Information Retrieval
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Answering Science Exam Questions Using Query Rewriting with Background Knowledge
• Joint Training of Candidate Extraction and Answer Selection for Reading Comprehension
• Dimsum @LaySumm 20: BART-based Approach for Scientific Document Summarization
• Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering
• Identifying Well-formed Natural Language Questions
• What do Deep Networks Like to Read?
• Adviser Networks: Learning What Question to Ask for Human-In-The-Loop Viewpoint Estimation
• Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
• Augmented Memory Networks for Streaming-Based Active One-Shot Learning
• Syntax-Enhanced Pre-trained Model
• Stronger Transformers for Neural Multi-Hop Question Generation
• Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System
• Are You Sure You Want To Do That? Classification with Verification
• Optimizing AD Pruning of Sponsored Search with Reinforcement Learning