Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning

Abstract

Query reformulation aims to alter noisy or ambiguous text sequences into coherent ones closer to natural language questions. This is to prevent errors from propagating in a client-facing pipeline and promote better communication with users. Besides, it is crucial to maintain performance in downstream environments like question answering when rephrased queries are given as input. We show that under the previous framework (AQA), attempts to alter RL algorithms do not bring significant benefits to either reward acquisition or sequence fluency. Instead, we leverage a query-reformulating text-to-text transformer (QRT5) and apply policy-based RL algorithms to further nudge this reformulator and obtain better answers downstream by generating reward-acquiring query trajectories. QRT5 shows better sample efficiency in RL to achieve the same level of QA performance as the previous approach. It can generate reformulations with more readability based on query well-formedness evaluations and can generalize to out-of-sample data. Our framework is demonstrated to be flexible, allowing reward signals to be sourced from different downstream environments such as intent classification.

arXiv commentWorkshop on the 9th Dialog System Technology Challenge (DSTC-9), AAAI 2021

References in corpus (17)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Playing Atari with Deep Reinforcement Learning
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Bidirectional Attention Flow for Machine Comprehension
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• A Deep Generative Framework for Paraphrase Generation
• Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
• Learning to summarize from human feedback
• Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models
• Pretrained Transformers Improve Out-of-Distribution Robustness
• Zero-Shot Paraphrase Generation with Multilingual Language Models
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Analyzing Language Learned by an Active Question Answering Agent