Paraphrase Generation with Deep Reinforcement Learning

Abstract

Automatic generation of paraphrases from a given sentence is an important yet challenging task in natural language processing (NLP), and plays a key role in a number of applications such as question answering, search, and dialogue. In this paper, we present a deep reinforcement learning approach to paraphrase generation. Specifically, we propose a new framework for the task, which consists of a \textit{generator} and an \textit{evaluator}, both of which are learned from data. The generator, built as a sequence-to-sequence learning model, can produce paraphrases given a sentence. The evaluator, constructed as a deep matching model, can judge whether two sentences are paraphrases of each other. The generator is first trained by deep learning and then further fine-tuned by reinforcement learning in which the reward is given by the evaluator. For the learning of the evaluator, we propose two methods based on supervised learning and inverse reinforcement learning respectively, depending on the type of available training data. Empirical study shows that the learned evaluator can guide the generator to produce more accurate paraphrases. Experimental results demonstrate the proposed models (the generators) outperform the state-of-the-art methods in paraphrase generation in both automatic evaluation and human evaluation.

EMNLP 2018

References in corpus (24)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Neural Conversational Model
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Generative Adversarial Imitation Learning
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• An Actor-Critic Algorithm for Sequence Prediction
• Neural Responding Machine for Short-Text Conversation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Adversarial Learning for Neural Dialogue Generation
• Long Text Generation via Adversarial Training with Leaked Information
• Modeling Coverage for Neural Machine Translation
• A Deep Generative Framework for Paraphrase Generation
• Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
• Neural Summarization by Extracting Sentences and Words
• Learning Natural Language Inference with LSTM
• Joint Copying and Restricted Generation for Paraphrase
• Model-Free Imitation Learning with Policy Optimization
• Learning to Paraphrase for Question Answering
• A Continuously Growing Dataset of Sentential Paraphrases
• Cross-domain Semantic Parsing via Paraphrasing

Cited by in corpus (8)

• Unsupervised Text Generation by Learning from Search
• Deep Reinforced Query Reformulation for Information Retrieval
• Towards Diverse Paraphrase Generation Using Multi-Class Wasserstein GAN
• Topic-Preserving Synthetic News Generation: An Adversarial Deep Reinforcement Learning Approach
• Discourse-Aware Neural Rewards for Coherent Text Generation
• Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System
• On Tree-Based Neural Sentence Modeling
• AGenT Zero: Zero-shot Automatic Multiple-Choice Question Generation for Skill Assessments