Deep Reinforcement Learning for Dialogue Generation

Abstract

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

References in corpus (16)

• Sequence to Sequence Learning with Neural Networks
• Playing Atari with Deep Reinforcement Learning
• A Neural Conversational Model
• Sequence Level Training with Recurrent Neural Networks
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
• Neural Responding Machine for Short-Text Conversation
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Persona-Based Neural Conversation Model
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Reinforcement Learning Neural Turing Machines - Revised
• Continuously Learning Neural Dialogue Management
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Attention with Intention for a Neural Network Conversation Model
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• LSTM based Conversation Models

Cited by in corpus (236)

• Recent Trends in Deep Learning Based Natural Language Processing
• The Curious Case of Neural Text Degeneration
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• Deep Reinforcement Learning: An Overview
• Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
• Adversarial Learning for Neural Dialogue Generation
• A Deep Reinforcement Learning Chatbot
• Natural Language Processing Advancements By Deep Learning: A Survey
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Clinically Accurate Chest X-Ray Report Generation
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Adversarial Feature Matching for Text Generation
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Generating Persona Consistent Dialogues by Exploiting Natural Language Inference
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Generative Deep Neural Networks for Dialogue: A Short Review
• Adversarial Evaluation of Dialogue Models
• A Reinforcement Learning-based Economic Model Predictive Control Framework for Autonomous Operation of Chemical Reactors
• End-to-End Task-Completion Neural Dialogue Systems
• The Eighth Dialog System Technology Challenge
• End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
• Zero-Resource Knowledge-Grounded Dialogue Generation
• QuAC : Question Answering in Context
• Plan-And-Write: Towards Better Automatic Storytelling
• CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• Recent advances in conversational NLP : Towards the standardization of Chatbot building
• Deep Learning Based Chatbot Models
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models
• A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• Beating Atari with Natural Language Guided Reinforcement Learning
• Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
• Neural Text Generation: A Practical Guide
• LSTM vs. GRU vs. Bidirectional RNN for script generation
• Teaching Machines to Describe Images via Natural Language Feedback
• Towards Explainable and Controllable Open Domain Dialogue Generation with Dialogue Acts
• Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
• PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
• Towards a Robust Deep Neural Network in Texts: A Survey
• Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation
• Latent Intention Dialogue Models
• Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
• Backplay: "Man muss immer umkehren"
• CASS: Towards Building a Social-Support Chatbot for Online Health Community
• A Hybrid Retrieval-Generation Neural Conversation Model
• Personalizing a Dialogue System with Transfer Reinforcement Learning
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• An Adversarial Approach to High-Quality, Sentiment-Controlled Neural Dialogue Generation
• Gunrock 2.0: A User Adaptive Social Conversational System
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• Learning Robust Dialog Policies in Noisy Environments
• Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
• Exploration on Generating Traditional Chinese Medicine Prescription from Symptoms with an End-to-End method
• A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
• Data Distillation for Controlling Specificity in Dialogue Generation
• Reinforcement Learning Based Emotional Editing Constraint Conversation Generation
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
• MojiTalk: Generating Emotional Responses at Scale
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Learning Conversational Systems that Interleave Task and Non-Task Content
• Neural Personalized Response Generation as Domain Adaptation
• An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control
• RTFM: Generalising to Novel Environment Dynamics via Reading
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Shaping the Narrative Arc: An Information-Theoretic Approach to Collaborative Dialogue
• RubyStar: A Non-Task-Oriented Mixture Model Dialog System
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Generative Encoder-Decoder Models for Task-Oriented Spoken Dialog Systems with Chatting Capability
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management
• End-to-End Trainable Non-Collaborative Dialog System
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• EmpDG: Multiresolution Interactive Empathetic Dialogue Generation
• Deep Reinforcement Learning for Chinese Zero pronoun Resolution
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Distributed Representation for Traditional Chinese Medicine Herb via Deep Learning Models
• Putting the Horse Before the Cart:A Generator-Evaluator Framework for Question Generation from Text
• Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce
• Sentence Simplification with Deep Reinforcement Learning
• Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems
• Decoupling Strategy and Generation in Negotiation Dialogues
• Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning
• Deep Active Learning for Dialogue Generation
• Towards Standard Criteria for human evaluation of Chatbots: A Survey
• Comprehensible Context-driven Text Game Playing
• Reference-Aware Language Models
• Defending Against Backdoor Attacks in Natural Language Generation
• Cortex: A Compiler for Recursive Deep Learning Models
• Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
• Chat More If You Like: Dynamic Cue Words Planning to Flow Longer Conversations
• Introducing Aspects of Creativity in Automatic Poetry Generation
• Natural Language Generation with Neural Variational Models
• Navigating with Graph Representations for Fast and Scalable Decoding of Neural Language Models
• Graph-Stega: Semantic Controllable Steganographic Text Generation Guided by Knowledge Graph
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning
• I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
• Topic-Preserving Synthetic News Generation: An Adversarial Deep Reinforcement Learning Approach
• Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization
• Science Driven Innovations Powering Mobile Product: Cloud AI vs. Device AI Solutions on Smart Device
• Text as Environment: A Deep Reinforcement Learning Text Readability Assessment Model
• Generating Multiple Diverse Responses for Short-Text Conversation
• An Ensemble Model with Ranking for Social Dialogue
• Probing Neural Dialog Models for Conversational Understanding
• A Benchmark Dataset for Learning to Intervene in Online Hate Speech
• MoEL: Mixture of Empathetic Listeners
• Thinking Globally, Acting Locally: Distantly Supervised Global-to-Local Knowledge Selection for Background Based Conversation
• Improving Conditional Sequence Generative Adversarial Networks by Stepwise Evaluation
• Discourse-Aware Neural Rewards for Coherent Text Generation
• Deep Reinforcement Learning for Conversational AI
• Self-Supervised Dialogue Learning
• An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation
• Retrieval-Enhanced Adversarial Training for Neural Response Generation
• A Deep Reinforced Sequence-to-Set Model for Multi-Label Text Classification
• Generating Responses Expressing Emotion in an Open-domain Dialogue System
• Discovering Dialog Structure Graph for Open-Domain Dialog Generation
• A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots
• Dialog State Tracking with Reinforced Data Augmentation
• Sublinear Least-Squares Value Iteration via Locality Sensitive Hashing
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• There is no Artificial General Intelligence
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
• Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
• How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics
• SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
• Improving Neural Relation Extraction with Positive and Unlabeled Learning
• Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering
• Neural Packet Classification
• Cross-domain Dialogue Policy Transfer via Simultaneous Speech-act and Slot Alignment
• A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• DYPLODOC: Dynamic Plots for Document Classification
• TextBox: A Unified, Modularized, and Extensible Framework for Text Generation
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Adversarial Training for Community Question Answer Selection Based on Multi-scale Matching
• Exploration of Reinforcement Learning for Event Camera using Car-like Robots
• Affective Neural Response Generation
• Policy Shaping and Generalized Update Equations for Semantic Parsing from Denotations
• MoodSwipe: A Soft Keyboard that Suggests Messages Based on User-Specified Emotions
• Finding Dominant User Utterances And System Responses in Conversations
• Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training
• Knowledge-Grounded Response Generation with Deep Attentional Latent-Variable Model
• A Discrete CVAE for Response Generation on Short-Text Conversation
• Reinforced Dynamic Reasoning for Conversational Question Generation
• Automating App Review Response Generation Based on Contextual Knowledge
• How to Evaluate Your Dialogue Models: A Review of Approaches
• An Overview of Natural Language State Representation for Reinforcement Learning
• Diversifying Dialogue Generation with Non-Conversational Text
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Investigation of Sentiment Controllable Chatbot
• Generating Dialogue Responses from a Semantic Latent Space
• Towards Automatic Generation of Entertaining Dialogues in Chinese Crosstalks
• Group-wise Contrastive Learning for Neural Dialogue Generation
• Lifelong Learning Dialogue Systems: Chatbots that Self-Learn On the Job
• Condition-Transforming Variational AutoEncoder for Conversation Response Generation
• Sequence-to-Sequence ASR Optimization via Reinforcement Learning
• Collaborative Training of GANs in Continuous and Discrete Spaces for Text Generation
• Relevance-Promoting Language Model for Short-Text Conversation
• Learning How to Self-Learn: Enhancing Self-Training Using Neural Reinforcement Learning
• DialogAct2Vec: Towards End-to-End Dialogue Agent by Multi-Task Representation Learning
• Meta-CoTGAN: A Meta Cooperative Training Paradigm for Improving Adversarial Text Generation
• Paraphrase Generation as Unsupervised Machine Translation
• An Adversarially-Learned Turing Test for Dialog Generation Models
• Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue Task
• Adversarial Bootstrapping for Dialogue Model Training
• Know More about Each Other: Evolving Dialogue Strategy via Compound Assessment
• A Multi-Turn Emotionally Engaging Dialog Model
• Controllable Dialogue Generation with Disentangled Multi-grained Style Specification and Attribute Consistency Reward
• Deep Reinforcement Learning for Personalized Search Story Recommendation
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots
• A Face-to-Face Neural Conversation Model
• Generalized Coarse-to-Fine Visual Recognition with Progressive Training
• Automatic, Personalized, and Flexible Playlist Generation using Reinforcement Learning
• A Bi-Encoder LSTM Model For Learning Unstructured Dialogs
• Ask No More: Deciding when to guess in referential visual dialogue
• "Love is as Complex as Math": Metaphor Generation System for Social Chatbot
• Posterior-GAN: Towards Informative and Coherent Response Generation with Posterior Generative Adversarial Network
• Review-Driven Answer Generation for Product-Related Questions in E-Commerce
• Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging
• Generative Adversarial Imitation Learning for Empathy-based AI
• Improving Computer Generated Dialog with Auxiliary Loss Functions and Custom Evaluation Metrics
• Know Deeper: Knowledge-Conversation Cyclic Utilization Mechanism for Open-domain Dialogue Generation
• Response-Anticipated Memory for On-Demand Knowledge Integration in Response Generation
• A Corpus of Controlled Opinionated and Knowledgeable Movie Discussions for Training Neural Conversation Models
• The RLLChatbot: a solution to the ConvAI challenge
• Learning Data-Driven Objectives to Optimize Interactive Systems
• Improving Automatic Source Code Summarization via Deep Reinforcement Learning
• Multitasking Inhibits Semantic Drift
• Data-Efficient Methods for Dialogue Systems
• DORB: Dynamically Optimizing Multiple Rewards with Bandits
• Ranking sentences from product description & bullets for better search
• Predicting User Engagement Status for Online Evaluation of Intelligent Assistants
• Improving Search through A3C Reinforcement Learning based Conversational Agent
• ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models
• Learning to Generate Multiple Style Transfer Outputs for an Input Sentence
• Targeted Data Acquisition for Evolving Negotiation Agents
• Addressing Objects and Their Relations: The Conversational Entity Dialogue Model
• Predict and Use Latent Patterns for Short-Text Conversation
• Learning Natural Language Generation from Scratch
• Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning
• Do Encoder Representations of Generative Dialogue Models Encode Sufficient Information about the Task ?
• AI-Powered Text Generation for Harmonious Human-Machine Interaction: Current State and Future Directions
• Atom Responding Machine for Dialog Generation
• Distantly Supervised Question Parsing
• Reinforced Natural Language Interfaces via Entropy Decomposition
• High Quality Related Search Query Suggestions using Deep Reinforcement Learning
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems
• Improving Matching Models with Hierarchical Contextualized Representations for Multi-turn Response Selection
• A Study of State Aliasing in Structured Prediction with RNNs
• GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue
• Integrating User and Agent Models: A Deep Task-Oriented Dialogue System
• On the Possibility of Rewarding Structure Learning Agents: Mutual Information on Linguistic Random Sets
• Modeling Coherency in Generated Emails by Leveraging Deep Neural Learners
• Towards meaningful, grounded conversations with intelligent agents