End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning

Abstract

In this paper, we present a neural network based task-oriented dialogue system that can be optimized end-to-end with deep reinforcement learning (RL). The system is able to track dialogue state, interface with knowledge bases, and incorporate query results into agent's responses to successfully complete task-oriented dialogues. Dialogue policy learning is conducted with a hybrid supervised and deep RL methods. We first train the dialogue agent in a supervised manner by learning directly from task-oriented dialogue corpora, and further optimize it with deep RL during its interaction with users. In the experiments on two different dialogue task domains, our model demonstrates robust performance in tracking dialogue state and producing reasonable system responses. We show that deep RL based optimization leads to significant improvement on task success rate and reduction in dialogue length comparing to supervised training model. We further show benefits of training task-oriented dialogue model end-to-end comparing to component-wise optimization with experiment results on dialogue simulations and human evaluations.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Deep Reinforcement Learning for Dialogue Generation
• An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models

Cited by in corpus (13)

• Building a Conversational Agent Overnight with Dialogue Self-Play
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Neural Language Generation: Formulation, Methods, and Evaluation
• Flexible and Scalable State Tracking Framework for Goal-Oriented Dialogue Systems
• Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
• An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking
• User Modeling for Task Oriented Dialogues
• RMM: A Recursive Mental Model for Dialog Navigation
• Chat More If You Like: Dynamic Cue Words Planning to Flow Longer Conversations
• Modeling Intent, Dialog Policies and Response Adaptation for Goal-Oriented Interactions
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems
• Transferable Dialogue Systems and User Simulators