Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
arXiv:1709.06136
Abstract
In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models.
Accepted at ASRU 2017
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- A User Simulator for Task-Completion Dialogues
- Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
- End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning
- An End-to-End Trainable Neural Network Model with Belief Tracking for Task-Oriented Dialog
- On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
Cited by in corpus (6)
- A Deep Reinforcement Learning Chatbot
- End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
- Recent Advances and Challenges in Task-oriented Dialog System
- Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
- Sentiment Adaptive End-to-End Dialog Systems
- Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning