Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning

Abstract

This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning. To obviate DDQ's high dependency on the quality of simulated experiences, we incorporate an RNN-based discriminator in D3Q to differentiate simulated experience from real user experience in order to control the quality of training data. Experiments show that D3Q significantly outperforms DDQ by controlling the quality of simulated experience used for planning. The effectiveness and robustness of D3Q is further demonstrated in a domain extension setting, where the agent's capability of adapting to a changing environment is tested.

arXiv comment11 pages, 10 figures, EMNLP 2018 long paper

References in corpus (6)

• The Predictron: End-To-End Learning and Planning
• A User Simulator for Task-Completion Dialogues
• Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
• Continuously Learning Neural Dialogue Management
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
• Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning

Cited by in corpus (5)

• Recent Advances and Challenges in Task-oriented Dialog System
• Towards Unsupervised Language Understanding and Generation by Joint Dual Learning
• Switch-based Active Deep Dyna-Q: Efficient Adaptive Planning for Task-Completion Dialogue Policy Learning
• Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
• Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems