Adversarial Learning of Task-Oriented Neural Dialog Models
arXiv:1805.11762
Abstract
In this work, we propose an adversarial learning method for reward estimation in reinforcement learning (RL) based task-oriented dialog models. Most of the current RL based task-oriented dialog systems require the access to a reward signal from either user feedback or user ratings. Such user ratings, however, may not always be consistent or available in practice. Furthermore, online dialog policy learning with RL typically requires a large number of queries to users, suffering from sample efficiency problem. To address these challenges, we propose an adversarial learning method to learn dialog rewards directly from dialog samples. Such rewards are further used to optimize the dialog policy with policy gradient based RL. In the evaluation in a restaurant search domain, we show that the proposed adversarial dialog learning method achieves advanced dialog success rate comparing to strong baseline methods. We further discuss the covariate shift problem in online adversarial dialog learning and show how we can address that with partial access to user feedback.
arXiv commentTo appear at SIGDIAL 2018
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
- Generative Adversarial Imitation Learning
- Adversarial Learning for Neural Dialogue Generation
- A Persona-Based Neural Conversation Model
- PARADISE: A Framework for Evaluating Spoken Dialogue Agents
- Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
- End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning
- Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
- Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
- Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning