How to Build User Simulators to Train RL-based Dialog Systems

Abstract

User simulators are essential for training reinforcement learning (RL) based dialog models. The performance of the simulator directly impacts the RL policy. However, building a good user simulator that models real user behaviors is challenging. We propose a method of standardizing user simulator building that can be used by the community to compare dialog system quality using the same set of user simulators fairly. We present implementations of six user simulators trained with different dialog planning and generation methods. We then calculate a set of automatic metrics to evaluate the quality of these simulators both directly and indirectly. We also ask human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems. This paper presents a comprehensive evaluation framework for user simulator study and provides a better understanding of the pros and cons of different user simulators, as well as their impacts on the trained systems.

Long Paper Accepted by EMNLP 2019

References in corpus (16)

• Sequence to Sequence Learning with Neural Networks
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Building a Conversational Agent Overnight with Dialogue Self-Play
• A User Simulator for Task-Completion Dialogues
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking
• Strategic Dialogue Management via Deep Reinforcement Learning
• Unifying Human and Statistical Evaluation for Natural Language Generation
• ConvLab: Multi-Domain End-to-End Dialog System Platform
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• Domain Adaptive Dialog Generation via Meta Learning
• Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management
• Unsupervised Dialog Structure Learning
• Decoupling Strategy and Generation in Negotiation Dialogues
• Sentiment Adaptive End-to-End Dialog Systems

Cited by in corpus (3)

• Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies
• Recent Advances and Challenges in Task-oriented Dialog System
• Every time I fire a conversational designer, the performance of the dialog system goes down