Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play

Abstract

End-to-end neural approaches are becoming increasingly common in conversational scenarios due to their promising performances when provided with sufficient amount of data. In this paper, we present a novel methodology to address the interpretability of neural approaches in such scenarios by creating challenge datasets using dialogue self-play over multiple tasks/intents. Dialogue self-play allows generating large amount of synthetic data; by taking advantage of the complete control over the generation process, we show how neural approaches can be evaluated in terms of unseen dialogue patterns. We propose several out-of-pattern test cases each of which introduces a natural and unexpected user utterance phenomenon. As a proof of concept, we built a single and a multiple memory network, and show that these two architectures have diverse performances depending on the peculiar dialogue patterns.

arXiv commentProceedings of Recent Advances in Natural Language Processing (RANLP) Conference, 2019

References in corpus (6)

• A Neural Conversational Model
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
• Building a Conversational Agent Overnight with Dialogue Self-Play
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Zero-Shot Dialog Generation with Cross-Domain Latent Actions