RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
arXiv:2012.14666
Abstract
For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities or domains. In pursuit of these goals, we introduce the RADDLE benchmark, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains. By including tasks with limited training data, RADDLE is designed to favor and encourage models with a strong generalization ability. RADDLE also includes a diagnostic checklist that facilitates detailed robustness analysis in aspects such as language variations, speech errors, unseen entities, and out-of-domain utterances. We evaluate recent state-of-the-art systems based on pre-training and fine-tuning, and find that grounded pre-training on heterogeneous dialog corpora performs better than training a separate model per domain. Overall, existing models are less than satisfactory in robustness evaluation, which suggests opportunities for future improvement.
12 pages; Project website at aka.ms/raddle
References in corpus (16)
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- A Network-based End-to-End Trainable Task-oriented Dialogue System
- Building a Conversational Agent Overnight with Dialogue Self-Play
- A Simple Language Model for Task-Oriented Dialogue
- DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
- SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
- Adversarial Training for Large Neural Language Models
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Adversarial NLI: A New Benchmark for Natural Language Understanding
- The Eighth Dialog System Technology Challenge
- Few-shot Natural Language Generation for Task-Oriented Dialog
- Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
- Robust Conversational AI with Grounded Text Generation
- Contextual Out-of-Domain Utterance Handling With Counterfeit Data Augmentation
- Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training