Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue

Abstract

We demonstrate that large language models are able to simulate Task Oriented Dialogues in novel domains, provided only with an API implementation and a list of goals. We show these simulations can formulate online, automatic metrics that correlate well with human evaluations. Furthermore, by checking for whether the User's goals are met, we can use simulation to repeatedly generate training data and improve the quality of simulations themselves. With no human intervention or domain-specific training data, our simulations bootstrap end-to-end models which achieve a 37\% error reduction in previously unseen domains. By including as few as 32 domain-specific conversations, bootstrapped models can match the performance of a fully-supervised model with $10\times$ more data. To our knowledge, this is the first time simulations have been shown to be effective at bootstrapping models without explicitly requiring any domain-specific training data, rule-engineering, or humans-in-the-loop.

References in corpus (13)

• Language Models are Few-Shot Learners
• Neural Text Generation with Unlikelihood Training
• A Simple Language Model for Task-Oriented Dialogue
• Why We Need New Evaluation Metrics for NLG
• A User Simulator for Task-Completion Dialogues
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
• Finetuned Language Models Are Zero-Shot Learners
• Towards Zero-Label Language Learning
• Learning Robust Dialog Policies in Noisy Environments
• Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning
• Zero-Shot Dialogue State Tracking via Cross-Task Transfer