A Tailored Pre-Training Model for Task-Oriented Dialog Generation
arXiv:2004.13835
Abstract
The recent success of large pre-trained language models such as BERT and GPT-2 has suggested the effectiveness of incorporating language priors in downstream dialog generation tasks. However, the performance of pre-trained models on the dialog task is not as optimal as expected. In this paper, we propose a Pre-trained Role Alternating Language model (PRAL), designed specifically for task-oriented conversational systems. We adopted (Wu et al., 2019) that models two speakers separately. We also design several techniques, such as start position randomization, knowledge distillation, and history discount to improve pre-training performance. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. We test PRAL on three different downstream tasks. The results show that PRAL performs better or on par with state-of-the-art methods.
7 pages, 1 figure
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
- DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
- Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
- Conversations Gone Awry: Detecting Early Signs of Conversational Failure
- Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems
- Talking to myself: self-dialogues as data for conversational agents
- Effective Incorporation of Speaker Information in Utterance Encoding in Dialog