Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation

Abstract

Transformer-based models have demonstrated excellent capabilities of capturing patterns and structures in natural language generation and achieved state-of-the-art results in many tasks. In this paper we present a transformer-based model for multi-turn dialog response generation. Our solution is based on a hybrid approach which augments a transformer-based generative model with a novel retrieval mechanism, which leverages the memorized information in the training data via k-Nearest Neighbor search. Our system is evaluated on two datasets made by customer/assistant dialogs: the Taskmaster-1, released by Google and holding high quality, goal-oriented conversational data and a proprietary dataset collected from a real customer service call center. Both achieve better BLEU scores over strong baselines.

arXiv commentThe International FLAIRS Conference Proceedings volume 34 issue 1

References in corpus (7)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• REALM: Retrieval-Augmented Language Model Pre-Training
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• A Diversity-Promoting Objective Function for Neural Conversation Models
• ERNIE: Enhanced Language Representation with Informative Entities
• Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
• Generalization through Memorization: Nearest Neighbor Language Models

Cited by in corpus (1)

• Towards Retrieval-based Conversational Recommendation