Local Knowledge Powered Conversational Agents

Abstract

State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models. However, even with these advancements, conversational agents still lack the ability to produce responses that are informative and coherent with the local context. In this work, we propose a dialog framework that incorporates both local knowledge as well as users' past dialogues to generate high quality conversations. We introduce an approach to build a dataset based on Reddit conversations, where outbound URL links are widely available in the conversations and the hyperlinked documents can be naturally included as local external knowledge. Using our framework and dataset, we demonstrate that incorporating local knowledge can largely improve informativeness, coherency and realisticness measures using human evaluations. In particular, our approach consistently outperforms the state-of-the-art conversational model on the Reddit dataset across all three measures. We also find that scaling the size of our models from 117M to 8.3B parameters yields consistent improvement of validation perplexity as well as human evaluated metrics. Our model with 8.3B parameters can generate human-like responses as rated by various human evaluations in a single-turn dialog setting.

References in corpus (14)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• A Neural Conversational Model
• Universal Sentence Encoder
• The Curious Case of Neural Text Degeneration
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Towards a Human-like Open-Domain Chatbot
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue