Enriching Conversation Context in Retrieval-based Chatbots

Abstract

Work on retrieval-based chatbots, like most sequence pair matching tasks, can be divided into Cross-encoders that perform word matching over the pair, and Bi-encoders that encode the pair separately. The latter has better performance, however since candidate responses cannot be encoded offline, it is also much slower. Lately, multi-layer transformer architectures pre-trained as language models have been used to great effect on a variety of natural language processing and information retrieval tasks. Recent work has shown that these language models can be used in text-matching scenarios to create Bi-encoders that perform almost as well as Cross-encoders while having a much faster inference speed. In this paper, we expand upon this work by developing a sequence matching architecture that %takes into account contexts in the training dataset at inference time. utilizes the entire training set as a makeshift knowledge-base during inference. We perform detailed experiments demonstrating that this architecture can be used to further improve Bi-encoders performance while still maintaining a relatively high inference speed.

arXiv comment8 pages, 1 figure, 3 tables

References in corpus (7)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
• Deconvolutional Latent-Variable Model for Text Sequence Matching
• Training Millions of Personalized Dialogue Agents
• Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus
• Knowledge-incorporating ESIM models for Response Selection in Retrieval-based Dialog Systems