Deep Learning Based Chatbot Models
arXiv:1908.08835
Abstract
A conversational agent (chatbot) is a piece of software that is able to communicate with humans using natural language. Modeling conversation is an important task in natural language processing and artificial intelligence. While chatbots can be used for various tasks, in general they have to understand users' utterances and provide responses that are relevant to the problem at hand. In my work, I conduct an in-depth survey of recent literature, examining over 70 publications related to chatbots published in the last 3 years. Then, I proceed to make the argument that the very nature of the general conversation domain demands approaches that are different from current state-of-of-the-art architectures. Based on several examples from the literature I show why current chatbot models fail to take into account enough priors when generating responses and how this affects the quality of the conversation. In the case of chatbots, these priors can be outside sources of information that the conversation is conditioned on like the persona or mood of the conversers. In addition to presenting the reasons behind this problem, I propose several ideas on how it could be remedied. The next section focuses on adapting the very recent Transformer model to the chatbot domain, which is currently state-of-the-art in neural machine translation. I first present experiments with the vanilla model, using conversations extracted from the Cornell Movie-Dialog Corpus. Secondly, I augment the model with some of my ideas regarding the issues of encoder-decoder architectures. More specifically, I feed additional features into the model like mood or persona together with the raw conversation data. Finally, I conduct a detailed analysis of how the vanilla model performs on conversational data by comparing it to previous chatbot models and how the additional features affect the quality of the generated responses.
67 pages. Written in October of 2017 for a university conference. In April of 2019, it won first place at the Hungarian Scientific Students' Associations Report, which is a national competition-like conference for students
References in corpus (48)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Playing Atari with Deep Reinforcement Learning
- Neural Architecture Search with Reinforcement Learning
- Convolutional Sequence to Sequence Learning
- A Neural Conversational Model
- A Structured Self-attentive Sentence Embedding
- A Deep Reinforced Model for Abstractive Summarization
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Deep Reinforcement Learning for Dialogue Generation
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- Neural Machine Translation in Linear Time
- Topic Aware Neural Response Generation
- One Model To Learn Them All
- Depthwise Separable Convolutions for Neural Machine Translation
- Adversarial Learning for Neural Dialogue Generation
- A Deep Reinforcement Learning Chatbot
- Long Short-Term Memory-Networks for Machine Reading
- A Survey of Available Corpora for Building Data-Driven Dialogue Systems
- A Persona-Based Neural Conversation Model
- A Network-based End-to-End Trainable Task-oriented Dialogue System
- Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
- Hierarchical Recurrent Attention Network for Response Generation
- Sequence-to-Sequence Learning as Beam-Search Optimization
- Addressing the Rare Word Problem in Neural Machine Translation
- Attention with Intention for a Neural Network Conversation Model
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
- Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Learning End-to-End Goal-Oriented Dialog
- Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation
- Personalization in Goal-Oriented Dialog
- A Decomposable Attention Model for Natural Language Inference
- A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
- Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
- Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
- A Conditional Variational Framework for Dialog Generation
- An Attentional Neural Conversation Model with Improved Specificity
- StalemateBreaker: A Proactive Content-Introducing Approach to Automatic Human-Computer Conversation
- Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems
- Domain Aware Neural Dialog System
- Memory-augmented Neural Machine Translation
- Generative Encoder-Decoder Models for Task-Oriented Spoken Dialog Systems with Chatting Capability
- Enterprise to Computer: Star Trek chatbot
- An online sequence-to-sequence model for noisy speech recognition