A Deep Reinforcement Learning Chatbot

Abstract

We present MILABOT: a deep reinforcement learning chatbot developed by the Montreal Institute for Learning Algorithms (MILA) for the Amazon Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models. By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed significantly better than many competing systems. Due to its machine learning architecture, the system is likely to improve with additional data.

40 pages, 9 figures, 11 tables

References in corpus (19)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Playing Atari with Deep Reinforcement Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• A Convolutional Neural Network for Modelling Sentences
• Deep Reinforcement Learning for Dialogue Generation
• Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
• Deep Learning for Answer Sentence Selection
• Learning Multiagent Communication with Backpropagation
• Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
• Continuously Learning Neural Dialogue Management
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
• Towards Multi-Agent Communication-Based Language Learning
• DialPort: Connecting the Spoken Dialog Research Community to Real User Data

Cited by in corpus (48)

• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• Arming the public with artificial intelligence to counter social bots
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Towards a Human-like Open-Domain Chatbot
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Assessing The Factual Accuracy of Generated Text
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understanding
• Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
• Deep Learning Based Chatbot Models
• Recent advances in conversational NLP : Towards the standardization of Chatbot building
• Emotionally-Aware Chatbots: A Survey
• Towards Explainable and Controllable Open Domain Dialogue Generation with Dialogue Acts
• AI Benchmark: Running Deep Neural Networks on Android Smartphones
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• Building a Chatbot on a Closed Domain using RASA
• Imitation Learning from Imperfect Demonstration
• Avoiding Echo-Responses in a Retrieval-Based Conversation System
• A Deep Reinforcement Learning Chatbot (Short Version)
• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Learning to Coordinate Multiple Reinforcement Learning Agents for Diverse Query Reformulation
• Science Driven Innovations Powering Mobile Product: Cloud AI vs. Device AI Solutions on Smart Device
• Deploying Lifelong Open-Domain Dialogue Learning
• The Eigenoption-Critic Framework
• Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer
• Multi-Domain Dialogue Acts and Response Co-Generation
• Text Generation by Learning from Demonstrations
• Contextual Text Style Transfer
• Diversifying Dialogue Generation with Non-Conversational Text
• An Ensemble Dialogue System for Facts-Based Sentence Generation
• Generate, Filter, and Rank: Grammaticality Classification for Production-Ready NLG Systems
• Tartan: A retrieval-based socialbot powered by a dynamic finite-state machine architecture
• When is ACL's Deadline? A Scientific Conversational Agent
• Proposal Towards a Personalized Knowledge-powered Self-play Based Ensemble Dialog System
• Dehumanizing Voice Technology: Phonetic & Experiential Consequences of Restricted Human-Machine Interaction
• Bootstrapped Q-learning with Context Relevant Observation Pruning to Generalize in Text-based Games
• Weakly-Supervised Neural Response Selection from an Ensemble of Task-Specialised Dialogue Agents
• Learning Representations and Agents for Information Retrieval
• Data-Efficient Methods for Dialogue Systems
• Automatically Exposing Problems with Neural Dialog Models
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention
• Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning
• Building A User-Centric and Content-Driven Socialbot
• The RLLChatbot: a solution to the ConvAI challenge