Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize

Abstract

Building open domain conversational systems that allow users to have engaging conversations on topics of their choice is a challenging task. Alexa Prize was launched in 2016 to tackle the problem of achieving natural, sustained, coherent and engaging open-domain dialogs. In the second iteration of the competition in 2018, university teams advanced the state of the art by using context in dialog models, leveraging knowledge graphs for language understanding, handling complex utterances, building statistical and hierarchical dialog managers, and leveraging model-driven signals from user responses. The 2018 competition also included the provision of a suite of tools and models to the competitors including the CoBot (conversational bot) toolkit, topic and dialog act detection models, conversation evaluators, and a sensitive content detection model so that the competing teams could focus on building knowledge-rich, coherent and engaging multi-turn dialog systems. This paper outlines the advances developed by the university teams as well as the Alexa Prize team to achieve the common goal of advancing the science of Conversational AI. We address several key open-ended problems such as conversational speech recognition, open domain natural language understanding, commonsense reasoning, statistical dialog management, and dialog evaluation. These collaborative efforts have driven improved experiences by Alexa users to an average rating of 3.61, the median duration of 2 mins 18 seconds, and average turns to 14.6, increases of 14%, 92%, 54% respectively since the launch of the 2018 competition. For conversational speech recognition, we have improved our relative Word Error Rate by 55% and our relative Entity Error Rate by 34% since the launch of the Alexa Prize. Socialbots improved in quality significantly more rapidly in 2018, in part due to the release of the CoBot toolkit.

2018 Alexa Prize Proceedings

References in corpus (7)

• Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
• Conversational AI: The Science Behind the Alexa Prize
• Rasa: Open Source Language Understanding and Dialogue Management
• Topic-based Evaluation for Conversational Bots
• RubyStar: A Non-Task-Oriented Mixture Model Dialog System
• Slugbot: An Application of a Novel and Scalable Open Domain Socialbot Framework
• Predicting Causes of Reformulation in Intelligent Assistants

Cited by in corpus (15)

• True Few-Shot Learning with Language Models
• Recipes for Safety in Open-domain Chatbots
• Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations
• Gunrock 2.0: A User Adaptive Social Conversational System
• Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• HarperValleyBank: A Domain-Specific Spoken Dialog Corpus
• Emora: An Inquisitive Social Chatbot Who Cares For You
• Audrey: A Personalized Open-Domain Conversational Bot
• Implicit Discourse Relation Identification for Open-domain Dialogues
• Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations
• Viola: A Topic Agnostic Generate-and-Rank Dialogue System
• Measuring Conversational Fluidity in Automated Dialogue Agents
• LIDA: Lightweight Interactive Dialogue Annotator
• Would you Like to Talk about Sports Now? Towards Contextual Topic Suggestion for Open-Domain Conversational Agents