Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

Abstract

A long-term goal of machine learning is to build intelligent conversational agents. One recent popular approach is to train end-to-end models on a large amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015). However, this approach leaves many questions unanswered as an understanding of the precise successes and shortcomings of each model is hard to assess. A contrasting recent proposal are the bAbI tasks (Weston et al., 2015b) which are synthetic data that measure the ability of learning machines at various reasoning tasks over toy language. Unfortunately, those tests are very small and hence may encourage methods that do not scale. In this work, we propose a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes. Choosing the domain of movies, we provide tasks that test the ability of models to answer factual questions (utilizing OMDB), provide personalization (utilizing MovieLens), carry short conversations about the two, and finally to perform on natural dialogs from Reddit. We provide a dataset covering 75k movie entities and with 3.5M training examples. We present results of various models on these tasks, and evaluate their performance.

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Large-scale Simple Question Answering with Memory Networks
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Question Answering with Subgraph Embeddings

Cited by in corpus (31)

• Recent Trends in Deep Learning Based Natural Language Processing
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• Dialog-based Language Learning
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Hierarchical Memory Networks
• Deep Learning Based Chatbot Models
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Training Millions of Personalized Dialogue Agents
• Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus
• Sentence Pair Scoring: Towards Unified Framework for Text Comprehension
• GuessWhat?! Visual object discovery through multi-modal dialogue
• Zero-Shot Dialog Generation with Cross-Domain Latent Actions
• Learning Conversational Systems that Interleave Task and Non-Task Content
• Integrating both Visual and Audio Cues for Enhanced Video Caption
• Multimodal Memory Modelling for Video Captioning
• Prototypical Q Networks for Automatic Conversational Diagnosis and Few-Shot New Disease Adaption
• Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
• Modeling Intent, Dialog Policies and Response Adaptation for Goal-Oriented Interactions
• Learning Personalized End-to-End Goal-Oriented Dialog
• Few-Shot Object Recognition from Machine-Labeled Web Images
• Towards End-to-End Learning for Efficient Dialogue Agent by Modeling Looking-ahead Ability
• Neural MultiVoice Models for Expressing Novel Personalities in Dialog
• Knowledge Grounded Conversational Symptom Detection with Graph Memory Networks
• Iterative Multi-document Neural Attention for Multiple Answer Prediction
• Knowledge-based end-to-end memory networks
• e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with Explanations
• Improving Search through A3C Reinforcement Learning based Conversational Agent
• Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
• Incremental Learning from Scratch for Task-Oriented Dialogue Systems