Towards a Human-like Open-Domain Chatbot

Abstract

We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

38 pages, 12 figures

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• A Neural Conversational Model
• The Curious Case of Neural Text Degeneration
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Large-Scale Evolution of Image Classifiers
• Topic Aware Neural Response Generation
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• A Deep Reinforcement Learning Chatbot
• The Evolved Transformer
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
• Hierarchical Neural Story Generation
• The price of debiasing automatic metrics in natural language evaluation

Cited by in corpus (112)

• AraBERT: Transformer-based Model for Arabic Language Understanding
• The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
• Extracting Training Data from Large Language Models
• A Simple Language Model for Task-Oriented Dialogue
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
• Recipes for Safety in Open-domain Chatbots
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• Multimodal Few-Shot Learning with Frozen Language Models
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Zero-Resource Knowledge-Grounded Dialogue Generation
• Towards Zero-Label Language Learning
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• Challenges in Building Intelligent Open-domain Dialog Systems
• Towards Ecologically Valid Research on Language User Interfaces
• Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
• Knowledge-Grounded Dialogue Flow Management for Social Robots and Conversational Agents
• Wordcraft: a Human-AI Collaborative Editor for Story Writing
• Controlling Style in Generated Dialogue
• Few-shot Natural Language Generation for Task-Oriented Dialog
• Piracy Resistant Watermarks for Deep Neural Networks
• EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training
• PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
• PTR: Prompt Tuning with Rules for Text Classification
• On future power system digital twins: A vision towards a standard architecture
• STAR: A Schema-Guided Dialog Dataset for Transfer Learning
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Deep Transformers with Latent Depth
• Gunrock 2.0: A User Adaptive Social Conversational System
• RealFormer: Transformer Likes Residual Attention
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• Trading Off Diversity and Quality in Natural Language Generation
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
• Automatic Evaluation and Moderation of Open-domain Dialogue Systems
• Pre-Trained Models: Past, Present and Future
• Robust Conversational AI with Grounded Text Generation
• What they do when in doubt: a study of inductive biases in seq2seq learners
• Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamics
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Primer: Searching for Efficient Transformers for Language Modeling
• Augmenting Decompiler Output with Learned Variable Names and Types
• Non-Autoregressive Neural Dialogue Generation
• Risk Communication in Asian Countries: COVID-19 Discourse on Twitter
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• Towards Standard Criteria for human evaluation of Chatbots: A Survey
• Truthful AI: Developing and governing AI that does not lie
• RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
• Evaluating the Robustness of Collaborative Agents
• Evaluating Empathetic Chatbots in Customer Service Settings
• A multi-component framework for the analysis and design of explainable artificial intelligence
• CareCall: a Call-Based Active Monitoring Dialog Agent for Managing COVID-19 Pandemic
• DynaEval: Unifying Turn and Dialogue Level Evaluation
• Emora: An Inquisitive Social Chatbot Who Cares For You
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data
• GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems
• Retrieval Augmentation Reduces Hallucination in Conversation
• A Unified Pre-training Framework for Conversational AI
• Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
• A Large-Scale Chinese Short-Text Conversation Dataset
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• Countering Language Drift with Seeded Iterated Learning
• An Evaluation Protocol for Generative Conversational Systems
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• An ontology-based chatbot for crises management: use case coronavirus
• On the Discrepancy between Density Estimation and Sequence Generation
• Pretraining the Noisy Channel Model for Task-Oriented Dialogue
• Language Models as a Knowledge Source for Cognitive Agents
• NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment
• Modelling Direct Messaging Networks with Multiple Recipients for Cyber Deception
• Enhancing Self-Disclosure In Neural Dialog Models By Candidate Re-ranking
• Improving Longer-range Dialogue State Tracking
• Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems
• EigenGame Unloaded: When playing games is better than optimizing
• Open-Domain Dialogue Generation Based on Pre-trained Language Models
• Cue-word Driven Neural Response Generation with a Shrinking Vocabulary
• A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19
• Toward a Human-Level Video Understanding Intelligence
• On the Ethics of Building AI in a Responsible Manner
• Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems
• A Response Retrieval Approach for Dialogue Using a Multi-Attentive Transformer
• Learning Neural Templates for Recommender Dialogue System
• Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency
• Which Kind Is Better in Open-domain Multi-turn Dialog,Hierarchical or Non-hierarchical Models? An Empirical Study
• StyleDGPT: Stylized Response Generation with Pre-trained Language Models
• TIMEDIAL: Temporal Commonsense Reasoning in Dialog
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots
• $Q^2$: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering
• Towards a Universal NLG for Dialogue Systems and Simulators with Future Bridging
• Investigating Label Bias in Beam Search for Open-ended Text Generation
• Maria: A Visual Experience Powered Conversational Agent
• Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
• Local Knowledge Powered Conversational Agents
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Improvement of a dedicated model for open domain persona-aware dialogue generation
• A Dataset and Baselines for Multilingual Reply Suggestion
• Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN
• Towards meaningful, grounded conversations with intelligent agents
• NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
• Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation
• Dialogue State Tracking with a Language Model using Schema-Driven Prompting
• Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation
• Modeling and Utilizing User's Internal State in Movie Recommendation Dialogue
• An Approach to Inference-Driven Dialogue Management within a Social Chatbot
• A Grounded Well-being Conversational Agent with Multiple Interaction Modes: Preliminary Results
• How BPE Affects Memorization in Transformers
• Pretrained Language Models are Symbolic Mathematics Solvers too!