What makes a good conversation? How controllable attributes affect human judgments

Abstract

A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.

Accepted to NAACL 2019

References in corpus (5)

• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Topic Aware Neural Response Generation
• Learning to Decode for Future Success
• Topic-based Evaluation for Conversational Bots
• Extending Neural Generative Conversational Model using External Knowledge Sources

Cited by in corpus (12)

• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems
• Towards Standard Criteria for human evaluation of Chatbots: A Survey
• Evaluating Empathetic Chatbots in Customer Service Settings
• Neural Response Generation with Meta-Words
• Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
• How to Evaluate Your Dialogue Models: A Review of Approaches
• Generating Interactive Worlds with Text
• StyleDGPT: Stylized Response Generation with Pre-trained Language Models
• Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation