An Evaluation Protocol for Generative Conversational Systems
arXiv:2010.12741
Abstract
There is a multitude of novel generative models for open-domain conversational systems; however, there is no systematic evaluation of different systems. Systematic comparisons require consistency in experimental design, evaluation sets, conversational systems and their outputs, and statistical analysis. We lay out a protocol for the evaluation of conversational models using head-to-head pairwise comparison. We analyze ten recent models that claim state-of-the-art performance using a paired head-to-head performance (win-loss-tie) on five evaluation datasets. Our findings show that DialoGPT and Blender are superior systems using Bradley-Terry model and TrueSkill ranking methods. These findings demonstrate the feasibility of our protocol to evaluate conversational agents and evaluation sets. Finally, we make all code and evaluations publicly available for researchers to compare their model to other state-of-the-art dialog models.
References in corpus (8)
- Language Models are Few-Shot Learners
- A Neural Conversational Model
- Wizard of Wikipedia: Knowledge-Powered Conversational agents
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- Towards a Human-like Open-Domain Chatbot
- ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
- Deep Learning Based Chatbot Models