Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

Abstract

Reliable automatic evaluation of dialogue systems under an interactive environment has long been overdue. An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interaction, which is usually not affordable for large-scale experiments. Though researchers have attempted to use metrics (e.g., perplexity, BLEU) in language generation tasks or some model-based reinforcement learning methods (e.g., self-play evaluation) for automatic evaluation, these methods only show a very weak correlation with the actual human evaluation in practice. To bridge such a gap, we propose a new framework named ENIGMA for estimating human evaluation scores based on recent advances of off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data (see details in Section 2), which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.

arXiv commentConference on Empirical Methods in Natural Language Processing (EMNLP), 2021

References in corpus (18)

• Gaussian Error Linear Units (GELUs)
• BERTScore: Evaluating Text Generation with BERT
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Towards a Human-like Open-Domain Chatbot
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• GenDICE: Generalized Offline Estimation of Stationary Values
• Machine Translation Evaluation with BERT Regressor
• Off-Policy Evaluation via the Regularized Lagrangian
• Reliable Off-policy Evaluation for Reinforcement Learning
• Nearly Horizon-Free Offline Reinforcement Learning
• On Landscape of Lagrangian Functions and Stochastic Search for Constrained Nonconvex Optimization