Assessing Dialogue Systems with Distribution Distances
arXiv:2105.02573
Abstract
An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Specifically, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.
7 pages, 2 figures
References in corpus (9)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- BERTScore: Evaluating Text Generation with BERT
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- A Neural Conversational Model
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- Assessing Generative Models via Precision and Recall
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- Towards Universal Paraphrastic Sentence Embeddings
- Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings