Meta-evaluation of Conversational Search Evaluation Metrics
arXiv:2104.13453 · doi:10.1145/3445029
Abstract
Conversational search systems, such as Google Assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challenging given that any natural language responses could be generated, and users commonly interact for multiple semantically coherent rounds to accomplish a search task. Although prior studies proposed many evaluation metrics, the extent of how those measures effectively capture user preference remains to be investigated. In this paper, we systematically meta-evaluate a variety of conversational search metrics. We specifically study three perspectives on those metrics: (1) reliability: the ability to detect "actual" performance differences as opposed to those observed by chance; (2) fidelity: the ability to agree with ultimate user preference; and (3) intuitiveness: the ability to capture any property deemed important: adequacy, informativeness, and fluency in the context of conversational search. By conducting experiments on two test collections, we find that the performance of different metrics varies significantly across different scenarios whereas consistent with prior studies, existing metrics only achieve a weak correlation with ultimate user preference and satisfaction. METEOR is, comparatively speaking, the best existing single-turn metric considering all three perspectives. We also demonstrate that adapted session-based evaluation metrics can be used to measure multi-turn conversational search, achieving moderate concordance with user satisfaction. To our knowledge, our work establishes the most comprehensive meta-evaluation for conversational search to date.
arXiv comment43 pages
References in corpus (25)
- Sequence to Sequence Learning with Neural Networks
- BERTScore: Evaluating Text Generation with BERT
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- Wizard of Wikipedia: Knowledge-Powered Conversational agents
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Survey on Evaluation Methods for Dialogue Systems
- Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization
- Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- Analyzing and Characterizing User Intent in Information-seeking Conversations
- deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
- Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
- Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models
- Explicit State Tracking with Semi-Supervision for Neural Dialogue Generation
- The Eighth Dialog System Technology Challenge
- Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems
- PONE: A Novel Automatic Evaluation Metric for Open-Domain Generative Dialogue Systems
- Deep Retrieval-Based Dialogue Systems: A Short Review
- Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
- Re-evaluating Automatic Metrics for Image Captioning
- WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval
- Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker
- When to Talk: Chatbot Controls the Timing of Talking during Multi-turn Open-domain Dialogue Generation
- Sentence-Level Fluency Evaluation: References Help, But Can Be Spared!