Metrics for Evaluating Dialogue Strategies in a Spoken Language System

Abstract

In this paper, we describe a set of metrics for the evaluation of different dialogue management strategies in an implemented real-time spoken language system. The set of metrics we propose offers useful insights in evaluating how particular choices in the dialogue management can affect the overall quality of the man-machine dialogue. The evaluation makes use of established metrics: the transaction success, the contextual appropriateness of system answers, the calculation of normal and correction turns in a dialogue. We also define a new metric, the implicit recovery, which allows to measure the ability of a dialogue manager to deal with errors by different levels of analysis. We report evaluation data from several experiments, and we compare two different approaches to dialogue repair strategies using the set of metrics we argue for.

6 pages, LaTex, uses aaai.sty; presented at the 1995 AAAI Spring Symposium Series (Symposium: Empirical Methods in Discourse Interpretation and Generation)

Cited by in corpus (11)

• Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
• PARADISE: A Framework for Evaluating Spoken Dialogue Agents
• Evaluating Competing Agent Strategies for a Voice Email Agent
• On the use of expectations for detecting and repairing human-machine miscommunication
• An Efficient Distribution of Labor in a Two Stage Robust Interpretation Process
• Contextual Information and Specific Language Models for Spoken Language Understanding
• How to Evaluate Your Dialogue Models: A Review of Approaches
• Task-oriented Dialogue Systems: performance vs. quality-optima, a review
• Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems
• Empirically Evaluating an Adaptable Spoken Dialogue System
• Dialogos: a Robust System for Human-Machine Spoken Dialogue on the Telephone