PARADISE: A Framework for Evaluating Spoken Dialogue Agents

Abstract

This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normalizing for task complexity.

10 pages, uses aclap, psfig, lingmacros, times

References in corpus (2)

• Metrics for Evaluating Dialogue Strategies in a Spoken Language System
• Response Generation in Collaborative Negotiation

Cited by in corpus (12)

• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Impacts of Personal Characteristics on User Trust in Conversational Recommender Systems
• Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• INSPIRE: Evaluation of a Smart-Home System for Infotainment Management and Device Control
• Task-oriented Dialogue Systems: performance vs. quality-optima, a review
• Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents
• Adversarial Learning of Task-Oriented Neural Dialog Models