Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto
arXiv:2506.11112 · doi:10.4230/DagMan.11.1.19
Abstract
During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Conversational Agents Framework for Evaluation (CAFE) for the evaluation of CONIAC systems, consisting of six major components: 1) goals of the system's stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen.
10 figures; Dagstuhl Manifestos, 11(1), pp 19-67. DOI: 10.4230/DagMan.11.1.19
Cited by in corpus (7)
- Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
- Entity-aware Transformers for Entity Search
- Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems
- Zero-shot Query Contextualization for Conversational Search
- Towards Filling the Gap in Conversational Search: From Passage Retrieval to Conversational Response Generation
- A Re-classification of Information Seeking Tasks and Their Computational Solutions
- ZeQR: Zero-shot Query Reformulation for Conversational Search