Evaluating Conversational Recommender Systems: A Landscape of Research

Abstract

Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural language processing, and AI in general, such systems received increased attention in recent years. Technically, conversational recommenders are usually complex multi-component applications and often consist of multiple machine learning models and a natural language user interface. Evaluating such a complex system in a holistic way can therefore be challenging, as it requires (i) the assessment of the quality of the different learning components, and (ii) the quality perception of the system as a whole by users. Thus, a mixed methods approach is often required, which may combine objective (computational) and subjective (perception-oriented) evaluation techniques. In this paper, we review common evaluation approaches for conversational recommender systems, identify possible limitations, and outline future directions towards more holistic evaluation practices.

References in corpus (7)

• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• A Survey on Conversational Recommender Systems
• Beyond Personalization: Research Directions in Multistakeholder Recommendation
• A Personalized System for Conversational Recommendations
• Measuring the Business Value of Recommender Systems
• Evaluating Conversational Recommender Systems via User Simulation
• Exploring Longitudinal Effects of Session-based Recommendations

Cited by in corpus (3)

• A Survey on Point-of-Interest Recommendations Leveraging Heterogeneous Data
• Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems
• EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context