CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems

Abstract

We introduce CRS Arena, a research platform for scalable benchmarking of Conversational Recommender Systems (CRS) based on human feedback. The platform displays pairwise battles between anonymous conversational recommender systems, where users interact with the systems one after the other before declaring either a winner or a draw. CRS Arena collects conversations and user feedback, providing a foundation for reliable evaluation and ranking of CRSs. We conduct experiments with CRS Arena on both open and closed crowdsourcing platforms, confirming that both setups produce highly correlated rankings of CRSs and conversations with similar characteristics. We release CRSArena-Dial, a dataset of 474 conversations and their corresponding user feedback, along with a preliminary ranking of the systems based on the Elo rating system. The platform is accessible at https://iai-group-crsarena.hf.space/.

arXiv commentProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining (WSDM '25), March 10--14, 2025, Hannover, Germany

References in corpus (6)

• A Survey on Conversational Recommender Systems
• Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
• Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search
• Towards Retrieval-based Conversational Recommendation
• User Simulation for Evaluating Information Access Systems