UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
arXiv:2301.05544 · doi:10.1145/3539597.3573029
Abstract
We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.
arXiv commentProceedings of the Sixteenth ACM International Conference on Web Search and Data Mining
References in corpus (9)
- Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
- DIET: Lightweight Language Understanding for Dialogue Systems
- Evaluating Conversational Recommender Systems via User Simulation
- Towards Question-based Recommender Systems
- Evaluating Mixed-initiative Conversational Search Systems via User Simulation
- Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
- Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems
- IAI MovieBot: A Conversational Movie Recommender System
- RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
Cited by in corpus (11)
- Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
- Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
- User Simulation for Evaluating Information Access Systems
- Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems
- RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems
- UXSim: Towards a Hybrid User Search Simulation
- CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment
- Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
- Identifying Breakdowns in Conversational Recommender Systems using User Simulation
- Informfully Recommenders -- Reproducibility Framework for Diversity-aware Intra-session Recommendations
- IAI MovieBot 2.0: An Enhanced Research Platform with Trainable Neural Components and Transparent User Modeling