UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
arXiv:2301.05544 · doi:10.1145/3539597.3573029
Abstract
We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.
Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining
References in corpus (8)
- Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
- DIET: Lightweight Language Understanding for Dialogue Systems
- Evaluating Conversational Recommender Systems via User Simulation
- Towards Question-based Recommender Systems
- Evaluating Mixed-initiative Conversational Search Systems via User Simulation
- Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems
- IAI MovieBot: A Conversational Movie Recommender System
- RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems