12 papers
User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation
Krisztian Balog, ChengXiang Zhai
User simulation is an emerging interdisciplinary topic with multiple critical applications in the era of Generative AI. It involves creating an intelligent agent that mimics the ac…
UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems
Nolwenn Bernard, Krisztian Balog
Resources for simulation-based evaluation of conversational recommender systems (CRSs) are scarce. The UserSimCRS toolkit was introduced to address this gap. In this work, we prese…
Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction
Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer +2
Validating user simulation is a difficult task due to the lack of established measures and benchmarks, which makes it challenging to assess whether a simulator accurately reflects…
SimLab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems
Nolwenn Bernard, Sharath Chandra Etagi Suresh, Krisztian Balog +1
Progress in conversational information access (CIA) systems has been hindered by the difficulty of evaluating such systems with reproducible experiments. While user simulation offe…
Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulation
Nolwenn Bernard, Krisztian Balog
Research and development on conversational recommender systems (CRSs) critically depends on sound and reliable evaluation methodologies. However, the interactive nature of these sy…
The Indispensable Role of User Simulation in the Pursuit of AGI
Krisztian Balog, ChengXiang Zhai
Progress toward Artificial General Intelligence (AGI) faces significant bottlenecks, particularly in rigorously evaluating complex interactive systems and acquiring the vast intera…