Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents

Abstract

We observe a change in the way users access information, that is, the rise of conversational information access (CIA) agents. However, the automatic evaluation of these agents remains an open challenge. Moreover, the training of CIA agents is cumbersome as it mostly relies on conversational corpora, expert knowledge, and reinforcement learning. User simulation has been identified as a promising solution to tackle automatic evaluation and has been previously used in reinforcement learning. In this research, we investigate how user simulation can be leveraged in the context of CIA. We organize the work in three parts. We begin with the identification of requirements for user simulators for training and evaluating CIA agents and compare existing types of simulator regarding these. Then, we plan to combine these different types of simulators into a new hybrid simulator. Finally, we aim to extend simulators to handle more complex information seeking scenarios.

Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM '24), 2024

References in corpus (8)

• Evaluating Conversational Recommender Systems via User Simulation
• Evaluating Mixed-initiative Conversational Search Systems via User Simulation
• Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
• UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
• MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce
• IAI MovieBot: A Conversational Movie Recommender System
• User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems