User Simulation for Evaluating Information Access Systems

Abstract

Information access systems, such as search engines, recommender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our information needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scientific challenge. This challenge is rooted in the difficulty of assessing a system's overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We begin with a background of information access system evaluation and explore the diverse applications of user simulation. Subsequently, we systematically review the major research progress in user simulation, covering both general frameworks for designing user simulators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the book with a detailed discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.

v1: initial draft; v2: final version to appear in Foundations and Trends in Information Retrieval

References in corpus (22)

• A Survey on Dialogue Systems: Recent Advances and New Frontiers
• A Survey on Conversational Recommender Systems
• Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System
• Survey on Evaluation Methods for Dialogue Systems
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
• Click-Through Rate Prediction in Online Advertising: A Literature Review
• Analyzing and Characterizing User Intent in Information-seeking Conversations
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
• Balancing Consumer and Business Value of Recommender Systems: A Simulation-based Analysis
• Evaluating Mixed-initiative Conversational Search Systems via User Simulation
• Meta-evaluation of Conversational Search Evaluation Metrics
• Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
• Carousel Personalization in Music Streaming Apps with Contextual Bandits
• Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
• UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
• Ranking for Relevance and Display Preferences in Complex Presentation Layouts
• MG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce
• Analyzing and Simulating User Utterance Reformulation in Conversational Recommender Systems
• A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels
• Improving Conversational Recommendation Systems via Counterfactual Data Simulation
• Asking Clarifying Questions Based on Negative Feedback in Conversational Search
• Users and Contemporary SERPs: A (Re-)Investigation Examining User Interactions and Experiences

Cited by in corpus (5)

• Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)
• A Survey on Recent Advances in Conversational Data Generation
• Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access
• CRS Arena: Crowdsourced Benchmarking of Conversational Recommender Systems
• Evaluating Contrastive Feedback for Effective User Simulations