4 papers
Formalized Information Needs Improve Large-Language-Model Relevance Judgments
Jüri Keller, Maik Fröbe, Björn Engelmann +4
Cranfield-style retrieval evaluations with too few or too many relevant documents or with low inter-assessor agreement on relevance can reduce the reliability of observations. In e…
Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction
Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer +2
Validating user simulation is a difficult task due to the lack of established measures and benchmarks, which makes it challenging to assess whether a simulator accurately reflects…
Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog +2
Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central…
Evaluating Contrastive Feedback for Effective User Simulations
Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer
The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their a…