32 citations · 106 across the 21 of their papers we have counts for
22 papers
Beyond States: Investigating the Effects of Context on User Modeling with Feature-Conditioned Markov Models
Jana Isabelle Friese, Andreas Konstantin Kruff, Timo Breuer +2
User behavior simulation is widely used to evaluate interactive information retrieval systems, but classical state-based approaches (e.g., Markov models) have limited ability to in…
SQLite is Enough. Lexical, Semantic, and Hybrid Search with scrydb
Timo Breuer
This work introduces scrydb, a Python library that enables lexical, semantic, and hybrid search within SQLite. For lexical search, scrydb leverages SQLite's full-text search extens…
Formalized Information Needs Improve Large-Language-Model Relevance Judgments
Jüri Keller, Maik Fröbe, Björn Engelmann +4
Cranfield-style retrieval evaluations with too few or too many relevant documents or with low inter-assessor agreement on relevance can reduce the reliability of observations. In e…
Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction
Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer +2
Validating user simulation is a difficult task due to the lack of established measures and benchmarks, which makes it challenging to assess whether a simulator accurately reflects…
Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)
Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog +2
Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central…
Evaluating Contrastive Feedback for Effective User Simulations
Andreas Konstantin Kruff, Timo Breuer, Philipp Schaer
The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their a…
Report on the Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024
Timo Breuer, Christin Katharina Kreutz, Norbert Fuhr +15
This paper is a report of the Workshop on Simulations for Information Access (Sim4IA) workshop at SIGIR 2024. The workshop had two keynotes, a panel discussion, nine lightning talk…
Replicability Measures for Longitudinal Information Retrieval Evaluation
Jüri Keller, Timo Breuer, Philipp Schaer
Information Retrieval (IR) systems are exposed to constant changes in most components. Documents are created, updated, or deleted, the information needs are changing, and even rele…
The Two Sides of the Coin: Hallucination Generation and Detection with LLMs as Evaluators for LLMs
Anh Thu Maria Bui, Saskia Felizitas Brech, Natalie Hußfeldt +5
Hallucination detection in Large Language Models (LLMs) is crucial for ensuring their reliability. This work presents our participation in the CLEF ELOQUENT HalluciGen shared task,…
Evaluation of Temporal Change in IR Test Collections
Jüri Keller, Timo Breuer, Philipp Schaer
Information retrieval systems have been evaluated using the Cranfield paradigm for many years. This paradigm allows a systematic, fair, and reproducible evaluation of different ret…
Context-Driven Interactive Query Simulations Based on Generative Large Language Models
Björn Engelmann, Timo Breuer, Jana Isabelle Friese +2
Simulating user interactions enables a more user-oriented evaluation of information retrieval (IR) systems. While user simulations are cost-efficient and reproducible, many approac…
Simulating Users in Interactive Web Table Retrieval
Björn Engelmann, Timo Breuer, Philipp Schaer
Considering the multimodal signals of search items is beneficial for retrieval effectiveness. Especially in web table retrieval (WTR) experiments, accounting for multimodal propert…
Validating Synthetic Usage Data in Living Lab Environments
Timo Breuer, Norbert Fuhr, Philipp Schaer
Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for…
Evaluating Temporal Persistence Using Replicability Measures
Jüri Keller, Timo Breuer, Philipp Schaer
In real-world Information Retrieval (IR) experiments, the Evaluation Environment (EE) is exposed to constant change. Documents are added, removed, or updated, and the information n…
Bibliometric Data Fusion for Biomedical Information Retrieval
Timo Breuer, Christin Katharina Kreutz, Philipp Schaer +1
Digital libraries in the scientific domain provide users access to a wide range of information to satisfy their diverse information needs. Here, ranking results play a crucial role…
Online Information Retrieval Evaluation using the STELLA Framework
Timo Breuer, Narges Tavakolpoursaleh, Johann Schaible +3
Involving users in early phases of software development has become a common strategy as it enables developers to consider user needs from the beginning. Once a system is in product…
ir_metadata: An Extensible Metadata Schema for IR Experiments
Timo Breuer, Jüri Keller, Philipp Schaer
The information retrieval (IR) community has a strong tradition of making the computational artifacts and resources available for future reuse, allowing the validation of experimen…
Overview of LiLAS 2021 -- Living Labs for Academic Search
Philipp Schaer, Timo Breuer, Leyla Jael Castro +3
The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for academic search. The methodological gap between real-world and lab-ba…
Evaluating Elements of Web-based Data Enrichment for Pseudo-Relevance Feedback Retrieval
Timo Breuer, Melanie Pest, Philipp Schaer
In this work, we analyze a pseudo-relevance retrieval method based on the results of web search engines. By enriching topics with text data from web search engine result pages and…
repro_eval: A Python Interface to Reproducibility Measures of System-oriented IR Experiments
Timo Breuer, Nicola Ferro, Maria Maistro +1
In this work we introduce repro_eval - a tool for reactive reproducibility studies of system-oriented information retrieval (IR) experiments. The corresponding Python package provi…
Validating Simulations of User Query Variants
Timo Breuer, Norbert Fuhr, Philipp Schaer
System-oriented IR evaluations are limited to rather abstract understandings of real user behavior. As a solution, simulating user interactions provides a cost-efficient way to sup…
How to Measure the Reproducibility of System-oriented IR Experiments
Timo Breuer, Nicola Ferro, Norbert Fuhr +4
Replicability and reproducibility of experimental results are primary concerns in all the areas of science and IR is not an exception. Besides the problem of moving the field towar…