565 citations · 607 across the 19 of their papers we have counts for
6 papers · 1 filter
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi +6
Using Large Language Models (LLMs) for relevance assessments offers promising opportunities to improve Information Retrieval (IR), Natural Language Processing (NLP), and related fi…
Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding
Kimia Ramezan, Alireza Amiri Bavandpour, Yifei Yuan +2
Conversational query clarification enables users to refine their search queries through interactive dialogue, improving search effectiveness. Traditional approaches rely on text-ba…
LLMJudge: LLMs for Relevance Judgments
Hossein A. Rahmani, Emine Yilmaz, Nick Craswell +6
The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation…
Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi +6
The first edition of the workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) took place in July 2024, co-located with the ACM SIGIR Conference…
Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. In a conversational setting such signals are usually unavailable due to the nature of t…
Understanding User Satisfaction with Task-oriented Dialogue Systems
Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke
Dialogue systems are evaluated depending on their type and purpose. Two categories are often distinguished: (1) task-oriented dialogue systems (TDS), which are typically evaluat…