8 citations · 16 across the 7 of their papers we have counts for
7 papers · 1 filter
Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations
Laura Dietz
Evaluating AI/Agentic system outputs reliably requires human judgment, but how one incorporates the human determines whether one gets a real quality signal or expensive theater. Th…
Criteria-Based LLM Relevance Judgments
Naghmeh Farzi, Laura Dietz
Relevance judgments are crucial for evaluating information retrieval systems, but traditional human-annotated labels are time-consuming and expensive. As a result, many researchers…
Does UMBRELA Work on Other LLMs?
Naghmeh Farzi, Laura Dietz
We reproduce the UMBRELA LLM Judge evaluation framework across a range of large language models (LLMs) to assess its generalizability beyond the original study. Our investigation e…
LLM-based relevance assessment still can't replace human relevance assessment
Charles L. A. Clarke, Laura Dietz
The use of large language models (LLMs) for relevance assessment in information retrieval has gained significant attention, with recent studies suggesting that LLM-based judgments…
Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3
Naghmeh Farzi, Laura Dietz
Traditional evaluation of information retrieval (IR) systems relies on human-annotated relevance labels, which can be both biased and costly at scale. In this context, large langua…
A Workbench for Autograding Retrieve/Generate Systems
Laura Dietz
This resource paper addresses the challenge of evaluating Information Retrieval (IR) systems in the era of autoregressive Large Language Models (LLMs). Traditional methods relying…
An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments
Naghmeh Farzi, Laura Dietz
Current IR evaluation is based on relevance judgments, created either manually or automatically, with decisions outsourced to Large Language Models (LLMs). We offer an alternative…