One-Shot Labeling for Automatic Relevance Estimation

Abstract

Dealing with unjudged documents ("holes") in relevance assessments is a perennial problem when evaluating search systems with offline experiments. Holes can reduce the apparent effectiveness of retrieval systems during evaluation and introduce biases in models trained with incomplete data. In this work, we explore whether large language models can help us fill such holes to improve offline evaluations. We examine an extreme, albeit common, evaluation setting wherein only a single known relevant document per query is available for evaluation. We then explore various approaches for predicting the relevance of unjudged documents with respect to a query and the known relevant document, including nearest neighbor, supervised, and prompting techniques. We find that although the predictions of these One-Shot Labelers (1SL) frequently disagree with human assessments, the labels they produce yield a far more reliable ranking of systems than the single labels do alone. Specifically, the strongest approaches can consistently reach system ranking correlations of over 0.86 with the full rankings over a variety of measures. Meanwhile, the approach substantially increases the reliability of t-tests due to filling holes in relevance assessments, giving researchers more confidence in results they find to be significant. Alongside this work, we release an easy-to-use software package to enable the use of 1SL for evaluation of other ad-hoc collections or systems.

SIGIR 2023

References in corpus (3)

• Scaling Instruction-Finetuned Language Models
• Adaptive Re-Ranking with a Corpus Graph
• On Survivorship Bias in MS MARCO

Cited by in corpus (12)

• Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
• Query Performance Prediction using Relevance Judgments Generated by Large Language Models
• Benchmarking LLM-based Relevance Judgment Methods
• A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
• Neural Passage Quality Estimation for Static Pruning
• Data Fusion of Synthetic Query Variants With Generative Large Language Models
• Criteria-Based LLM Relevance Judgments
• LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?
• Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation
• The Viability of Crowdsourcing for RAG Evaluation
• Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning
• Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications