Perspectives on Large Language Models for Relevance Judgment

Abstract

When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in evaluations of retrieval systems. In this perspectives paper, we discuss possible ways for LLMs to support relevance judgments along with concerns and issues that arise. We devise a human--machine collaboration spectrum that allows to categorize different relevance judgment strategies, based on how much humans rely on machines. For the extreme point of "fully automated judgments", we further include a pilot experiment on whether LLM-based relevance judgments correlate with judgments from trained human assessors. We conclude the paper by providing opposing perspectives for and against the use of~LLMs for automatic relevance judgments, and a compromise perspective, informed by our analyses of the literature, our preliminary experimental evidence, and our experience as IR researchers.

References in corpus (8)

• Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
• Large Language Models Are Human-Level Prompt Engineers
• Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks
• Faithful Reasoning Using Large Language Models
• Report from Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education
• Faithful Chain-of-Thought Reasoning
• Multimodal Knowledge Alignment with Reinforcement Learning
• ExaRanker: Explanation-Augmented Neural Ranker

Cited by in corpus (24)

• Large Language Models for Information Retrieval: A Survey
• Leveraging LLMs for the Quality Assurance of Software Requirements
• Evaluating Generative Ad Hoc Information Retrieval
• LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations
• Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
• Leveraging LLMs for Unsupervised Dense Retriever Ranking
• Benchmarking LLM-based Relevance Judgment Methods
• Generative Information Retrieval Evaluation
• A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
• Criteria-Based LLM Relevance Judgments
• LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?
• Does UMBRELA Work on Other LLMs?
• Demographically-Inspired Query Variants Using an LLM
• The Viability of Crowdsourcing for RAG Evaluation
• Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation
• Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search
• Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I
• Can Large Language Models Assess Serendipity in Recommender Systems?
• The Effects of Demographic Instructions on LLM Personas
• When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
• Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement
• Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
• A Universal Framework for Offline Serendipity Evaluation in Recommender Systems via Large Language Models
• Searching Personal Collections