The Viability of Crowdsourcing for RAG Evaluation

Abstract

How good are humans at writing and judging responses in retrieval-augmented generation (RAG) scenarios? To answer this question, we investigate the efficacy of crowdsourcing for RAG through two complementary studies: response writing and response utility judgment. We present the Crowd RAG Corpus 2025 (CrowdRAG-25), which consists of 903 human-written and 903 LLM-generated responses for the 301 topics of the TREC RAG'24 track, across the three discourse styles 'bulleted list', 'essay', and 'news'. For a selection of 65 topics, the corpus further contains 47,320 pairwise human judgments and 10,556 pairwise LLM judgments across seven utility dimensions (e.g., coverage and coherence). Our analyses give insights into human writing behavior for RAG and the viability of crowdsourcing for RAG evaluation. Human pairwise judgments provide reliable and cost-effective results compared to LLM-based pairwise or human/LLM-based pointwise judgments, as well as automated comparisons with human-written reference responses. All our data and tools are freely available.

11 pages, 9 tables, 5 figures. Accepted at SIGIR'25

References in corpus (22)

• Retrieval-Augmented Generation for Large Language Models: A Survey
• News Summarization and Evaluation in the Era of GPT-3
• Perspectives on Large Language Models for Relevance Judgment
• One-Shot Labeling for Automatic Relevance Estimation
• Evaluating Generative Ad Hoc Information Retrieval
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution
• RankME: Reliable Human Ratings for Natural Language Generation
• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
• RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
• A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
• Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval
• RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge
• Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
• LLMs Can Patch Up Missing Relevance Judgments in Evaluation
• DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation
• SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
• Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
• JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
• GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems