Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets

Abstract

Retrieval-Augmented Generation (RAG) has advanced significantly in recent years. The complexity of RAG systems, which involve multiple components-such as indexing, retrieval, and generation-along with numerous other parameters, poses substantial challenges for systematic evaluation and quality enhancement. Previous research highlights that evaluating RAG systems is essential for documenting advancements, comparing configurations, and identifying effective approaches for domain-specific applications. This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies, focusing on four key areas: datasets, retrievers, indexing and databases, and the generator component. We observe the feasibility of an automated evaluation approach for each component of a RAG system, leveraging an LLM capable of both generating evaluation datasets and conducting evaluations. In addition, we found that further practical research is essential to provide companies with clear guidance on the do's and don'ts of implementing and evaluating RAG systems. By synthesizing evaluation approaches for key RAG components and emphasizing the creation and adaptation of domain-specific datasets for benchmarking, we contribute to the advancement of systematic evaluation methods and the improvement of evaluation rigor for RAG systems. Furthermore, by examining the interplay between automated approaches leveraging LLMs and human judgment, we contribute to the ongoing discourse on balancing automation and human input, clarifying their respective contributions, limitations, and challenges in achieving robust and reliable evaluations.

8 Pages. This paper has been accepted for presentation at the IEEE Swiss Conference on Data Science (SDS25)

References in corpus (33)

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• BERTScore: Evaluating Text Generation with BERT
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Evaluation of Retrieval-Augmented Generation: A Survey
• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
• RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
• RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
• CRAG -- Comprehensive RAG Benchmark
• LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
• Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Need
• Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation
• Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese
• Lynx: An Open Source Hallucination Evaluation Model
• Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems
• DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation
• A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation
• Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study
• FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
• RAGProbe: An Automated Approach for Evaluating RAG Applications
• Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems
• CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
• Evaluating the Retrieval Component in LLM-Based Question Answering Systems
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
• CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
• UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis
• Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
• VERA: Validation and Evaluation of Retrieval-Augmented Systems
• Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG
• IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios
• ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions
• Long${}^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall