Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets

Abstract

Retrieval-Augmented Generation (RAG) has advanced significantly in recent years. The complexity of RAG systems, which involve multiple components-such as indexing, retrieval, and generation-along with numerous other parameters, poses substantial challenges for systematic evaluation and quality enhancement. Previous research highlights that evaluating RAG systems is essential for documenting advancements, comparing configurations, and identifying effective approaches for domain-specific applications. This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies, focusing on four key areas: datasets, retrievers, indexing and databases, and the generator component. We observe the feasibility of an automated evaluation approach for each component of a RAG system, leveraging an LLM capable of both generating evaluation datasets and conducting evaluations. In addition, we found that further practical research is essential to provide companies with clear guidance on the do's and don'ts of implementing and evaluating RAG systems. By synthesizing evaluation approaches for key RAG components and emphasizing the creation and adaptation of domain-specific datasets for benchmarking, we contribute to the advancement of systematic evaluation methods and the improvement of evaluation rigor for RAG systems. Furthermore, by examining the interplay between automated approaches leveraging LLMs and human judgment, we contribute to the ongoing discourse on balancing automation and human input, clarifying their respective contributions, limitations, and challenges in achieving robust and reliable evaluations.

8 Pages. This paper has been accepted for presentation at the IEEE Swiss Conference on Data Science (SDS25)

References in corpus (21)

• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
• RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
• LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
• Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese
• DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems
• Lynx: An Open Source Hallucination Evaluation Model
• FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
• Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study
• A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation
• RAGProbe: An Automated Approach for Evaluating RAG Applications
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
• Evaluating the Retrieval Component in LLM-Based Question Answering Systems
• CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
• IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios
• VERA: Validation and Evaluation of Retrieval-Augmented Systems
• CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
• ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions
• Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
• Long${}^2$RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
• Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG