6 papers
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Laura Dietz, Bryan Li, Eugene Yang +3
RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in…
Incorporating Q&A Nuggets into Retrieval-Augmented Generation
Laura Dietz, Bryan Li, Gabrielle Liu +5
RAGE systems integrate ideas from automatic evaluation (E) into Retrieval-augmented Generation (RAG). As one such example, we present Crucible, a Nugget-Augmented Generation System…
Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
Gabrielle Kaili-May Liu, Bryan Li, Arman Cohan +2
Real-world use cases often present RAG systems with complex queries for which relevant information is missing from the corpus or is incomplete. In these settings, RAG systems must…
Auto-ARGUE: LLM-Based Report Generation Evaluation
William Walden, Marc Mason, Orion Weller +10
Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools…
How Grounded is Wikipedia? A Study on Structured Evidential Support and Retrieval
William Walden, Kathryn Ricci, Miriam Wanner +4
Wikipedia is a critical resource for modern NLP, serving as a rich repository of up-to-date and citation-backed information on a wide variety of subjects. The reliability of Wikipe…
Cross-Document Event-Keyed Summarization
William Walden, Pavlo Kuchmiichuk, Alexander Martin +5
Event-keyed summarization (EKS) requires summarizing a specific event described in a document given the document text and an event representation extracted from it. In this work, w…