25 citations · 38 across the 18 of their papers we have counts for
9 papers · 1 filter
SFBench: The SciFy Scientific Feasibility Benchmark
Cash Costello, James Mayfield, Elsbeth Turcan +7
We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated wi…
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
Bryan Li, William Walden, Yu Hou +6
Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generation (RAG) systems. Core to m…
ColBERTSaR: Sparsified ColBERT Index via Product Quantization
Eugene Yang, Andrew Yates, Dawn Lawrie +3
While ColBERT is an effective neural retrieval architecture, it requires a heavy index structure to support candidate set retrieval based on approximated token embeddings, gatherin…
NeuCLIRTech: Chinese Monolingual and Cross-Language Information Retrieval Evaluation in a Challenging Domain
Dawn Lawrie, James Mayfield, Eugene Yang +6
Measuring advances in retrieval requires test collections with relevance judgments that can faithfully distinguish systems. This paper presents NeuCLIRTech, an evaluation collectio…
Overview of the TREC 2025 RAGTIME Track
Dawn Lawrie, Sean MacAvaney, James Mayfield +3
The principal goal of the RAG TREC Instrument for Multilingual Evaluation (RAGTIME) track at TREC is to study report generation from multilingual source documents. The track has cr…
RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation
Eugene Yang, Andrew Yates, Dawn Lawrie +2
Retrieval models are key components of Retrieval-Augmented Generation (RAG) systems, which generate search queries, process the documents returned, and generate a response. RAG sys…
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Laura Dietz, Bryan Li, Eugene Yang +3
RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in…
Incorporating Q&A Nuggets into Retrieval-Augmented Generation
Laura Dietz, Bryan Li, Gabrielle Liu +5
RAGE systems integrate ideas from automatic evaluation (E) into Retrieval-augmented Generation (RAG). As one such example, we present Crucible, a Nugget-Augmented Generation System…
FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG
Maxime Dassen, Rebecca Kotula, Kenton Murray +5
Retrieval-Augmented Generation (RAG) models are critically undermined by citation hallucinations, a deceptive failure where a model cites a source that fails to support its claim.…