9 citations · 24 across the 27 of their papers we have counts for
27 papers
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
Bryan Li, William Walden, Yu Hou +6
Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generation (RAG) systems. Core to m…
ColBERTSaR: Sparsified ColBERT Index via Product Quantization
Eugene Yang, Andrew Yates, Dawn Lawrie +3
While ColBERT is an effective neural retrieval architecture, it requires a heavy index structure to support candidate set retrieval based on approximated token embeddings, gatherin…
CoverageBench: Evaluating Information Coverage across Tasks and Domains
Saron Samuel, Andrew Yates, Dawn Lawrie +4
We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Inf…
Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage
Saron Samuel, Alexander Martin, Eugene Yang +5
Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation. While the relat…
NeuCLIRTech: Chinese Monolingual and Cross-Language Information Retrieval Evaluation in a Challenging Domain
Dawn Lawrie, James Mayfield, Eugene Yang +6
Measuring advances in retrieval requires test collections with relevance judgments that can faithfully distinguish systems. This paper presents NeuCLIRTech, an evaluation collectio…
Overview of the TREC 2025 RAGTIME Track
Dawn Lawrie, Sean MacAvaney, James Mayfield +3
The principal goal of the RAG TREC Instrument for Multilingual Evaluation (RAGTIME) track at TREC is to study report generation from multilingual source documents. The track has cr…
RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation
Eugene Yang, Andrew Yates, Dawn Lawrie +2
Retrieval models are key components of Retrieval-Augmented Generation (RAG) systems, which generate search queries, process the documents returned, and generate a response. RAG sys…
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Laura Dietz, Bryan Li, Eugene Yang +3
RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in…
Incorporating Q&A Nuggets into Retrieval-Augmented Generation
Laura Dietz, Bryan Li, Gabrielle Liu +5
RAGE systems integrate ideas from automatic evaluation (E) into Retrieval-augmented Generation (RAG). As one such example, we present Crucible, a Nugget-Augmented Generation System…
FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG
Maxime Dassen, Rebecca Kotula, Kenton Murray +5
Retrieval-Augmented Generation (RAG) models are critically undermined by citation hallucinations, a deceptive failure where a model cites a source that fails to support its claim.…
NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval
Dawn Lawrie, James Mayfield, Eugene Yang +6
To measure advances in retrieval, test collections with relevance judgments that can faithfully distinguish systems are required. This paper presents NeuCLIRBench, an evaluation co…
Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction
Debashish Chakraborty, Eugene Yang, Daniel Khashabi +2
Retrieval-Augmented Generation (RAG) enhances factual grounding in large language models (LLMs) by incorporating retrieved evidence, but LLM accuracy declines when long or noisy co…
Query Decomposition for RAG: Balancing Exploration-Exploitation
Roxana Petcu, Kenton Murray, Daniel Khashabi +4
Retrieval-augmented generation (RAG) systems address complex user requests by decomposing them into subqueries, retrieving potentially relevant documents for each, and then aggrega…
HLTCOE at TREC 2024 NeuCLIR Track
Eugene Yang, Dawn Lawrie, Orion Weller +1
The HLTCOE team applied PLAID, an mT5 reranker, GPT-4 reranker, score fusion, and document translation to the TREC 2024 NeuCLIR track. For PLAID we included a variety of models and…
Overview of the TREC 2024 NeuCLIR Track
Dawn Lawrie, Sean MacAvaney, James Mayfield +4
The principal goal of the TREC Neural Cross-Language Information Retrieval (NeuCLIR) track is to study the effect of neural approaches on cross-language information access. The tra…
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Marc Marone, Orion Weller, William Fleshman +3
Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent…
Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
Dayeon Ki, Marine Carpuat, Paul McNamee +4
Multilingual Retrieval-Augmented Generation (mRAG) systems enable language models to answer knowledge-intensive queries with citation-supported responses across languages. Despite…
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Orion Weller, Kathryn Ricci, Marc Marone +3
The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of th…
HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval
Kevin Duh, Eugene Yang, Orion Weller +2
The HLTCOE LiveRAG submission utilized the GPT-researcher framework for researching the context of the question, filtering the returned results, and generating the final answer. Th…
Rank-K: Test-Time Reasoning for Listwise Reranking
Eugene Yang, Andrew Yates, Kathryn Ricci +4
Retrieve-and-rerank is a popular retrieval pipeline because of its ability to make slow but effective rerankers efficient enough at query time by reducing the number of comparisons…
MURR: Model Updating with Regularized Replay for Searching a Document Stream
Eugene Yang, Nicola Tonellotto, Dawn Lawrie +4
The Internet produces a continuous stream of new documents and user-generated queries. These naturally change over time based on events in the world and the evolution of language.…
Rank1: Test-Time Compute for Reranking in Information Retrieval
Orion Weller, Kathryn Ricci, Eugene Yang +3
We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language mo…
mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval
Orion Weller, Benjamin Chang, Eugene Yang +7
Retrieval systems generally focus on web-style queries that are short and underspecified. However, advances in language models have facilitated the nascent rise of retrieval models…
Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models
Orion Weller, Benjamin Van Durme, Dawn Lawrie +3
Instruction-tuned language models (LM) are able to respond to imperative commands, providing a more natural user interface compared to their base counterparts. In this work, we pre…