6 papers
LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation
David Carmel, Simone Filice, Guy Horowitz +4
With Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness.…
Redefining Retrieval Evaluation in the Era of LLMs
Giovanni Trappolini, Florin Cuconasu, Simone Filice +2
Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR, assume that human users sequentially examine documents with diminishing attention to lower ranks. This a…
Do RAG Systems Really Suffer From Positional Bias?
Florin Cuconasu, Simone Filice, Guy Horowitz +2
Retrieval Augmented Generation enhances LLM accuracy by adding passages retrieved from an external corpus to the LLM prompt. This paper investigates how positional bias - the tende…
SIGIR 2025 -- LiveRAG Challenge Report
David Carmel, Simone Filice, Guy Horowitz +6
The LiveRAG Challenge at SIGIR 2025, held between March and May 2025, provided a competitive platform for advancing Retrieval-Augmented Generation (RAG) technologies. Participants…
The Distracting Effect: Understanding Irrelevant Passages in RAG
Chen Amiraz, Florin Cuconasu, Simone Filice +1
A well-known issue with Retrieval Augmented Generation (RAG) is that retrieved passages that are irrelevant to the query sometimes distract the answer-generating LLM, causing it to…
Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana
Simone Filice, Guy Horowitz, David Carmel +3
Evaluating Retrieval-Augmented Generation (RAG) systems, especially in domain-specific contexts, requires benchmarks that address the distinctive requirements of the applicative sc…