Evaluating Generative Ad Hoc Information Retrieval

Abstract

Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.

14 pages, 6 figures, 1 table. Published at SIGIR'24 perspective paper track

References in corpus (8)

• Survey of Hallucination in Natural Language Generation
• Perspectives on Large Language Models for Relevance Judgment
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Expansion via Prediction of Importance with Contextualization
• Explain and Predict, and then Predict Again
• Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study
• The Infinite Index: Information Retrieval on Generative Text-To-Image Models
• The Archive Query Log: Mining Millions of Search Result Pages of Hundreds of Search Engines from 25 Years of Web Archives

Cited by in corpus (2)

• The Viability of Crowdsourcing for RAG Evaluation
• Searching Personal Collections