176 citations · 914 across the 61 of their papers we have counts for
63 papers · 1 filter
Towards Critical IR Theories and Practices
Bhaskar Mitra
Belkin and Robertson urged us, half a century ago, to develop a theoretical foundation for understanding what constitutes societal good that can inform information retrieval (IR) r…
From Noise to Order: Learning to Rank via Denoising Diffusion
Sajad Ebrahimi, Bhaskar Mitra, Negar Arabzadeh +4
In information retrieval (IR), learning-to-rank (LTR) methods have traditionally limited themselves to discriminative machine learning approaches that model the probability of the…
Overview of the TREC 2022 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +4
This is the fourth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels…
Overview of the TREC 2023 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +5
This is the fifth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels a…
Overview of the TREC 2021 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels a…
Towards Understanding Bias in Synthetic Data for Evaluation
Hossein A. Rahmani, Varsha Ramineni, Emine Yilmaz +2
Test collections are crucial for evaluating Information Retrieval (IR) systems. Creating a diverse set of user queries for these collections can be challenging, and obtaining relev…
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi +6
Using Large Language Models (LLMs) for relevance assessments offers promising opportunities to improve Information Retrieval (IR), Natural Language Processing (NLP), and related fi…
Tip of the Tongue Query Elicitation for Simulated Evaluation
Yifan He, To Eun Kim, Fernando Diaz +2
Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effective…
Emancipatory Information Retrieval
Bhaskar Mitra
Our world today is facing a confluence of several mutually reinforcing crises each of which intersects with concerns of social justice and emancipation. This paper is a provocation…
JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
Hossein A. Rahmani, Emine Yilmaz, Nick Craswell +1
The effective training and evaluation of retrieval systems require a substantial amount of relevance judgments, which are traditionally collected from human assessors -- a process…
LLMJudge: LLMs for Relevance Judgments
Hossein A. Rahmani, Emine Yilmaz, Nick Craswell +6
The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation…
Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi +6
The first edition of the workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) took place in July 2024, co-located with the ACM SIGIR Conference…
SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
Hossein A. Rahmani, Xi Wang, Emine Yilmaz +3
Large-scale test collections play a crucial role in Information Retrieval (IR) research. However, according to the Cranfield paradigm and the research into publicly available datas…
Synthetic Test Collections for Retrieval Evaluation
Hossein A. Rahmani, Nick Craswell, Emine Yilmaz +2
Test collections play a vital role in evaluation of information retrieval (IR) systems. Obtaining a diverse set of user queries for test collection construction can be challenging,…
Towards Group-aware Search Success
Haolun Wu, Bhaskar Mitra, Nick Craswell
Traditional measures of search success often overlook the varying information needs of different demographic groups. To address this gap, we introduce a novel metric, named Group-a…
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
Information retrieval (IR) research must understand and contend with the social implications of the technology it produces. Instead of adopting a reactionary strategy of trying to…
Large language models can accurately predict searcher preferences
Paul Thomas, Seth Spielman, Nick Craswell +1
Relevance labels, which indicate whether a search result is valuable to a searcher, are key to evaluating and optimising search systems. The best way to capture the true preference…
Patterns of gender-specializing query reformulation
Amifa Raj, Bhaskar Mitra, Nick Craswell +1
Users of search systems often reformulate their queries by adding query terms to reflect their evolving information need or to more precisely express their information need when th…
Recall, Robustness, and Lexicographic Evaluation
Fernando Diaz, Michael D. Ekstrand, Bhaskar Mitra
Although originally developed to evaluate sets of items, recall is often used to evaluate rankings of items, including those produced by recommender, retrieval, and other machine l…
Taking Search to Task
Chirag Shah, Ryen W. White, Paul Thomas +3
The importance of tasks in information retrieval (IR) has been long argued for, addressed in different ways, often ignored, and frequently revisited. For decades, scholars made a c…
Result Diversification in Search and Recommendation: A Survey
Haolun Wu, Yansen Zhang, Chen Ma +4
Diversifying return results is an important research topic in retrieval systems in order to satisfy both the various interests of customers and the equal market exposure of provide…
Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systems
Sebastian Hofstätter, Nick Craswell, Bhaskar Mitra +2
Recently, several dense retrieval (DR) models have demonstrated competitive performance to term-based retrieval that are ubiquitous in search systems. In contrast to term-based mat…
Joint Multisided Exposure Fairness for Recommendation
Haolun Wu, Bhaskar Mitra, Chen Ma +2
Prior research on exposure fairness in the context of recommender systems has focused mostly on disparities in the exposure of individual or groups of items to individual users of…
Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?
Gabriella Kazai, Bhaskar Mitra, Anlei Dong +2
Traditional information retrieval (IR) ranking models process the full text of documents. Newer models based on Transformers, however, would incur a high computational cost when pr…