176 citations · 697 across the 27 of their papers we have counts for
18 papers · 1 filter
Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision
Fernando Diaz
Across a variety of ranking tasks, researchers use reciprocal rank to measure the effectiveness for users interested in exactly one relevant item. Despite its widespread use, evide…
Overview of the TREC 2021 Fair Ranking Track
Michael D. Ekstrand, Graham McDonald, Amifa Raj +1
The TREC Fair Ranking Track aims to provide a platform for participants to develop and evaluate novel retrieval algorithms that can provide a fair exposure to a mixture of demograp…
Recall, Robustness, and Lexicographic Evaluation
Fernando Diaz, Michael D. Ekstrand, Bhaskar Mitra
Although originally developed to evaluate sets of items, recall is often used to evaluate rankings of items, including those produced by recommender, retrieval, and other machine l…
Retrieval Augmentation for T5 Re-ranker using External Sources
Kai Hui, Tao Chen, Zhen Qin +4
Retrieval augmentation has shown promising improvements in different tasks. However, whether such augmentation can assist a large language model based re-ranker remains unclear. We…
Offline Retrieval Evaluation Without Evaluation Metrics
Fernando Diaz, Andres Ferraro
Offline evaluation of information retrieval and recommendation has traditionally focused on distilling the quality of a ranking into a scalar metric such as average precision or no…
Exposing Query Identification for Search Transparency
Ruohan Li, Jianxiang Li, Bhaskar Mitra +2
Search systems control the exposure of ranked content to searchers. In many cases, creators value not only the exposure of their content but, moreover, an understanding of the spec…
Estimation of Fair Ranking Metrics with Incomplete Judgments
Ömer Kırnap, Fernando Diaz, Asia Biega +3
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identi…
Overview of the TREC 2020 Fair Ranking Track
Asia J. Biega, Fernando Diaz, Michael D. Ekstrand +2
This paper provides an overview of the NIST TREC 2020 Fair Ranking track. For 2020, we again adopted an academic search task, where we have a corpus of academic article abstracts a…
Fairness in Information Access Systems
Michael D. Ekstrand, Anubrata Das, Robin Burke +1
Recommendation, information retrieval, and other information access systems pose unique challenges for investigating and applying the fairness and non-discrimination concepts that…
Tip of the Tongue Known-Item Retrieval: A Case Study in Movie Identification
Jaime Arguello, Adam Ferguson, Emery Fine +3
While current information retrieval systems are effective for known-item retrieval where the searcher provides a precise name or identifier for the item being sought, systems tend…
Analyzing and Learning from User Interactions for Search Clarification
Hamed Zamani, Bhaskar Mitra, Everest Chen +5
Asking clarifying questions in response to search queries has been recognized as a useful technique for revealing the underlying intent of the query. Clarification has applications…
Evaluating Stochastic Rankings with Expected Exposure
Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand +2
We introduce the concept of \emph{expected exposure} as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for…
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
Bhaskar Mitra, Corby Rosset, David Hawking +3
Classical information retrieval (IR) methods, such as query likelihood and BM25, score documents independently w.r.t. each query term, and then accumulate the scores. Assuming quer…
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz +3
Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or o…
Luandri: a Clean Lua Interface to the Indri Search Engine
Bhaskar Mitra, Fernando Diaz, Nick Craswell
In recent years, the information retrieval (IR) community has witnessed the first successful applications of deep neural network models to short-text matching and ad-hoc retrieval.…
Learning to Match Using Local and Distributed Representations of Text for Web Search
Bhaskar Mitra, Fernando Diaz, Nick Craswell
Models such as latent semantic analysis and those based on neural embeddings learn distributed representations of text, and match the query against the document in the latent seman…
Query Expansion with Locally-Trained Word Embeddings
Fernando Diaz, Bhaskar Mitra, Nick Craswell
Continuous space word embeddings have received a great deal of attention in the natural language processing and machine learning communities for their ability to model term similar…
Pseudo-Query Reformulation
Fernando Diaz
Automatic query reformulation refers to rewriting a user's original query in order to improve the ranking of retrieval results compared to the original query. We present a general…