44 citations · 223 across the 40 of their papers we have counts for
9 papers · 1 filter
New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches
Mehmet Deniz Türkmen, Matthew Lease, Mucahid Kutlu
In evaluation campaigns, participants often explore variations of popular, state-of-the-art baselines as a low-risk strategy to achieve competitive results. While effective, this c…
Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Md Mustafizur Rahman, Mucahid Kutlu, Matthew Lease
Research community evaluations in information retrieval, such as NIST's Text REtrieval Conference (TREC), build reusable test collections by pooling document rankings submitted by…
A Conceptual Framework for Evaluating Fairness in Search
Anubrata Das, Matthew Lease
While search efficacy has been evaluated traditionally on the basis of result relevance, fairness of search has attracted recent attention. In this work, we define a notion of dist…
CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking
Anubrata Das, Kunjan Mehta, Matthew Lease
The effect of user bias in fact-checking has not been explored extensively from a user-experience perspective. We estimate the user bias as a function of the user's perceived reput…
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably
Mucahid Kutlu, Tyler McDonnell, Aashish Sheshadri +2
Crowdsourcing offers an affordable and scalable means to collect relevance judgments for IR test collections. However, crowd assessors may show higher variance in judgment quality…
Correlation and Prediction of Evaluation Metrics in Information Retrieval
Mucahid Kutlu, Vivek Khetan, Matthew Lease
Because researchers typically do not have the time or space to present more than a few evaluation metrics in any published study, it can be difficult to assess relative effectivene…
Efficient Test Collection Construction via Active Learning
Md Mustafizur Rahman, Mucahid Kutlu, Tamer Elsayed +1
To create a new IR test collection at low cost, it is valuable to carefully select which documents merit human relevance judgments. Shared task campaigns such as NIST TREC pool doc…
Neural Information Retrieval: A Literature Review
Ye Zhang, Md Mustafizur Rahman, Alex Braylan +12
A recent "third wave" of Neural Network (NN) approaches now delivers state-of-the-art performance in many machine learning tasks, spanning speech recognition, computer vision, and…
Bullseye: Structured Passage Retrieval and Document Highlighting for Scholarly Search
Xi Zheng, Akanksha Bansal, Matthew Lease
We present the Bullseye system for scholarly search. Given a collection of research papers, Bullseye: 1) identifies relevant passages using any on-the-shelf algorithm; 2) automatic…