117 citations · 511 across the 34 of their papers we have counts for
21 papers · 1 filter
Overview of the TREC 2021 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels a…
A Survey on Asking Clarification Questions Datasets in Conversational Systems
Hossein A. Rahmani, Xi Wang, Yue Feng +3
The ability to understand a user's underlying needs is critical for conversational systems, especially with limited input from users in a conversation. Thus, in such a domain, Aski…
Query-specific Variable Depth Pooling via Query Performance Prediction towards Reducing Relevance Assessment Effort
Debasis Ganguly, Emine Yilmaz
Due to the massive size of test collections, a standard practice in IR evaluation is to construct a 'pool' of candidate relevant documents comprised of the top-k documents retrieve…
Task2KB: A Public Task-Oriented Knowledge Base
Procheta Sen, Xi Wang, Ruiqing Xu +1
Search engines and conversational assistants are commonly used to help users complete their every day tasks such as booking travel, cooking, etc. While there are some existing data…
Evaluation Metrics for Measuring Bias in Search Engine Results
Gizem Gezici, Aldo Lipani, Yucel Saygin +1
Search engines decide what we see for a given search query. Since many people are exposed to information through search engines, it is fair to expect that search engines are neutra…
Towards More Accountable Search Engines: Online Evaluation of Representation Bias
Aldo Lipani, Florina Piroi, Emine Yilmaz
Information availability affects people's behavior and perception of the world. Notably, people rely on search engines to satisfy their need for information. Search engines deliver…
PEEK: A Large Dataset of Learner Engagement with Educational Videos
Sahan Bulathwela, Maria Perez-Ortiz, Erik Novak +2
Educational recommenders have received much less attention in comparison to e-commerce and entertainment-related recommenders, even though efficient intelligent tutors have great p…
Estimation of Fair Ranking Metrics with Incomplete Judgments
Ömer Kırnap, Fernando Diaz, Asia Biega +3
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identi…
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big ques…
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +3
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that…
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Jimmy Lin, Daniel Campos, Nick Craswell +2
Leaderboards are a ubiquitous part of modern research in applied machine learning. By design, they sort entries into some linear order, where the top-scoring entry is recognized as…
Overview of the TREC 2020 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +1
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a p…
ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search
Nick Craswell, Daniel Campos, Bhaskar Mitra +2
Users of Web search engines reveal their information needs through queries and clicks, making click logs a useful asset for information retrieval. However, click logs have not been…
Outlier-Resilient Web Service QoS Prediction
Fanghua Ye, Zhiwei Lin, Chuan Chen +2
The proliferation of Web services makes it difficult for users to select the most appropriate one among numerous functionally identical or similar service candidates. Quality-of-Se…
On the Reliability of Test Collections for Evaluating Systems of Different Types
Emine Yilmaz, Nick Craswell, Bhaskar Mitra +1
As deep learning based models are increasingly being used for information retrieval (IR), a major challenge is to ensure the availability of test collections for measuring their qu…
Overview of the TREC 2019 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets,…
Towards an Integrative Educational Recommender for Lifelong Learners
Sahan Bulathwela, Maria Perez-Ortiz, Emine Yilmaz +1
One of the most ambitious use cases of computer-assisted learning is to build a recommendation system for lifelong learning. Most recommender algorithms exploit similarities betwee…
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks
Bhaskar Mitra, Corby Rosset, David Hawking +3
Classical information retrieval (IR) methods, such as query likelihood and BM25, score documents independently w.r.t. each query term, and then accumulate the scores. Assuming quer…
Towards Task Understanding in Visual Settings
Sebastin Santy, Wazeer Zulfikar, Rishabh Mehrotra +1
We consider the problem of understanding real world tasks depicted in visual images. While most existing image captioning methods excel in producing natural language descriptions o…
Extracting Hierarchies of Search Tasks & Subtasks via a Bayesian Nonparametric Approach
Rishabh Mehrotra, Emine Yilmaz
A significant amount of search queries originate from some real world information need or tasks. In order to improve the search experience of the end users, it is important to have…
Auditing Search Engines for Differential Satisfaction Across Demographics
Rishabh Mehrotra, Ashton Anderson, Fernando Diaz +3
Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or o…