117 citations · 511 across the 34 of their papers we have counts for
37 papers
Overview of the TREC 2021 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels a…
A Survey on Asking Clarification Questions Datasets in Conversational Systems
Hossein A. Rahmani, Xi Wang, Yue Feng +3
The ability to understand a user's underlying needs is critical for conversational systems, especially with limited input from users in a conversation. Thus, in such a domain, Aski…
Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues
Yue Feng, Hossein A. Rahmani, Aldo Lipani +1
Task-oriented dialogue systems aim at providing users with task-specific services. Users of such systems often do not know all the information about the task they are trying to acc…
Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process
Fanghua Ye, Zhiyuan Hu, Emine Yilmaz
Dialogue systems have received increasing attention while automatically evaluating their performance remains challenging. User satisfaction estimation (USE) has been proposed as an…
Query-specific Variable Depth Pooling via Query Performance Prediction towards Reducing Relevance Assessment Effort
Debasis Ganguly, Emine Yilmaz
Due to the massive size of test collections, a standard practice in IR evaluation is to construct a 'pool' of candidate relevant documents comprised of the top-k documents retrieve…
Task2KB: A Public Task-Oriented Knowledge Base
Procheta Sen, Xi Wang, Ruiqing Xu +1
Search engines and conversational assistants are commonly used to help users complete their every day tasks such as booking travel, cooking, etc. While there are some existing data…
Evaluation Metrics for Measuring Bias in Search Engine Results
Gizem Gezici, Aldo Lipani, Yucel Saygin +1
Search engines decide what we see for a given search query. Since many people are exposed to information through search engines, it is fair to expect that search engines are neutra…
ViralBERT: A User Focused BERT-Based Approach to Virality Prediction
Rikaz Rameez, Hossein A. Rahmani, Emine Yilmaz
Recently, Twitter has become the social network of choice for sharing and spreading information to a multitude of users through posts called 'tweets'. Users can easily re-share the…
Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking
Yue Feng, Aldo Lipani, Fanghua Ye +2
Dialogue State Tracking (DST) aims to keep track of users' intentions during the course of a conversation. In DST, modelling the relations among domains and slots is still an under…
ASSIST: Towards Label Noise-Robust Dialogue State Tracking
Fanghua Ye, Yue Feng, Emine Yilmaz
The MultiWOZ 2.0 dataset has greatly boosted the research on dialogue state tracking (DST). However, substantial noise has been discovered in its state annotations. Such noise brin…
Towards More Accountable Search Engines: Online Evaluation of Representation Bias
Aldo Lipani, Florina Piroi, Emine Yilmaz
Information availability affects people's behavior and perception of the world. Notably, people rely on search engines to satisfy their need for information. Search engines deliver…
Sample Efficient Model Evaluation
Emine Yilmaz, Peter Hayes, Raza Habib +2
Labelling data is a major practical bottleneck in training and testing classifiers. Given a collection of unlabelled data points, we address how to select which subset to label to…
PEEK: A Large Dataset of Learner Engagement with Educational Videos
Sahan Bulathwela, Maria Perez-Ortiz, Erik Novak +2
Educational recommenders have received much less attention in comparison to e-commerce and entertainment-related recommenders, even though efficient intelligent tutors have great p…
Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
Fangyu Liu, Yunlong Jiao, Jordan Massiah +2
In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g. sentence similarity and paraphrase identification). Predominantly, two formulations are used…
Estimation of Fair Ranking Metrics with Incomplete Judgments
Ömer Kırnap, Fernando Diaz, Asia Biega +3
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identi…
Learning to Detect Few-Shot-Few-Clue Misinformation
Qiang Zhang, Hongbin Huang, Shangsong Liang +2
The quality of digital information on the web has been disquieting due to the lack of careful manual review. Consequently, a large volume of false textual information has been diss…
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +2
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big ques…
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +3
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that…
MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
Fanghua Ye, Jarana Manotumruksa, Emine Yilmaz
The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented dialogue systems. However, its state annotations contain substantial noise, which hinders a proper eva…
Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
Jimmy Lin, Daniel Campos, Nick Craswell +2
Leaderboards are a ubiquitous part of modern research in applied machine learning. By design, they sort entries into some linear order, where the top-scoring entry is recognized as…
Overview of the TREC 2020 deep learning track
Nick Craswell, Bhaskar Mitra, Emine Yilmaz +1
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a p…
Slot Self-Attentive Dialogue State Tracking
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang +2
An indispensable component in task-oriented dialogue systems is the dialogue state tracker, which keeps track of users' intentions in the course of conversation. The typical approa…
VLEngagement: A Dataset of Scientific Video Lectures for Evaluating Population-based Engagement
Sahan Bulathwela, Maria Perez-Ortiz, Emine Yilmaz +1
With the emergence of e-learning and personalised education, the production and distribution of digital educational resources have boomed. Video lectures have now become one of the…
Predicting Engagement in Video Lectures
Sahan Bulathwela, María Pérez-Ortiz, Aldo Lipani +2
The explosion of Open Educational Resources (OERs) in the recent years creates the demand for scalable, automatic approaches to process and evaluate OERs, with the end goal of iden…