27 citations · 51 across the 12 of their papers we have counts for
12 papers
SWAN: A Generic Framework for Auditing Textual Conversational Systems
Tetsuya Sakai
We present a simple and generic framework for auditing a given textual conversational system, given some samples of its conversation sessions as its input. The framework computes a…
Relevance Assessments for Web Search Evaluation: Should We Randomise or Prioritise the Pooled Documents? (CORRECTED VERSION)
Tetsuya Sakai, Sijie Tao, Zhaohao Zeng
In the context of depth- pooling for constructing web search test collections, we compare two approaches to ordering pooled documents for relevance assessors: the prioritisation…
Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks
Tetsuya Sakai, Sijie Tao, Maria Maistro +7
Unfortunately, the official English (sub)task results reported in the NTCIR-14 WWW-2, NTCIR-15 WWW-3, and NTCIR-16 WWW-4 overview papers are incorrect due to noise in the official…
On Variants of Root Normalised Order-aware Divergence and a Divergence based on Kendall's Tau
Tetsuya Sakai
This paper reports on a follow-up study of the work reported in Sakai, which explored suitable evaluation measures for ordinal quantification tasks. More specifically, the present…
A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance
Tetsuya Sakai, Jin Young Kim, Inho Kang
We present a simple and versatile framework for evaluating ranked lists in terms of group fairness and relevance, where the groups (i.e., possible attribute values) can be either n…
AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval
Riku Togashi, Mayu Otani, Yuta Nakashima +3
Evaluation measures have a crucial impact on the direction of research. Therefore, it is of utmost importance to develop appropriate and reliable evaluation measures for new applic…
DCH-2: A Parallel Customer-Helpdesk Dialogue Corpus with Distributions of Annotators' Labels
Zhaohao Zeng, Tetsuya Sakai
We introduce a data set called DCH-2, which contains 4,390 real customer-helpdesk dialogues in Chinese and their English translations. DCH-2 also contains dialogue-level annotation…
Scalable Personalised Item Ranking through Parametric Density Estimation
Riku Togashi, Masahiro Kato, Mayu Otani +2
Learning from implicit feedback is challenging because of the difficult nature of the one-class problem: we can observe only positive examples. Most conventional methods use a pair…
How to Measure the Reproducibility of System-oriented IR Experiments
Timo Breuer, Nicola Ferro, Norbert Fuhr +4
Replicability and reproducibility of experimental results are primary concerns in all the areas of science and IR is not an exception. Besides the problem of moving the field towar…
Different Types of Voice User Interface Failures May Cause Different Degrees of Frustration
Shiyoh Goetsu, Tetsuya Sakai
We report on an investigation into how different types of failures in a voice user interface (VUI) affects user frustration. To this end, we conducted a pilot user study () a…
RSL19BD at DBDC4: Ensemble of Decision Tree-based and LSTM-based Models
Chih-Hao Wang, Sosuke Kato, Tetsuya Sakai
RSL19BD (Waseda University Sakai Laboratory) participated in the Fourth Dialogue Breakdown Detection Challenge (DBDC4) and submitted five runs to both English and Japanese subtasks…
Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years
Tetsuya Sakai
NTCIR was the first large-scale IR evaluation conference to construct test collections with graded relevance assessments: the NTCIR-1 test collections from 1998 already featured re…