Publications (7)
Perspectives on Large Language Models for Relevance Judgment
Guglielmo Faggioli, Laura Dietz, Charles Clarke +8
When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in ev…
Evaluation-as-a-Service: Overview and Outlook
Allan Hanbury, Henning Müller, Krisztian Balog +11
Evaluation in empirical computer science is essential to show progress and assess technologies developed. Several research domains such as information retrieval have long relied on…
Assessing the Effectiveness of LLMs in Delivering Cognitive Behavioral Therapy
Navdeep Singh Bedi, Ana-Maria Bucur, Noriko Kando +1
As mental health issues continue to rise globally, there is an increasing demand for accessible and scalable therapeutic solutions. Many individuals currently seek support from Lar…
Characterizing Personality from Eye-Tracking: The Role of Gaze and Its Absence in Interactive Search Environments
Jiaman He, Marta Micheli, Damiano Spina +3
Personality traits influence how individuals engage, behave, and make decisions during the information-seeking process. However, few studies have linked personality to observable s…
Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark
Yao Wang, Xin Liu, Zhuochen Liu +5
Existing human value datasets do not directly support value understanding in factual news: many are actor-agnostic, rely on isolated utterances or synthetic scenarios, and lack exp…
Test Collections for Patent-to-Patent Retrieval and Patent Map Generation in NTCIR-4 Workshop
Atsushi Fujii, Makoto Iwayama, Noriko Kando
This paper describes the Patent Retrieval Task in the Fourth NTCIR Workshop, and the test collections produced in this task. We perform the invalidity search task, in which each pa…