Evaluating Information Retrieval Systems for Kids

Abstract

Evaluation of information retrieval systems (IRS) is a prominent topic among information retrieval researchers--mainly directed at a general population. Children require unique IRS and by extension different ways to evaluate these systems, but as a large population that use IRS have largely been ignored on the evaluation front. In this position paper, we explore many perspectives that must be considered when evaluating IRS; we specially discuss problems faced by researchers who work with children IRS, including lack of evaluation frameworks, limitations of data, and lack of user judgment understanding.

Accepted at the 4th International and Interdisciplinary Perspectives on Children & Recommender and Information Retrieval Systems (KidRec '20), co-located with the 19th ACM International Conference on Interaction Design and Children (IDC '20), https://kidrec.github.io/