Pessimistic Evaluation

Abstract

Traditional evaluation of information access systems has focused primarily on average utility across a set of information needs (information retrieval) or users (recommender systems). In this work, we argue that evaluating only with average metric measurements assumes utilitarian values not aligned with traditions of information access based on equal access. We advocate for pessimistic evaluation of information access systems focusing on worst case utility. These methods are (a) grounded in ethical and pragmatic concepts, (b) theoretically complementary to existing robustness and fairness methods, and (c) empirically validated across a set of retrieval and recommendation tasks. These results suggest that pessimistic evaluation should be included in existing experimentation processes to better understand the behavior of systems, especially when concerned with principles of social good.

References in corpus (4)

• Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness
• Distributionally-Informed Recommender System Evaluation
• Measuring Commonality in Recommendation of Cultural Content: Recommender Systems to Enhance Cultural Citizenship
• Lexicographic Optimization: Algorithms and Stability

Cited by in corpus (2)

• Demographically-Inspired Query Variants Using an LLM
• Evaluating and Addressing Fairness Across User Groups in Negative Sampling for Recommender Systems