Properties of Group Fairness Metrics for Rankings

Abstract

In recent years, several metrics have been developed for evaluating group fairness of rankings. Given that these metrics were developed with different application contexts and ranking algorithms in mind, it is not straightforward which metric to choose for a given scenario. In this paper, we perform a comprehensive comparative analysis of existing group fairness metrics developed in the context of fair ranking. By virtue of their diverse application contexts, we argue that such a comparative analysis is not straightforward. Hence, we take an axiomatic approach whereby we design a set of thirteen properties for group fairness metrics that consider different ranking settings. A metric can then be selected depending on whether it satisfies all or a subset of these properties. We apply these properties on eleven existing group fairness metrics, and through both empirical and theoretical results we demonstrate that most of these metrics only satisfy a small subset of the proposed properties. These findings highlight limitations of existing metrics, and provide insights into how to evaluate and interpret different fairness metrics in practical deployment. The proposed properties can also assist practitioners in selecting appropriate metrics for evaluating fairness in a specific application.

arXiv comment26 pages, 7 figures

References in corpus (11)

• Fairness of Exposure in Rankings
• Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
• Equity of Attention: Amortizing Individual Fairness in Rankings
• Controlling Fairness and Bias in Dynamic Learning-to-Rank
• Evaluating Stochastic Rankings with Expected Exposure
• Fairness in Ranking: A Survey
• An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric
• Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study
• Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
• Evaluating Fairness in Argument Retrieval
• The Role of Relevance in Fair Ranking

Cited by in corpus (2)

• Stairway to Fairness: Connecting Group and Individual Fairness
• hyperFA*IR: A hypergeometric approach to fair rankings with finite candidate pool