The Role of Relevance in Fair Ranking
arXiv:2305.05608 · doi:10.1145/3539618.3591933
Abstract
Online platforms mediate access to opportunity: relevance-based rankings create and constrain options by allocating exposure to job openings and job candidates in hiring platforms, or sellers in a marketplace. In order to do so responsibly, these socially consequential systems employ various fairness measures and interventions, many of which seek to allocate exposure based on worthiness. Because these constructs are typically not directly observable, platforms must instead resort to using proxy scores such as relevance and infer them from behavioral signals such as searcher clicks. Yet, it remains an open question whether relevance fulfills its role as such a worthiness score in high-stakes fair rankings. In this paper, we combine perspectives and tools from the social sciences, information retrieval, and fairness in machine learning to derive a set of desired criteria that relevance scores should satisfy in order to meaningfully guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. We assess the impact of these violations on the estimated system fairness and analyze whether existing fairness interventions may mitigate the identified issues. Our analyses and results surface the pressing need for new approaches to relevance collection and generation that are suitable for use in fair ranking.
Published in SIGIR 2023
References in corpus (14)
- Controlling Fairness and Bias in Dynamic Learning-to-Rank
- Evaluating Stochastic Rankings with Expected Exposure
- Fairness in Information Access Systems
- Policy Learning for Fairness in Ranking
- Maximizing Marginal Fairness for Dynamic Learning to Rank
- Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank
- Exploring Artist Gender Bias in Music Recommendation
- Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
- When Fair Ranking Meets Uncertain Inference
- Overview of the TREC 2020 Fair Ranking Track
- Overview of the TREC 2019 Fair Ranking Track
- Pareto-Optimal Fairness-Utility Amortizations in Rankings with a DBN Exposure Model
- Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification
- A Pre-processing Method for Fairness in Ranking