Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance

Abstract

Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We empirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.

Accepted to SIGIR 2024 as full paper

References in corpus (8)

• Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning
• A Survey on the Fairness of Recommender Systems
• Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics
• Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study
• Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking
• FAIR: Fairness-Aware Information Retrieval Evaluation
• Marginal-Certainty-aware Fair Ranking Algorithm
• FARA: Future-aware Ranking Algorithm for Fairness Optimization

Cited by in corpus (2)

• Stairway to Fairness: Connecting Group and Individual Fairness
• Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier