Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier

Abstract

Fairness and relevance are two important aspects of recommender systems (RSs). Typically, they are evaluated either (i) separately by individual measures of fairness and relevance, or (ii) jointly using a single measure that accounts for fairness with respect to relevance. However, approach (i) often does not provide a reliable joint estimate of the goodness of the models, as it has two different best models: one for fairness and another for relevance. Approach (ii) is also problematic because these measures tend to be ad-hoc and do not relate well to traditional relevance measures, like NDCG. Motivated by this, we present a new approach for jointly evaluating fairness and relevance in RSs: Distance to Pareto Frontier (DPFR). Given some user-item interaction data, we compute their Pareto frontier for a pair of existing relevance and fairness measures, and then use the distance from the frontier as a measure of the jointly achievable fairness and relevance. Our approach is modular and intuitive as it can be computed with existing measures. Experiments with 4 RS models, 3 re-ranking strategies, and 6 datasets show that existing metrics have inconsistent associations with our Pareto-optimal solution, making DPFR a more robust and theoretically well-founded joint measure for assessing fairness and relevance. Our code: https://github.com/theresiavr/DPFR-recsys-evaluation

Accepted to TheWebConf/WWW 2025 (Oral)

References in corpus (10)

• Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning
• A Survey on the Fairness of Recommender Systems
• Toward Pareto Efficient Fairness-Utility Trade-off inRecommendation through Reinforcement Learning
• A Graph-based Approach for Mitigating Multi-sided Exposure Bias in Recommender Systems
• Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics
• Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study
• Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking
• Principled Multi-Aspect Evaluation Measures of Rankings
• FAIR: Fairness-Aware Information Retrieval Evaluation
• Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance