User-centered Evaluation of Popularity Bias in Recommender Systems

Abstract

Recommendation and ranking systems are known to suffer from popularity bias; the tendency of the algorithm to favor a few popular items while under-representing the majority of other items. Prior research has examined various approaches for mitigating popularity bias and enhancing the recommendation of long-tail, less popular, items. The effectiveness of these approaches is often assessed using different metrics to evaluate the extent to which over-concentration on popular items is reduced. However, not much attention has been given to the user-centered evaluation of this bias; how different users with different levels of interest towards popular items are affected by such algorithms. In this paper, we show the limitations of the existing metrics to evaluate popularity bias mitigation when we want to assess these algorithms from the users' perspective and we propose a new metric that can address these limitations. In addition, we present an effective approach that mitigates popularity bias from the user-centered point of view. Finally, we investigate several state-of-the-art approaches proposed in recent years to mitigate popularity bias and evaluate their performances using the existing metrics and also from the users' perspective. Our experimental results using two publicly-available datasets show that existing popularity bias mitigation techniques ignore the users' tolerance towards popular items. Our proposed user-centered method can tackle popularity bias effectively for different users while also improving the existing metrics.

arXiv commentProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (UMAP '21), June 21--25, 2021, Utrecht, Netherlands. arXiv admin note: text overlap with arXiv:2007.12230

References in corpus (5)

• FA*IR: A Fair Top-k Ranking Algorithm
• Beyond Parity: Fairness Objectives for Collaborative Filtering
• The Unfairness of Popularity Bias in Recommendation
• Using Stable Matching to Optimize the Balance between Accuracy and Diversity in Recommendation
• Popularity Bias in Recommendation: A Multi-stakeholder Perspective

Cited by in corpus (14)

• Fairness in Recommender Systems: Research Landscape and Future Directions
• A Survey on Popularity Bias in Recommender Systems
• Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems
• Debiasing Recommendation with Personal Popularity
• Long-Tail Session-based Recommendation from Calibration
• Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M
• Oh, Behave! Country Representation Dynamics Created by Feedback Loops in Music Recommender Systems
• Unmasking Gender Bias in Recommendation Systems and Enhancing Category-Aware Fairness
• Impacts of Mainstream-Driven Algorithms on Recommendations for Children Across Domains: A Reproducibility Study
• Performative Debias with Fair-exposure Optimization Driven by Strategic Agents in Recommender Systems
• Exploring the Effect of Context-Awareness and Popularity Calibration on Popularity Bias in POI Recommendations
• On Inherited Popularity Bias in Cold-Start Item Recommendation
• Investigating Characteristics of Media Recommendation Solicitation in r/ifyoulikeblank
• Extending MovieLens-32M to Provide New Evaluation Objectives