Measuring the Business Value of Recommender Systems

Abstract

Recommender Systems are nowadays successfully used by all major web sites (from e-commerce to social media) to filter content and make suggestions in a personalized way. Academic research largely focuses on the value of recommenders for consumers, e.g., in terms of reduced information overload. To what extent and in which ways recommender systems create business value is, however, much less clear, and the literature on the topic is scattered. In this research commentary, we review existing publications on field tests of recommender systems and report which business-related performance measures were used in such real-world deployments. We summarize common challenges of measuring the business value in practice and critically discuss the value of algorithmic improvements and offline experiments as commonly done in academic environments. Overall, our review indicates that various open questions remain both regarding the realistic quantification of the business effects of recommenders and the performance assessment of recommendation algorithms in academia.

References in corpus (4)

• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• A systematic review and taxonomy of explanations in decision support and recommender systems
• Degenerate Feedback Loops in Recommender Systems
• Offline A/B testing for Recommender Systems

Cited by in corpus (27)

• A Survey on Conversational Recommender Systems
• Fairness in Recommender Systems: Research Landscape and Future Directions
• Digital Nudging with Recommender Systems: Survey and Future Directions
• Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
• Balancing Consumer and Business Value of Recommender Systems: A Simulation-based Analysis
• Recommender Systems for Sustainability: Overview and Research Issues
• Interplay between Upsampling and Regularization for Provider Fairness in Recommender Systems
• Evaluating Conversational Recommender Systems: A Landscape of Research
• Unbiased Learning for the Causal Effect of Recommendation
• Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation
• A Survey on Point-of-Interest Recommendations Leveraging Heterogeneous Data
• From Data to Decisions: The Transformational Power of Machine Learning in Business Recommendations
• EB-NeRD: A Large-Scale Dataset for News Recommendation
• Economic Recommender Systems -- A Systematic Review
• Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters
• Diversity of What? On the Different Conceptualizations of Diversity in Recommender Systems
• Top-Personalized-K Recommendation
• Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random
• Adversarial learning for product recommendation
• Semi-supervised Adversarial Learning for Complementary Item Recommendation
• Peer Recommendation Interventions for Health-related Social Support: a Feasibility Assessment
• Adversarial Text Rewriting for Text-aware Recommender Systems
• EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context
• Normative Alignment of Recommender Systems via Internal Label Shift
• M-$LL{M}^3$REC: A Motivation-Aware User-Item Interaction Framework for Enhancing Recommendation Accuracy with LLMs
• Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems
• Searching Personal Collections