Widespread Flaws in Offline Evaluation of Recommender Systems

Abstract

Even though offline evaluation is just an imperfect proxy of online performance -- due to the interactive nature of recommenders -- it will probably remain the primary way of evaluation in recommender systems research for the foreseeable future, since the proprietary nature of production recommenders prevents independent validation of A/B test setups and verification of online results. Therefore, it is imperative that offline evaluation setups are as realistic and as flawless as they can be. Unfortunately, evaluation flaws are quite common in recommender systems research nowadays, due to later works copying flawed evaluation setups from their predecessors without questioning their validity. In the hope of improving the quality of offline evaluation of recommender systems, we discuss four of these widespread flaws and why researchers should avoid them.

arXiv commentAppearing in the Proceedings of the 17th ACM Conference on Recommender Systems

References in corpus (9)

• Neural Graph Collaborative Filtering
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Evaluation of Session-based Recommendation Algorithms
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Fast ALS-based tensor factorization for context-aware recommendation from implicit feedback
• CARCA: Context and Attribute-Aware Next-Item Recommendation via Cross-Attention
• A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models

Cited by in corpus (12)

• A Survey on Intent-aware Recommender Systems
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• A Reproducible Analysis of Sequential Recommender Systems
• Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• Enhancing Sequential Music Recommendation with Personalized Popularity Awareness
• Semantic IDs for Music Recommendation
• Industry Insights from Comparing Deep Learning and GBDT Models for E-Commerce Learning-to-Rank
• Autoregressive Generation Strategies for Top-K Sequential Recommendations
• Recommendation Is a Dish Better Served Warm
• eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion
• Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems