On the Difficulty of Evaluating Baselines: A Study on Recommender Systems

Abstract

Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.

Cited by in corpus (29)

• A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• On Sampling Top-K Recommendation Evaluation
• Recommendations and User Agency: The Reachability of Collaboratively-Filtered Information
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• Contemporary Recommendation Systems on Big Data and Their Applications: A Survey
• DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
• Towards a Better Understanding of Linear Models for Recommendation
• Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems
• On component interactions in two-stage recommender systems
• Do Offline Metrics Predict Online Performance in Recommender Systems?
• A Lightweight Method for Modeling Confidence in Recommendations with Learned Beta Distributions
• User Response Prediction in Online Advertising
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• An Inertial Block Majorization Minimization Framework for Nonsmooth Nonconvex Optimization
• Exploring Data Splitting Strategies for the Evaluation of Recommendation Models
• Content Based Player and Game Interaction Model for Game Recommendation in the Cold Start setting
• It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendation
• Accelerated learning from recommender systems using multi-armed bandit
• UserReg: A Simple but Strong Model for Rating Prediction
• On Estimating Recommendation Evaluation Metrics under Sampling
• How robust is MovieLens? A dataset analysis for recommender systems
• Review Regularized Neural Collaborative Filtering
• Prevention is Better than Cure: Handling Basis Collapse and Transparency in Dense Networks
• Ideas for Improving the Field of Machine Learning: Summarizing Discussion from the NeurIPS 2019 Retrospectives Workshop
• Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning
• New Recommendation Algorithm for Implicit Data Motivated by the Multivariate Normal Distribution
• Local Search Algorithms for Rank-Constrained Convex Optimization
• Statistical Inference: The Missing Piece of RecSys Experiment Reliability Discourse